Add new Mintlify documentation site (docs_new/) (#23001)

Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com>
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com>
Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com>
Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com>
Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com>
Co-authored-by: Maitri Shah <shah29maitri@gmail.com>
Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com>
Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com>
Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com>
Co-authored-by: IshhanKheria <ishhankheria06@gmail.com>
Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com>
Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com>
Co-authored-by: longGGGGGG <553746008@qq.com>
Co-authored-by: Richard <richardchen@radixark.ai>
Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com>
Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com>
Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu>
Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com>
Co-authored-by: nimeshas <nimesha.s106@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>

.gitignore

@@ -192,6 +192,7 @@ work_dirs/
 *.csv
 
 !logo.png
+!docs_new/images/*.png
 
 # Prerequisites
 *.d

docs_new/.github/workflows/sync-lmsys-sglang-blogs.yml

@@ -0,0 +1,39 @@
+name: Sync LMSYS SGLang blogs
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 */12 * * *"
+
+permissions:
+  contents: write
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Sync blog cards
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: python scripts/update_lmsys_sglang_blogs.py
+
+      - name: Commit and push changes
+        run: |
+          if git diff --quiet; then
+            echo "No changes to commit."
+            exit 0
+          fi
+
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git add index.mdx src/generated/lmsys_sglang_blogs.json
+          git commit -m "docs: sync LMSYS SGLang blog cards"
+          git push

docs_new/.gitignore

@@ -0,0 +1,30 @@
+# Node
+node_modules/
+.env
+.DS_Store
+.cache/
+dist/
+.next/
+*.log
+
+# OS
+Thumbs.db
+Desktop.ini
+
+# VSCode
+.vscode/
+
+# Mintlify
+.mintlify/
+
+# Python (if any)
+__pycache__/
+*.pyc
+
+# Misc
+*.swp
+*.swo
+
+.agents
+.claude
+skills-lock.json

docs_new/.mintignore

@@ -0,0 +1,7 @@
+# Mintlify automatically ignores these files and directories:
+# .git, .github, .claude, .agents, .idea, node_modules,
+# README.md, LICENSE.md, CHANGELOG.md, CONTRIBUTING.md
+
+# Draft content
+drafts/
+*.draft.mdx

docs_new/AGENTS.md

@@ -0,0 +1,381 @@
+---
+name: sglang-docs-mintlify
+description: Build and maintain the SGLang documentation site and integrated cookbook using Mintlify. Use when
+  creating docs pages, configuring navigation, adding components, or setting up
+  API references.
+license: Apache-2.0
+compatibility: Requires Node.js for CLI. Works with any Git-based workflow.
+metadata:
+  author: SGLang Team
+  version: "1.0"
+  mintlify-proj: mintlify
+---
+
+# SGLang Mintlify documentation guide for agents
+
+## Non-negotiables
+
+- **Do not guess flags, defaults, or behavior.**
+  If you’re documenting CLI args, env vars, APIs, or performance behavior, verify against:
+  - the upstream codebase (`sgl-project/sglang`)
+  - the current public docs (`docs.sglang.io`) until the migration is complete
+  - or an authoritative vendor doc when platform-specific (ROCm, CANN/Ascend, Intel XPU).
+- **Prefer fixing the docs-site version of an internal link** instead of copying links from older docs.
+- **Keep examples copy/pasteable.** Use placeholders consistently (e.g., `MODEL_PATH`, `HF_TOKEN`, `HOST`, `PORT`).
+
+## Source of truth hierarchy
+
+1. **This repo**
+   - `docs.json` for site structure + navigation
+   - existing MDX pages for voice + conventions
+2. **Canonical current docs**
+   - `docs.sglang.io` (Sphinx site) is currently the reference structure and content baseline.
+3. **Implementation**
+   - `sgl-project/sglang` for anything that can change with releases (flags, env vars, defaults, supported models).
+4. **Cookbook**
+   - `cookbook.sglang.io` / `sgl-project/sgl-cookbook` for recipe patterns and model-specific operational guidance.
+
+## Writing standards (SGLang-specific)
+
+### Voice and structure
+
+* Second person (“you”), active voice.
+* Prefer **short, scannable sections** with clear outcomes.
+* Headings in **sentence case**.
+* Put prerequisites before commands.
+
+### Technical accuracy patterns
+
+For pages that include commands/configs, always specify:
+
+* **Platform** (NVIDIA CUDA / AMD ROCm / Intel XPU / Ascend NPU / CPU)
+* **OS** (if relevant) and **version constraints**
+* **Model identifier** format (e.g., Hugging Face repo id) and where it goes (`--model-path`, `--model`, etc.)
+* **Parallelism knobs** used in the example (`--tp`, `--dp`, node count, etc.)
+* Any required secrets/tokens (`HF_TOKEN`) and where they are used.
+
+# Mintlify best practices
+
+**Always consult [mintlify.com/docs](https://mintlify.com/docs) for components, configuration, and latest features.**
+
+If you are not already connected to the Mintlify MCP server, [https://mintlify.com/docs/mcp](https://mintlify.com/docs/mcp), add it so that you can search more efficiently.
+
+**Always** favor searching the current Mintlify documentation over whatever is in your training data about Mintlify.
+
+Mintlify is a documentation platform that transforms MDX files into documentation sites. Configure site-wide settings in the `docs.json` file, write content in MDX with YAML frontmatter, and favor built-in components over custom components.
+
+Full schema at [mintlify.com/docs.json](https://mintlify.com/docs.json).
+
+## Before you write
+
+### Understand the project
+
+Read `docs.json` in the project root. This file defines the entire site: navigation structure, theme, colors, links, API and specs.
+
+Understanding the project tells you:
+
+* What pages exist and how they're organized
+* What navigation groups are used (and their naming conventions)
+* How the site navigation is structured
+* What theme and configuration the site uses
+
+### Check for existing content
+
+Search the docs before creating new pages. You may need to:
+
+* Update an existing page instead of creating a new one
+* Add a section to an existing page
+* Link to existing content rather than duplicating
+
+### Read surrounding content
+
+Before writing, read 2-3 similar pages to understand the site's voice, structure, formatting conventions, and level of detail.
+
+### Understand Mintlify components
+
+Review the Mintlify [components](https://www.mintlify.com/docs/components) to select and use any relevant components for the documentation request that you are working on.
+
+## Quick reference
+
+### CLI commands
+
+* `npm i -g mint` - Install the Mintlify CLI
+* `mint dev` - Local preview at localhost:3000
+* `mint broken-links` - Check internal links
+* `mint a11y` - Check for accessibility issues in content
+* `mint rename` - Rename/move files and update references
+* `mint validate` - Validate documentation builds
+
+### Required files
+
+* `docs.json` - Site configuration (navigation, theme, integrations, etc.). See [global settings](https://www.mintlify.com/docs/organize/settings) for all options.
+* `*.mdx` files - Documentation pages with YAML frontmatter
+
+### Example file structure
+
+```
+project/
+├── docs.json           # Site configuration
+├── introduction.mdx
+├── quickstart.mdx
+├── guides/
+│   └── example.mdx
+├── openapi.yml         # API specification
+├── images/             # Static assets
+│   └── example.png
+└── snippets/           # Reusable components
+    └── component.jsx
+```
+
+## Page frontmatter
+
+Every page requires `title` in its frontmatter. Include `description` for SEO and navigation.
+
+```yaml theme={null}
+---
+title: "Clear, descriptive title"
+description: "Concise summary for SEO and navigation."
+---
+```
+
+Optional frontmatter fields:
+
+* `sidebarTitle`: Short title for sidebar navigation.
+* `icon`: Lucide or Font Awesome icon name, URL, or file path.
+* `tag`: Label next to the page title in the sidebar (for example, "NEW").
+* `mode`: Page layout mode (`default`, `wide`, `custom`).
+* `keywords`: Array of terms related to the page content for local search and SEO.
+* Any custom YAML fields for use with personalization or conditional content.
+
+## File conventions
+
+* Match existing naming patterns in the directory
+* If there are no existing files or inconsistent file naming patterns, use kebab-case: `getting-started.mdx`, `api-reference.mdx`
+* Use root-relative paths without file extensions for internal links: `/getting-started/quickstart`
+* Do not use relative paths (`../`) or absolute URLs for internal pages
+* When you create a new page, add it to `docs.json` navigation or it won't appear in the sidebar
+
+## Organize content
+
+When a user asks about anything related to site-wide configurations, start by understanding the [global settings](https://www.mintlify.com/docs/organize/settings). See if a setting in the `docs.json` file can be updated to achieve what the user wants.
+
+### Navigation
+
+The `navigation` property in `docs.json` controls site structure. Choose one primary pattern at the root level, then nest others within it.
+
+**Choose your primary pattern:**
+
+| Pattern       | When to use                                                                                    |
+| ------------- | ---------------------------------------------------------------------------------------------- |
+| **Groups**    | Default. Single audience, straightforward hierarchy                                            |
+| **Tabs**      | Distinct sections with different audiences (Guides vs API Reference) or content types          |
+| **Anchors**   | Want persistent section links at sidebar top. Good for separating docs from external resources |
+| **Dropdowns** | Multiple doc sections users switch between, but not distinct enough for tabs                   |
+| **Products**  | Multi-product company with separate documentation per product                                  |
+| **Versions**  | Maintaining docs for multiple API/product versions simultaneously                              |
+| **Languages** | Localized content                                                                              |
+
+**Within your primary pattern:**
+
+* **Groups** - Organize related pages. Can nest groups within groups, but keep hierarchy shallow
+* **Menus** - Add dropdown navigation within tabs for quick jumps to specific pages
+* **`expanded: false`** - Collapse nested groups by default. Use for reference sections users browse selectively
+* **`openapi`** - Auto-generate pages from OpenAPI spec. Add at group/tab level to inherit
+
+**Common combinations:**
+
+* Tabs containing groups (most common for docs with API reference)
+* Products containing tabs (multi-product SaaS)
+* Versions containing tabs (versioned API docs)
+* Anchors containing groups (simple docs with external resource links)
+
+### Links and paths
+
+* **Internal links:** Root-relative, no extension: `/getting-started/quickstart`
+* **Images:** Store in `/images`, reference as `/images/example.png`
+* **External links:** Use full URLs, they open in new tabs automatically
+
+## Customize docs sites
+
+**What to customize where:**
+
+* **Brand colors, fonts, logo** → `docs.json`. See [global settings](https://www.mintlify.com/docs/organize/settings)
+* **Component styling, layout tweaks** → `custom.css` at project root
+* **Dark mode** → Enabled by default. Only disable with `"appearance": "light"` in `docs.json` if brand requires it
+
+Start with `docs.json`. Only add `custom.css` when you need styling that config doesn't support.
+
+## Write content
+
+### Components
+
+The [components overview](https://mintlify.com/docs/components) organizes all components by purpose: structure content, draw attention, show/hide content, document APIs, link to pages, and add visual context. Start there to find the right component.
+
+**Common decision points:**
+
+| Need                       | Use                     |
+| -------------------------- | ----------------------- |
+| Hide optional details      | `<Accordion>`           |
+| Long code examples         | `<Expandable>`          |
+| User chooses one option    | `<Tabs>`                |
+| Linked navigation cards    | `<Card>` in `<Columns>` |
+| Sequential instructions    | `<Steps>`               |
+| Code in multiple languages | `<CodeGroup>`           |
+| API parameters             | `<ParamField>`          |
+| API response fields        | `<ResponseField>`       |
+
+**Callouts by severity:**
+
+* `<Note>` - Supplementary info, safe to skip
+* `<Info>` - Helpful context such as permissions
+* `<Tip>` - Recommendations or best practices
+* `<Warning>` - Potentially destructive actions
+* `<Check>` - Success confirmation
+
+### Reusable content
+
+**When to use snippets:**
+
+* Exact content appears on more than one page
+* Complex components you want to maintain in one place
+* Shared content across teams/repos
+
+**When NOT to use snippets:**
+
+* Slight variations needed per page (leads to complex props)
+
+Import snippets with `import { Component } from "/path/to/snippet-name.jsx"`.
+
+## Writing standards
+
+### Voice and structure
+
+* Second-person voice ("you")
+* Active voice, direct language
+* Sentence case for headings ("Getting started", not "Getting Started")
+* Sentence case for code block titles ("Expandable example", not "Expandable Example")
+* Lead with context: explain what something is before how to use it
+* Prerequisites at the start of procedural content
+
+### What to avoid
+
+**Never use:**
+
+* Marketing language ("powerful", "seamless", "robust", "cutting-edge")
+* Filler phrases ("it's important to note", "in order to")
+* Excessive conjunctions ("moreover", "furthermore", "additionally")
+* Editorializing ("obviously", "simply", "just", "easily")
+
+**Watch for AI-typical patterns:**
+
+* Overly formal or stilted phrasing
+* Unnecessary repetition of concepts
+* Generic introductions that don't add value
+* Concluding summaries that restate what was just said
+
+### Formatting
+
+* All code blocks must have language tags
+* All images and media must have descriptive alt text
+* Use bold and italics only when they serve the reader's understanding--never use text styling just for decoration
+* No decorative formatting or emoji
+
+### Code examples
+
+* Keep examples simple and practical
+* Use realistic values (not "foo" or "bar")
+* One clear example is better than multiple variations
+* Test that code works before including it
+
+## Deploy
+
+Mintlify deploys automatically when changes are pushed to the connected Git repository.
+
+**What agents can configure:**
+
+* **Redirects** → Add to `docs.json` with `"redirects": [{"source": "/old", "destination": "/new"}]`
+* **SEO indexing** → Control with `"seo": {"indexing": "all"}` to include hidden pages in search
+
+**Requires dashboard setup (human task):**
+
+* Custom domains and subdomains
+* Preview deployment settings
+* DNS configuration
+
+For `/docs` subpath hosting with Vercel or Cloudflare, agents can help configure rewrite rules. See [/docs subpath](https://mintlify.com/docs/deploy/vercel).
+
+## Workflow
+
+### 1. Understand the task
+
+Identify what needs to be documented, which pages are affected, and what the reader should accomplish afterward. If any of these are unclear, ask.
+
+### 2. Research
+
+* Read `docs.json` to understand the site structure
+* Search existing docs for related content
+* Read similar pages to match the site's style
+
+### 3. Plan
+
+* Synthesize what the reader should accomplish after reading the docs and the current content
+* Propose any updates or new content
+* Verify that your proposed changes will help readers be successful
+
+### 4. Write
+
+* Start with the most important information
+* Keep sections focused and scannable
+* Use components appropriately (don't overuse them)
+* Mark anything uncertain with a TODO comment:
+
+```mdx theme={null}
+{/* TODO: Verify the default timeout value */}
+```
+
+### 5. Update navigation
+
+If you created a new page, add it to the appropriate group in `docs.json`.
+
+### 6. Verify
+
+Before submitting:
+
+* [ ] Frontmatter includes title and description
+* [ ] All code blocks have language tags
+* [ ] Internal links use root-relative paths without file extensions
+* [ ] New pages are added to `docs.json` navigation
+* [ ] Content matches the style of surrounding pages
+* [ ] No marketing language or filler phrases
+* [ ] TODOs are clearly marked for anything uncertain
+* [ ] Run `mint broken-links` to check links
+* [ ] Run `mint validate` to find any errors
+
+## Edge cases
+
+### Migrations
+
+If a user asks about migrating to Mintlify, ask if they are using ReadMe or Docusaurus. If they are, use the [@mintlify/scraping](https://www.npmjs.com/package/@mintlify/scraping) CLI to migrate content. If they are using a different platform to host their documentation, help them manually convert their content to MDX pages using Mintlify components.
+
+### Hidden pages
+
+Any page that is not included in the `docs.json` navigation is hidden. Use hidden pages for content that should be accessible by URL or indexed for the assistant or search, but not discoverable through the sidebar navigation.
+
+### Exclude pages
+
+The `.mintignore` file is used to exclude files from a documentation repository from being processed.
+
+## Common gotchas
+
+1. **Component imports** - JSX components need explicit import, MDX components don't
+2. **Frontmatter required** - Every MDX file needs `title` at minimum
+3. **Code block language** - Always specify language identifier
+4. **Never use `mint.json`** - `mint.json` is deprecated. Only ever use `docs.json`
+
+## Resources
+
+* [Documentation](https://mintlify.com/docs)
+* [https://github.com/sgl-project/sglang](https://github.com/sgl-project/sglang)
+* [Configuration schema](https://mintlify.com/docs.json)
+* [Feature requests](https://github.com/orgs/mintlify/discussions/categories/feature-requests)
+* [Bugs and feedback](https://github.com/orgs/mintlify/discussions/categories/bugs-feedback)

docs_new/CONTRIBUTING.md

@@ -0,0 +1,34 @@
+> **Customize this file**: Tailor this template to your project by noting specific contribution types you're looking for, adding a Code of Conduct, or adjusting the writing guidelines to match your style.
+
+# Contribute to the documentation
+
+Thank you for your interest in contributing to our documentation! This guide will help you get started.
+
+## How to contribute
+
+### Option 1: Edit directly on GitHub
+
+1. Navigate to the page you want to edit
+2. Click the "Edit this file" button (the pencil icon)
+3. Make your changes and submit a pull request
+
+### Option 2: Local development
+
+1. Fork and clone this repository
+2. Install the Mintlify CLI: `npm i -g mint`
+3. Create a branch for your changes
+4. Make changes
+5. Run `mint dev`
+6. Preview your changes at `http://localhost:3000`
+7. Commit your changes and submit a pull request
+
+For more details on local development, see our [development guide](development.mdx).
+
+## Writing guidelines
+
+- **Use active voice**: "Run the command" not "The command should be run"
+- **Address the reader directly**: Use "you" instead of "the user"
+- **Keep sentences concise**: Aim for one idea per sentence
+- **Lead with the goal**: Start instructions with what the user wants to accomplish
+- **Use consistent terminology**: Don't alternate between synonyms for the same concept
+- **Include examples**: Show, don't just tell

docs_new/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

docs_new/README.md

@@ -0,0 +1,126 @@
+# SGLang Documentation
+
+The official documentation and cookbook for [SGLang](https://github.com/sgl-project/sglang) — a high-performance serving framework for large language models and vision-language models.
+
+- **Docs**: Getting started guides, installation, and reference
+- **Cookbook**: Battle-tested recipes for deploying specific models (Qwen, DeepSeek, Llama, GLM, etc.) on various hardware
+
+
+## Project structure
+
+```
+.
+├── docs.json              # Site configuration (navigation, theme, metadata)
+├── index.mdx              # Homepage
+├── docs/                  # Documentation pages
+│   └── get-started/
+│       └── install.mdx    # Installation guide
+└── cookbook/               # Model deployment recipes
+    ├── intro.mdx           # Cookbook overview and recipe index
+    └── autoregressive/     # Autoregressive model recipes
+        └── Qwen/
+            └── Qwen3.5.mdx
+```
+
+Pages are `.mdx` files with YAML frontmatter. Navigation is defined in `docs.json`.
+
+## Local development
+
+### Prerequisites
+
+- Node.js >= 20
+
+### Setup
+
+```bash
+# Install the CLI
+npm i -g mint
+
+# Start the dev server (with hot reload)
+mint dev
+```
+
+Preview at `http://localhost:3000`.
+
+### Useful commands
+
+```bash
+mint dev            # Start local preview server
+mint broken-links   # Check for broken links
+mint update         # Update the CLI
+```
+
+## Contributing
+
+We welcome contributions! Whether you want to add a recipe for a new model, improve existing docs, or fix a typo — PRs are appreciated.
+
+### Quick edit (GitHub)
+
+1. Navigate to the file you want to edit on GitHub
+2. Click the pencil icon to edit
+3. Submit a pull request
+
+### Local development workflow
+
+```bash
+# 1. Fork and clone the repo
+git clone https://github.com/<YOUR_USERNAME>/sgl-docs.git
+cd sgl-docs
+
+# 2. Create a branch
+git checkout -b my-changes
+
+# 3. Start the dev server and make your changes
+mint dev
+
+# 4. Verify links aren't broken
+mint broken-links
+
+# 5. Commit and push
+git add <files>
+git commit -m "docs: describe your change"
+git push origin my-changes
+
+# 6. Open a pull request on GitHub
+```
+
+### Adding a new cookbook recipe
+
+1. Create a new `.mdx` file under `cookbook/` following the existing directory structure (e.g., `cookbook/llm/<Vendor>/<Model>.mdx` or `cookbook/vlm/<Vendor>/<Model>.mdx`)
+2. Use an existing recipe like `cookbook/llm/Qwen/Qwen3.5.mdx` as a template
+3. Add your page to the navigation in `docs.json`
+4. Each recipe should include:
+   - Model introduction and key specs
+   - Installation / environment setup
+   - Deployment configuration (with hardware recommendations)
+   - Usage examples (basic + advanced)
+   - Benchmarks (if available)
+
+### Writing guidelines
+
+- Use active voice: "Run the command" not "The command should be run"
+- Address the reader as "you"
+- Keep sentences concise — one idea per sentence
+- Lead with the goal, then the steps
+- Use consistent terminology
+- Include concrete examples and code snippets
+
+## Acknowledgements
+
+Thank you to all the authors who contributed to the original documentation in [`sglang/docs/`](https://github.com/sgl-project/sglang/tree/main/docs) and the original cookbook in [`sgl-cookbook`](https://github.com/sgl-project/sgl-cookbook). The migration to the new Mintlify-based documentation was led by the following [ACM_VIT](https://github.com/ACM-VIT) students:
+
+[@Adhyan Jain](https://github.com/Adhyan-Jain), [@Maitri-shah29](https://github.com/Maitri-shah29), [@architnigam](https://github.com/architnigam), [@Nakul-Sinha](https://github.com/Nakul-Sinha), [@divyamagrawal06](https://github.com/divyamagrawal06), [@A-Taman](https://github.com/A-Taman), [@nimeshas](https://github.com/nimeshas), [@IshhanKheria](https://github.com/IshhanKheria), [@Krishang-Zinzuwadia](https://github.com/Krishang-Zinzuwadia), [@pokymono](https://github.com/pokymono), [@Ishitajoshii](https://github.com/Ishitajoshii), [@AdityaVKochar](https://github.com/AdityaVKochar)
+
+Advised by [@adarshxs](https://github.com/adarshxs) (ACM_VIT) and [@wisclmy0611](https://github.com/wisclmy0611), [@Richardczl98](https://github.com/Richardczl98) (LMSYS).
+
+## Community
+
+- [GitHub](https://github.com/sgl-project/sglang)
+- [Slack](https://slack.sglang.io/)
+- [Discord](https://discord.gg/4ugb2t6YY2)
+- [X / Twitter](https://x.com/lmsysorg)
+- [LinkedIn](https://www.linkedin.com/company/sgl-project/)
+
+## License
+
+Apache License 2.0 — see the [LICENSE](LICENSE) for details.

docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-Math-V2.mdx

@@ -0,0 +1,522 @@
+---
+title: DeepSeek-Math-V2
+metatags:
+    description: "Deploy DeepSeek-Math-V2 with SGLang - advanced mathematical reasoning model with gold-level IMO/CMO performance and theorem-proving capabilities."
+---
+
+import { DeepSeekMathV2Deployment } from '/src/snippets/autoregressive/deepseek-math-v2-deployment.jsx';
+
+## 1. Model Introduction
+
+[DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
+
+**Key Features:**
+
+- **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024
+- **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy
+- **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024
+- **Large MoE Model**: ~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB)
+
+**Available Models:**
+
+- **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights
+
+**License:**
+To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details.
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
+
+<DeepSeekMathV2Deployment />
+
+### 3.2 Configuration Tips
+
+**Hardware Requirements:**
+
+- **B200 (183GB)**: BF16 tp=8
+- **B300 (275GB)**: BF16 tp=8
+
+**DP Attention:**
+
+- Enable DP attention for high-throughput scenarios
+- The `--dp` value commonly matches the `--tp` value
+- Trade-off: Higher throughput at the cost of slightly increased latency
+
+## 4. Model Invocation
+
+### 4.1 Deployment Command
+
+Deploy the model using the command generated above. Example for B200:
+
+```shell Command
+sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \
+  --tp 8 \
+  --ep 8 \
+  --reasoning-parser deepseek-r1 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.2 Mathematical Reasoning
+
+DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning.
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Mathematical reasoning problem
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-Math-V2",
+    messages=[
+        {"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"}
+    ],
+    max_tokens=4096,
+    stream=True
+)
+
+# Process the stream
+thinking_started = False
+has_thinking = False
+has_answer = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2.
+
+This is a classic formula for the sum of the first n natural numbers. We can prove by induction.
+
+Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds.
+
+Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1
+)((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n.
+
+...
+=============== Content =================
+We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below.
+
+---
+
+### 1. Proof by mathematical induction
+
+**Base case (\(n=1\))**:
+\[
+1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1,
+\]
+so the formula holds for \(n=1\).
+
+**Inductive hypothesis:**
+Assume that for some positive integer \(k\) the formula is true, i.e.
+\[
+1+2+\dots+k = \frac{k(k+1)}{2}.
+\]
+
+**Inductive step (\(k \to k+1\))**:
+Consider the sum up to \(k+1\):
+\[
+\begin{aligned}
+1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt]
+&= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt]
+&= (k+1)\left(\frac{k}{2}+1\right)\\[4pt]
+&= (k+1)\frac{k+2}{2}\\[4pt]
+&= \frac{(k+1)(k+2)}{2}\\[4pt]
+&= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}.
+\end{aligned}
+\]
+Thus the formula also holds for \(n=k+1\).
+
+By the principle of mathematical induction,
+\[
+1+2+3+\dots+n = \frac{n(n+1)}{2}
+\]
+for every positive integer \(n\).
+
+---
+
+### 2. Proof by pairing (Gauss’s trick)
+
+Let
+\[
+S = 1 + 2 + 3 + \dots + n.
+\]
+
+Write the same sum in reverse order:
+\[
+S = n + (n-1) + (n-2) + \dots + 1.
+\]
+
+Add the two equalities term‑by‑term:
+\[
+\begin{aligned}
+2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\
+    &= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\
+    &= n\,(n+1).
+\end{aligned}
+\]
+
+Therefore
+\[
+S = \frac{n(n+1)}{2}.
+\]
+
+Both proofs are rigorous and show that the formula holds for all positive integers \(n\).
+```
+
+### 4.3 Competition-Level Problems
+
+**Example: IMO-style Problem:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# IMO-style problem
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-Math-V2",
+    messages=[
+        {"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."}
+    ],
+    max_tokens=8192,
+    stream=True
+)
+
+# Process the stream
+thinking_started = False
+has_thinking = False
+has_answer = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        if delta.content:
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+We need to prove that for positive real numbers a,b,c with abc = 1, we have:
+
+\[
+(a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1.
+\]
+
+We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite:
+
+...
+=============== Content =================
+
+We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove
+
+\[
+\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
+\]
+
+---
+
+### 1.  A convenient substitution
+
+Because \(abc=1\), we can write
+
+\[
+a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x}
+\]
+
+with positive numbers \(x,y,z\).
+(For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).)
+
+---
+
+### 2.  Rewriting the factors
+
+\[
+\begin{aligned}
+a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm]
+b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm]
+c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}.
+\end{aligned}
+\]
+
+Hence the product becomes
+
+\[
+P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)
+   =\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}.
+\]
+
+---
+
+### 3.  Reducing to a known inequality
+
+We have to show \(P\le1\), i.e.
+
+\[
+(x+z-y)(x+y-z)(y+z-x)\le xyz .
+\tag{1}
+\]
+
+Set
+
+\[
+p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz .
+\]
+
+Notice that
+
+\[
+x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x .
+\]
+
+Therefore
+
+\[
+\begin{aligned}
+(x+z-y)(x+y-z)(y+z-x)
+&=(p-2x)(p-2y)(p-2z)\\
+&=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\
+&=-p^{3}+4pq-8r .
+\end{aligned}
+\]
+
+Inequality (1) is thus equivalent to
+
+\[
+-p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r .
+\tag{2}
+\]
+
+---
+
+### 4.  Applying Schur’s inequality
+
+Schur’s inequality of third degree states that for any non‑negative \(x,y,z\)
+
+\[
+p^{3}+9r\ge 4pq .
+\]
+
+Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2).
+Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds.
+
+Consequently (1) is true, and we obtain \(P\le1\).
+
+---
+
+### 5.  Equality case
+
+Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\).
+Then \(a=b=c=1\), and indeed the product equals \(1\).
+
+---
+
+Thus for all positive \(a,b,c\) with \(abc=1\),
+
+\[
+\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
+\]
+
+∎
+```
+
+## 5. Benchmark
+
+### 5.1 Accuracy Benchmark
+
+#### 5.1.1 GSM8K Benchmark
+
+**Benchmark Command:**
+
+```shell Command
+python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --port 30000
+```
+
+**Test Results:**
+
+```text Output
+Accuracy: 0.975
+Invalid: 0.000
+Latency: 34.358 s
+Output throughput: 540.162 token/s
+```
+
+### 5.2 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (8x, 183GB each)
+- Model: DeepSeek-Math-V2
+- Tensor Parallelism: 8
+- SGLang Version: 0.5.8
+
+#### 5.2.1 Latency Benchmark
+
+**Benchmark Command:**
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-Math-V2 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+**Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  53.34
+Total input tokens:                      1972
+Total input text tokens:                 1972
+Total generated tokens:                  2784
+Total generated tokens (retokenized):    2778
+Request throughput (req/s):              0.19
+Input token throughput (tok/s):          36.97
+Output token throughput (tok/s):         52.19
+Peak output token throughput (tok/s):    56.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          89.16
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5330.72
+Median E2E Latency (ms):                 5879.28
+P90 E2E Latency (ms):                    8320.33
+P99 E2E Latency (ms):                    9921.29
+---------------Time to First Token----------------
+Mean TTFT (ms):                          183.38
+Median TTFT (ms):                        177.92
+P99 TTFT (ms):                           217.64
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          17.96
+Median TPOT (ms):                        18.39
+P99 TPOT (ms):                           19.03
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           18.57
+Median ITL (ms):                         18.63
+P95 ITL (ms):                            19.26
+P99 ITL (ms):                            19.48
+Max ITL (ms):                            24.93
+==================================================
+```
+
+#### 5.2.2 Throughput Benchmark
+
+**Benchmark Command:**
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-Math-V2 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+**Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  217.36
+Total input tokens:                      301701
+Total input text tokens:                 301701
+Total generated tokens:                  188375
+Total generated tokens (retokenized):    187456
+Request throughput (req/s):              4.60
+Input token throughput (tok/s):          1388.05
+Output token throughput (tok/s):         866.67
+Peak output token throughput (tok/s):    2589.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          2254.72
+Concurrency:                             89.81
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   19521.73
+Median E2E Latency (ms):                 12076.76
+P90 E2E Latency (ms):                    47248.87
+P99 E2E Latency (ms):                    86862.79
+---------------Time to First Token----------------
+Mean TTFT (ms):                          790.40
+Median TTFT (ms):                        456.81
+P99 TTFT (ms):                           4223.33
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          106.52
+Median TPOT (ms):                        107.24
+P99 TPOT (ms):                           238.33
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           100.29
+Median ITL (ms):                         38.34
+P95 ITL (ms):                            237.00
+P99 ITL (ms):                            347.49
+Max ITL (ms):                            3642.56
+==================================================
+```

docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx

@@ -0,0 +1,250 @@
+---
+title: DeepSeek-OCR-2
+metatags:
+    description: "Deploy DeepSeek-OCR-2 with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
+---
+
+import { DeepSeekOCR2Deployment } from '/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx';
+
+## 1. Model Introduction
+
+[DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats.
+
+**Key Features:**
+
+- **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents).
+- **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version.
+- **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks.
+- **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR.
+
+**Available Models:**
+
+- **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks
+
+**License:**
+To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details.
+
+For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2).
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, and AMD MI300X, MI355X, and MI325X GPUs.
+
+<DeepSeekOCR2Deployment />
+
+**Note**: DeepSeek-OCR-2 has ~3.58B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details.
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+**OpenAI-compatible request example**
+
+```python Example
+import requests
+
+url = "http://localhost:30000/v1/chat/completions"
+
+data = {
+    "model": "deepseek-ai/DeepSeek-OCR-2",
+    "messages": [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "<image>\n<|grounding|>Convert the document to markdown."},
+                {"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}},
+            ],
+        }
+    ],
+    "max_tokens": 512,
+}
+
+response = requests.post(url, json=data)
+print(response.text)
+```
+
+**Reference**
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Recommended Prompts
+
+The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts).
+
+**Structured document conversion** — extracts text while preserving layout:
+
+```text Example
+<image>
+<|grounding|>Convert the document to markdown.
+```
+
+**Free-form OCR** — extracts without layouts:
+
+```text Example
+<image>
+Free OCR.
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA H200 GPU (1x)
+- Model: DeepSeek-OCR-2
+- Tensor Parallelism: 1
+- sglang version: 0.0.0.dev1+g93fca0bbc
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models).
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+sglang serve \
+  --model-path deepseek-ai/DeepSeek-OCR-2 \
+  --enable-multimodal \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-OCR-2 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  3.54
+Total input tokens:                      1972
+Total input text tokens:                 1972
+Total generated tokens:                  2784
+Total generated tokens (retokenized):    2710
+Request throughput (req/s):              2.83
+Input token throughput (tok/s):          557.53
+Output token throughput (tok/s):         787.10
+Peak output token throughput (tok/s):    818.00
+Peak concurrent requests:                5
+Total token throughput (tok/s):          1344.63
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   352.69
+Median E2E Latency (ms):                 392.34
+P90 E2E Latency (ms):                    540.64
+P99 E2E Latency (ms):                    639.01
+---------------Time to First Token----------------
+Mean TTFT (ms):                          18.08
+Median TTFT (ms):                        16.57
+P99 TTFT (ms):                           25.67
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          1.18
+Median TPOT (ms):                        1.21
+P99 TPOT (ms):                           1.22
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           1.21
+Median ITL (ms):                         1.21
+P95 ITL (ms):                            1.28
+P99 ITL (ms):                            1.44
+Max ITL (ms):                            4.32
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+sglang serve \
+  --model-path deepseek-ai/DeepSeek-OCR-2 \
+  --enable-multimodal \
+  --tp 1 \
+  --ep 1 \
+  --dp 1 \
+  --enable-dp-attention \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-OCR-2 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  14.79
+Total input tokens:                      301698
+Total input text tokens:                 301698
+Total generated tokens:                  188375
+Total generated tokens (retokenized):    185236
+Request throughput (req/s):              67.63
+Input token throughput (tok/s):          20402.54
+Output token throughput (tok/s):         12738.99
+Peak output token throughput (tok/s):    17508.00
+Peak concurrent requests:                187
+Total token throughput (tok/s):          33141.53
+Concurrency:                             86.87
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   1284.50
+Median E2E Latency (ms):                 866.07
+P90 E2E Latency (ms):                    3027.32
+P99 E2E Latency (ms):                    5490.63
+---------------Time to First Token----------------
+Mean TTFT (ms):                          86.08
+Median TTFT (ms):                        50.09
+P99 TTFT (ms):                           613.92
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          7.79
+Median TPOT (ms):                        6.54
+P99 TPOT (ms):                           50.10
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.42
+Median ITL (ms):                         4.64
+P95 ITL (ms):                            23.65
+P99 ITL (ms):                            39.62
+Max ITL (ms):                            452.65
+==================================================
+```

docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-OCR.mdx

@@ -0,0 +1,204 @@
+---
+title: DeepSeek-OCR
+metatags:
+    description: "Deploy DeepSeek-OCR with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
+---
+
+## 1. Model Introduction
+
+[DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks.
+
+**Key Features:**
+
+- **Advanced OCR**: High-accuracy text recognition from images and documents
+- **Multi-Modality**: Supports various image formats and document types
+
+**Available Models:**
+
+- **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks
+
+**License:**
+To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details.
+
+For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR).
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
+
+import { DeepSeekOCRDeployment } from "/src/snippets/autoregressive/deepseek-ocr-deployment.jsx";
+
+<DeepSeekOCRDeployment />
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X GPU (1x)
+- Model: DeepSeek-OCR
+- Tensor Parallelism: 1
+- sglang version: 0.5.7
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-OCR \
+  --tp 1 \
+  --dtype float16 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model deepseek-ai/DeepSeek-OCR \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  4.45
+Total input tokens:                      1972
+Total input text tokens:                 1972
+Total input vision tokens:               0
+Total generated tokens:                  2784
+Total generated tokens (retokenized):    2770
+Request throughput (req/s):              2.25
+Input token throughput (tok/s):          442.89
+Output token throughput (tok/s):         625.26
+Peak output token throughput (tok/s):    635.00
+Peak concurrent requests:                4
+Total token throughput (tok/s):          1068.16
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   443.32
+Median E2E Latency (ms):                 493.29
+---------------Time to First Token----------------
+Mean TTFT (ms):                          21.59
+Median TTFT (ms):                        20.89
+P99 TTFT (ms):                           24.81
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          1.47
+Median TPOT (ms):                        1.52
+P99 TPOT (ms):                           1.53
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           1.52
+Median ITL (ms):                         1.51
+P95 ITL (ms):                            1.76
+P99 ITL (ms):                            1.93
+Max ITL (ms):                            8.28
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-OCR \
+  --tp 1 \
+  --ep 1 \
+  --dp 1 \
+  --enable-dp-attention \
+  --dtype float16 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model deepseek-ai/DeepSeek-OCR \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  16.24
+Total input tokens:                      301698
+Total input text tokens:                 301698
+Total input vision tokens:               0
+Total generated tokens:                  188375
+Total generated tokens (retokenized):    186927
+Request throughput (req/s):              61.59
+Input token throughput (tok/s):          18582.90
+Output token throughput (tok/s):         11602.84
+Peak output token throughput (tok/s):    15479.00
+Peak concurrent requests:                179
+Total token throughput (tok/s):          30185.75
+Concurrency:                             85.53
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   1388.60
+Median E2E Latency (ms):                 901.43
+---------------Time to First Token----------------
+Mean TTFT (ms):                          73.36
+Median TTFT (ms):                        50.21
+P99 TTFT (ms):                           349.53
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          7.42
+Median TPOT (ms):                        7.31
+P99 TPOT (ms):                           27.99
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           7.04
+Median ITL (ms):                         4.62
+P95 ITL (ms):                            21.11
+P99 ITL (ms):                            36.92
+Max ITL (ms):                            172.15
+==================================================
+```

docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx

@@ -0,0 +1,910 @@
+---
+title: DeepSeek-R1
+metatags:
+    description: "Deploy DeepSeek-R1 reasoning model with SGLang - advanced step-by-step reasoning with FP8/FP4 quantization for NVIDIA and AMD GPUs."
+---
+
+import { DeepSeekR1BasicDeployment } from '/src/snippets/autoregressive/deepseek-r1-basic-deployment.jsx';
+import { DeepSeekR1AdvancedDeployment } from '/src/snippets/autoregressive/deepseek-r1-advanced-deployment.jsx';
+
+## 1. Model Introduction
+
+[DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms.
+
+**Key Features:**
+
+- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
+- **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
+- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, and AMD MI300X, MI325X and MI355X GPUs
+- **High Performance**: Optimized for both throughput and latency scenarios
+
+**Available Models:**
+
+- **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X
+- **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X
+
+**License:**
+To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details.
+
+For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1).
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy.
+
+<DeepSeekR1BasicDeployment />
+
+### 3.2 Optimal Configurations
+
+Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware.
+
+<DeepSeekR1AdvancedDeployment />
+
+### 3.3 Configuration Tips
+
+For more detailed configuration tips and advanced tuning, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-R1-0528 \
+  --reasoning-parser deepseek-r1 \
+  --tp 8
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-0528",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To solve this problem, I need to calculate 15% of 240.
+Step 1: Convert 15% to decimal: 15% = 0.15
+Step 2: Multiply 240 by 0.15
+Step 3: 240 × 0.15 = 36
+=============== Content =================
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-R1-0528 \
+  --reasoning-parser deepseek-r1 \
+  --tool-call-parser deepseekv3 \
+  --chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \
+  --tp 8
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-0528",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"🔧 Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+🔧 Tool Call: get_weather
+   Arguments:
+🔧 Tool Call: None
+   Arguments: {"location": "Beijing"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-R1-0528",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: B200 GPU (8x)
+- Model: DeepSeek-R1-0528
+- Tensor Parallelism: 8
+- SGLang Version: 0.5.6.post1
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Test Scenarios
+
+Three core scenarios reflect real-world usage patterns:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 Concurrency Levels
+
+Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
+
+- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
+- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
+- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
+
+#### 5.1.3 Number of Prompts
+
+For each concurrency level, configure `num_prompts` to simulate realistic user loads:
+
+- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
+- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
+- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
+
+---
+
+#### 5.1.4 Benchmark Commands
+
+**Scenario 1: Chat (1K/1K) - Most Important**
+
+- **Model Deployment**
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-R1-0528 \
+  --tp 8
+```
+
+- Low Concurrency (Latency-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  40.00
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4210
+Total generated tokens (retokenized):    4205
+Request throughput (req/s):              0.25
+Input token throughput (tok/s):          152.52
+Output token throughput (tok/s):         105.24
+Peak output token throughput (tok/s):    110.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          257.76
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3998.40
+Median E2E Latency (ms):                 3207.53
+---------------Time to First Token----------------
+Mean TTFT (ms):                          153.00
+Median TTFT (ms):                        140.76
+P99 TTFT (ms):                           214.66
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          9.16
+Median TPOT (ms):                        9.15
+P99 TPOT (ms):                           9.21
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           9.16
+Median ITL (ms):                         9.15
+P95 ITL (ms):                            9.47
+P99 ITL (ms):                            9.63
+Max ITL (ms):                            15.45
+==================================================
+```
+
+- Medium Concurrency (Balanced)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  51.21
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40725
+Total generated tokens (retokenized):    40458
+Request throughput (req/s):              1.56
+Input token throughput (tok/s):          774.66
+Output token throughput (tok/s):         795.30
+Peak output token throughput (tok/s):    1088.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          1569.96
+Concurrency:                             13.93
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   8918.33
+Median E2E Latency (ms):                 9466.16
+---------------Time to First Token----------------
+Mean TTFT (ms):                          273.51
+Median TTFT (ms):                        131.71
+P99 TTFT (ms):                           839.57
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          17.56
+Median TPOT (ms):                        17.46
+P99 TPOT (ms):                           28.68
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           17.02
+Median ITL (ms):                         14.70
+P95 ITL (ms):                            16.41
+P99 ITL (ms):                            112.38
+Max ITL (ms):                            461.90
+==================================================
+```
+
+- High Concurrency (Throughput-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  110.46
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252162
+Total generated tokens (retokenized):    251441
+Request throughput (req/s):              4.53
+Input token throughput (tok/s):          2261.80
+Output token throughput (tok/s):         2282.90
+Peak output token throughput (tok/s):    3900.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          4544.71
+Concurrency:                             92.26
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   20380.71
+Median E2E Latency (ms):                 19391.65
+---------------Time to First Token----------------
+Mean TTFT (ms):                          563.14
+Median TTFT (ms):                        147.62
+P99 TTFT (ms):                           2632.11
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          40.11
+Median TPOT (ms):                        41.98
+P99 TPOT (ms):                           50.10
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           39.37
+Median ITL (ms):                         26.36
+P95 ITL (ms):                            98.16
+P99 ITL (ms):                            150.08
+Max ITL (ms):                            2052.85
+==================================================
+```
+
+**Scenario 2: Reasoning (1K/8K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  411.34
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  44452
+Total generated tokens (retokenized):    44390
+Request throughput (req/s):              0.02
+Input token throughput (tok/s):          14.83
+Output token throughput (tok/s):         108.07
+Peak output token throughput (tok/s):    110.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          122.90
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   41132.04
+Median E2E Latency (ms):                 44288.71
+---------------Time to First Token----------------
+Mean TTFT (ms):                          125.76
+Median TTFT (ms):                        126.19
+P99 TTFT (ms):                           137.69
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          9.21
+Median TPOT (ms):                        9.20
+P99 TPOT (ms):                           9.27
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           9.23
+Median ITL (ms):                         9.22
+P95 ITL (ms):                            9.64
+P99 ITL (ms):                            9.86
+Max ITL (ms):                            15.18
+==================================================
+```
+
+- Medium Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  348.93
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  318226
+Total generated tokens (retokenized):    317630
+Request throughput (req/s):              0.23
+Input token throughput (tok/s):          113.69
+Output token throughput (tok/s):         912.02
+Peak output token throughput (tok/s):    1088.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          1025.70
+Concurrency:                             14.07
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   61360.70
+Median E2E Latency (ms):                 62071.20
+---------------Time to First Token----------------
+Mean TTFT (ms):                          176.02
+Median TTFT (ms):                        153.75
+P99 TTFT (ms):                           268.44
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          15.42
+Median TPOT (ms):                        15.59
+P99 TPOT (ms):                           16.07
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           15.39
+Median ITL (ms):                         15.17
+P95 ITL (ms):                            16.62
+P99 ITL (ms):                            18.13
+Max ITL (ms):                            226.59
+==================================================
+```
+
+- High Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  589.31
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total input vision tokens:               0
+Total generated tokens:                  1300705
+Total generated tokens (retokenized):    1297658
+Request throughput (req/s):              0.54
+Input token throughput (tok/s):          269.70
+Output token throughput (tok/s):         2207.16
+Peak output token throughput (tok/s):    2944.00
+Peak concurrent requests:                68
+Total token throughput (tok/s):          2476.86
+Concurrency:                             57.03
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   105032.36
+Median E2E Latency (ms):                 108229.09
+---------------Time to First Token----------------
+Mean TTFT (ms):                          223.91
+Median TTFT (ms):                        158.15
+P99 TTFT (ms):                           474.86
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          25.94
+Median TPOT (ms):                        26.72
+P99 TPOT (ms):                           27.99
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           25.79
+Median ITL (ms):                         25.37
+P95 ITL (ms):                            26.70
+P99 ITL (ms):                            105.49
+Max ITL (ms):                            237.91
+==================================================
+```
+
+**Scenario 3: Summarization (8K/1K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  40.65
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total input vision tokens:               0
+Total generated tokens:                  4210
+Total generated tokens (retokenized):    4195
+Request throughput (req/s):              0.25
+Input token throughput (tok/s):          1031.65
+Output token throughput (tok/s):         103.56
+Peak output token throughput (tok/s):    110.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          1135.20
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   4063.62
+Median E2E Latency (ms):                 3296.13
+---------------Time to First Token----------------
+Mean TTFT (ms):                          165.91
+Median TTFT (ms):                        154.96
+P99 TTFT (ms):                           240.92
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          9.26
+Median TPOT (ms):                        9.27
+P99 TPOT (ms):                           9.42
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           9.28
+Median ITL (ms):                         9.28
+P95 ITL (ms):                            9.66
+P99 ITL (ms):                            9.83
+Max ITL (ms):                            14.06
+==================================================
+```
+
+- Medium Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  56.71
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total input vision tokens:               0
+Total generated tokens:                  41589
+Total generated tokens (retokenized):    41490
+Request throughput (req/s):              1.41
+Input token throughput (tok/s):          5290.75
+Output token throughput (tok/s):         733.41
+Peak output token throughput (tok/s):    1024.00
+Peak concurrent requests:                20
+Total token throughput (tok/s):          6024.16
+Concurrency:                             14.25
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   10098.99
+Median E2E Latency (ms):                 10623.46
+---------------Time to First Token----------------
+Mean TTFT (ms):                          486.80
+Median TTFT (ms):                        189.59
+P99 TTFT (ms):                           2138.73
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          19.06
+Median TPOT (ms):                        19.23
+P99 TPOT (ms):                           30.69
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           18.53
+Median ITL (ms):                         15.63
+P95 ITL (ms):                            16.64
+P99 ITL (ms):                            109.71
+Max ITL (ms):                            1471.36
+==================================================
+```
+
+- High Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-R1-0528 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  115.55
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total input vision tokens:               0
+Total generated tokens:                  169680
+Total generated tokens (retokenized):    169275
+Request throughput (req/s):              2.77
+Input token throughput (tok/s):          11024.93
+Output token throughput (tok/s):         1468.50
+Peak output token throughput (tok/s):    2254.00
+Peak concurrent requests:                70
+Total token throughput (tok/s):          12493.43
+Concurrency:                             59.45
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   21465.98
+Median E2E Latency (ms):                 20686.26
+---------------Time to First Token----------------
+Mean TTFT (ms):                          913.93
+Median TTFT (ms):                        224.92
+P99 TTFT (ms):                           6257.83
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          39.93
+Median TPOT (ms):                        40.99
+P99 TPOT (ms):                           60.91
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           38.83
+Median ITL (ms):                         26.29
+P95 ITL (ms):                            113.81
+P99 ITL (ms):                            176.94
+Max ITL (ms):                            5521.53
+==================================================
+```
+
+#### 5.1.5 Understanding the Results
+
+**Key Metrics:**
+
+- **Request Throughput (req/s)**: Number of requests processed per second
+- **Output Token Throughput (tok/s)**: Total tokens generated per second
+- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
+- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
+- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
+
+**Why These Configurations Matter:**
+
+- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
+- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
+- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
+- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
+
+**Interpreting Results:**
+
+- Compare your results against baseline numbers for your hardware
+- Higher throughput at same latency = better performance
+- Lower TTFT = more responsive user experience
+- Lower TPOT = faster generation speed
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py \
+  --num-shots 8 \
+  --num-questions 1316 \
+  --parallel 1316
+```
+
+**Test Results:**
+
+```text Output
+Accuracy: 0.959
+Invalid: 0.000
+Latency: 29.185 s
+Output throughput: 4854.672 token/s
+```

docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx

@@ -0,0 +1,520 @@
+---
+title: "DeepSeek-V3"
+metatags:
+    description: "Deploy DeepSeek-V3 MoE model with SGLang - efficient architecture with strong reasoning, coding, and tool-augmented capabilities."
+---
+
+
+## 1. Model Introduction
+
+[DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost.
+
+Key highlights include:
+
+- **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable.
+- **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies.
+- **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively.
+- **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
+
+import { DeepSeekV3Deployment } from "/src/snippets/autoregressive/deepseek-v3-deployment.jsx";
+
+<DeepSeekV3Deployment />
+
+### 3.2 Configuration Tips
+For more detailed configuration tips, please refer to [DeepSeek-V3 Usage](../../../docs/basic_usage/deepseek_v3).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [Basic API Usage](../../../docs/get-started/quickstart)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model deepseek-ai/DeepSeek-V3 \
+  --reasoning-parser deepseek-v3 \
+  --tp 8
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    extra_body = {"chat_template_kwargs": {"thinking": True}},
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To determine 15% of a number, follow these steps:
+
+**Step 1: Understand the Problem**
+You need to find 15% of a given number. Let's assume the number is 240 for this example.
+
+**Step 2: Convert the Percentage to a Decimal**
+To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100.
+
+\[ 15\% = \frac{15}{100} = 0.15 \]
+
+**Step 3: Multiply the Decimal by the Number**
+Now, multiply the decimal form of the percentage by the number you want to find the percentage of.
+
+\[ 0.15 \times 240 \]
+
+**Step 4: Perform the Multiplication**
+Calculate the product:
+
+\[ 0.15 \times 240 = 36 \]
+
+**Step 5: Conclusion**
+Therefore, 15% of 240 is:
+
+\boxed{36}
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser:
+
+**Deployment Command:**
+
+```shell Command
+python -m sglang.launch_server \
+  --model deepseek-ai/DeepSeek-V3 \
+  --tool-call-parser deepseekv3 \
+  --reasoning-parser deepseek-v3 \
+  --chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    extra_body = {"chat_template_kwargs": {"thinking": True}},
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather
+```json
+{"location": "Beijing", "unit": "celsius"}
+```<｜tool▁call▁end｜><｜tool▁calls▁end｜>
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+Please attach the code blocks below to the previous Python script.
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X GPU (8x)
+- Model: DeepSeek-V3
+- Tensor Parallelism: 8
+- sglang version: 0.5.7
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3 \
+  --tp 8 \
+  --dp 8 \
+  --enable-dp-attention \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model deepseek-ai/DeepSeek-V3 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  81.27
+Total input tokens:                      1972
+Total input text tokens:                 1972
+Total input vision tokens:               0
+Total generated tokens:                  2784
+Total generated tokens (retokenized):    2774
+Request throughput (req/s):              0.12
+Input token throughput (tok/s):          24.27
+Output token throughput (tok/s):         34.26
+Peak output token throughput (tok/s):    65.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          58.52
+Concurrency:                             1.00
+Accept length:                           2.61
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   8123.17
+Median E2E Latency (ms):                 7982.65
+---------------Time to First Token----------------
+Mean TTFT (ms):                          1080.76
+Median TTFT (ms):                        1248.82
+P99 TTFT (ms):                           1896.37
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          25.04
+Median TPOT (ms):                        24.76
+P99 TPOT (ms):                           32.09
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           25.41
+Median ITL (ms):                         20.14
+P95 ITL (ms):                            60.28
+P99 ITL (ms):                            60.99
+Max ITL (ms):                            61.49
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3 \
+  --tp 8 \
+  --ep 8 \
+  --dp 8 \
+  --enable-dp-attention \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model deepseek-ai/DeepSeek-V3 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  406.16
+Total input tokens:                      301701
+Total input text tokens:                 301701
+Total input vision tokens:               0
+Total generated tokens:                  188375
+Total generated tokens (retokenized):    187542
+Request throughput (req/s):              2.46
+Input token throughput (tok/s):          742.81
+Output token throughput (tok/s):         463.80
+Peak output token throughput (tok/s):    1299.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          1206.61
+Concurrency:                             87.53
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   35552.98
+Median E2E Latency (ms):                 21466.07
+---------------Time to First Token----------------
+Mean TTFT (ms):                          1521.51
+Median TTFT (ms):                        476.80
+P99 TTFT (ms):                           8329.50
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          214.73
+Median TPOT (ms):                        152.00
+P99 TPOT (ms):                           1155.85
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           182.10
+Median ITL (ms):                         79.18
+P95 ITL (ms):                            398.60
+P99 ITL (ms):                            1488.96
+Max ITL (ms):                            43465.60
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
+```
+
+- **Test Results**:
+  - DeepSeek-V3
+    ```text Output
+    Accuracy: 0.960
+    Invalid: 0.000
+    Latency: 32.450 s
+    Output throughput: 614.211 token/s
+    ```
+
+#### 5.2.2 MMLU Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+cd sglang
+bash benchmark/mmlu/download_data.sh
+python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 8000
+```
+
+- **Test Results**:
+  - DeepSeek-V3
+    ```text Output
+    subject: abstract_algebra, #q:100, acc: 0.800
+    subject: anatomy, #q:135, acc: 0.874
+    subject: astronomy, #q:152, acc: 0.928
+    subject: business_ethics, #q:100, acc: 0.880
+    subject: clinical_knowledge, #q:265, acc: 0.928
+    subject: college_biology, #q:144, acc: 0.965
+    subject: college_chemistry, #q:100, acc: 0.670
+    subject: college_computer_science, #q:100, acc: 0.840
+    subject: college_mathematics, #q:100, acc: 0.800
+    subject: college_medicine, #q:173, acc: 0.861
+    Total latency: 58.339
+    Average accuracy: 0.871
+    ```

docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx

@@ -0,0 +1,941 @@
+---
+title: DeepSeek-V3.1
+metatags:
+    description: "Deploy DeepSeek-V3.1 MoE model with SGLang - hybrid reasoning, improved tool calling, and agentic behavior for complex multi-step tasks."
+---
+
+## 1. Model Introduction
+
+[DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture.
+
+**[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases.
+
+
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
+
+import { DeepSeekV31Deployment } from "/src/snippets/autoregressive/deepseek-v31-deployment.jsx";
+
+<DeepSeekV31Deployment />
+
+### 3.2 Configuration Tips
+For more detailed configuration tips, please refer to [DeepSeek V3/V3.1/R1 Usage](../../../docs/basic_usage/deepseek_v3).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [Basic API Usage](../../../docs/get-started/quickstart)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model deepseek-ai/DeepSeek-V3.1-Terminus \
+  --reasoning-parser deepseek-v3 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3.1-Terminus",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    extra_body = {"chat_template_kwargs": {"thinking": True}},
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100.
+
+To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15.
+
+Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240.
+
+I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36.
+
+I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer.
+
+So, 15% of 240 is 36.
+
+The problem says "step by step," so I should present it clearly.
+=============== Content =================
+To find 15% of 240, follow these steps:
+
+1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \).
+2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \).
+3. Multiply the decimal by 240: \( 0.15 \times 240 \).
+4. Perform the multiplication:
+   - \( 0.15 \times 200 = 30 \)
+   - \( 0.15 \times 40 = 6 \)
+   - Add the results: \( 30 + 6 = 36 \).
+
+Alternatively, you can find 15% by breaking it into parts:
+- 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \).
+- 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \).
+- Add 10% and 5%: \( 24 + 12 = 36 \).
+
+Thus, 15% of 240 is 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser:
+
+**Note:** DeepSeek-V3.1-Speciale does **NOT** support tool calling. It is designed exclusively for deep reasoning tasks.
+
+**Deployment Command:**
+
+```shell Command
+python -m sglang.launch_server \
+  --model deepseek-ai/DeepSeek-V3.1-Terminus \
+  --tool-call-parser deepseekv31 \
+  --reasoning-parser deepseek-v3 \
+  --chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well.
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3.1-Terminus",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    extra_body = {"chat_template_kwargs": {"thinking": True}},
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide.
+
+I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world.
+
+The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you.
+=============== Content =================
+
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+Please attach the code blocks below to the previous Python script.
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3.1-Terminus",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "Currently, it is **22°C and sunny** in Beijing."
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X GPU (8x)
+- Model: DeepSeek-V3.1-Terminus
+- Tensor Parallelism: 8
+- sglang version: 0.5.7
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Test Scenarios
+
+Three core scenarios reflect real-world usage patterns:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 Concurrency Levels
+
+Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
+
+- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
+- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
+- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
+
+#### 5.1.3 Number of Prompts
+
+For each concurrency level, configure `num_prompts` to simulate realistic user loads:
+
+- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
+- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
+- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
+
+---
+
+#### 5.1.4 Benchmark Commands
+
+**Scenario 1: Chat (1K/1K) - Most Important**
+
+- **Model Deployment**
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path deepseek-ai/DeepSeek-V3.1 \
+  --tp 8
+```
+
+- Low Concurrency (Latency-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  106.24
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4201
+Request throughput (req/s):              0.09
+Input token throughput (tok/s):          57.43
+Output token throughput (tok/s):         39.72
+Peak output token throughput (tok/s):    43.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          97.15
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   10620.29
+Median E2E Latency (ms):                 8868.09
+---------------Time to First Token----------------
+Mean TTFT (ms):                          557.85
+Median TTFT (ms):                        213.58
+P99 TTFT (ms):                           1625.28
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          23.84
+Median TPOT (ms):                        23.90
+P99 TPOT (ms):                           24.03
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           23.90
+Median ITL (ms):                         23.92
+P95 ITL (ms):                            24.15
+P99 ITL (ms):                            24.25
+Max ITL (ms):                            25.44
+==================================================
+```
+
+- Medium Concurrency (Balanced)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  107.71
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40625
+Request throughput (req/s):              0.74
+Input token throughput (tok/s):          368.28
+Output token throughput (tok/s):         378.84
+Peak output token throughput (tok/s):    508.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          747.12
+Concurrency:                             13.72
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   18473.65
+Median E2E Latency (ms):                 19558.42
+---------------Time to First Token----------------
+Mean TTFT (ms):                          607.91
+Median TTFT (ms):                        191.32
+P99 TTFT (ms):                           2135.13
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          35.50
+Median TPOT (ms):                        35.99
+P99 TPOT (ms):                           43.62
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           35.10
+Median ITL (ms):                         32.18
+P95 ITL (ms):                            33.03
+P99 ITL (ms):                            159.99
+Max ITL (ms):                            453.99
+==================================================
+```
+
+- High Concurrency (Throughput-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  207.65
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    251238
+Request throughput (req/s):              2.41
+Input token throughput (tok/s):          1203.15
+Output token throughput (tok/s):         1216.79
+Peak output token throughput (tok/s):    2100.00
+Peak concurrent requests:                106
+Total token throughput (tok/s):          2419.94
+Concurrency:                             91.02
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   37800.20
+Median E2E Latency (ms):                 35921.56
+---------------Time to First Token----------------
+Mean TTFT (ms):                          835.15
+Median TTFT (ms):                        236.88
+P99 TTFT (ms):                           2868.52
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          73.33
+Median TPOT (ms):                        76.35
+P99 TPOT (ms):                           97.63
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           73.30
+Median ITL (ms):                         50.82
+P95 ITL (ms):                            180.67
+P99 ITL (ms):                            186.83
+Max ITL (ms):                            1661.39
+==================================================
+```
+
+**Scenario 2: Reasoning (1K/8K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  1097.29
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  44462
+Total generated tokens (retokenized):    44313
+Request throughput (req/s):              0.01
+Input token throughput (tok/s):          5.56
+Output token throughput (tok/s):         40.52
+Peak output token throughput (tok/s):    43.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          46.08
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   109725.52
+Median E2E Latency (ms):                 117748.67
+---------------Time to First Token----------------
+Mean TTFT (ms):                          156.67
+Median TTFT (ms):                        156.19
+P99 TTFT (ms):                           159.87
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          24.41
+Median TPOT (ms):                        24.51
+P99 TPOT (ms):                           24.96
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           24.65
+Median ITL (ms):                         24.58
+P95 ITL (ms):                            25.68
+P99 ITL (ms):                            25.93
+Max ITL (ms):                            29.80
+==================================================
+```
+
+- Medium Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  775.02
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  318306
+Total generated tokens (retokenized):    317426
+Request throughput (req/s):              0.10
+Input token throughput (tok/s):          51.18
+Output token throughput (tok/s):         410.70
+Peak output token throughput (tok/s):    512.00
+Peak concurrent requests:                18
+Total token throughput (tok/s):          461.89
+Concurrency:                             13.86
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   134236.65
+Median E2E Latency (ms):                 135181.28
+---------------Time to First Token----------------
+Mean TTFT (ms):                          214.35
+Median TTFT (ms):                        194.12
+P99 TTFT (ms):                           300.27
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          33.72
+Median TPOT (ms):                        34.00
+P99 TPOT (ms):                           34.75
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           33.69
+Median ITL (ms):                         33.71
+P95 ITL (ms):                            34.50
+P99 ITL (ms):                            34.92
+Max ITL (ms):                            164.76
+==================================================
+```
+
+- High Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  1231.97
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total input vision tokens:               0
+Total generated tokens:                  1301025
+Total generated tokens (retokenized):    1296845
+Request throughput (req/s):              0.26
+Input token throughput (tok/s):          129.01
+Output token throughput (tok/s):         1056.05
+Peak output token throughput (tok/s):    1472.00
+Peak concurrent requests:                67
+Total token throughput (tok/s):          1185.07
+Concurrency:                             56.17
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   216256.25
+Median E2E Latency (ms):                 224192.84
+---------------Time to First Token----------------
+Mean TTFT (ms):                          317.68
+Median TTFT (ms):                        235.28
+P99 TTFT (ms):                           649.39
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          53.30
+Median TPOT (ms):                        55.10
+P99 TPOT (ms):                           56.58
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           53.13
+Median ITL (ms):                         52.95
+P95 ITL (ms):                            56.23
+P99 ITL (ms):                            181.04
+Max ITL (ms):                            208.61
+==================================================
+```
+
+**Scenario 3: Summarization (8K/1K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  114.47
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4194
+Request throughput (req/s):              0.09
+Input token throughput (tok/s):          366.39
+Output token throughput (tok/s):         36.87
+Peak output token throughput (tok/s):    42.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          403.26
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   11442.86
+Median E2E Latency (ms):                 9508.87
+---------------Time to First Token----------------
+Mean TTFT (ms):                          883.78
+Median TTFT (ms):                        481.38
+P99 TTFT (ms):                           2217.45
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          24.93
+Median TPOT (ms):                        25.05
+P99 TPOT (ms):                           26.11
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           25.08
+Median ITL (ms):                         25.08
+P95 ITL (ms):                            26.18
+P99 ITL (ms):                            26.28
+Max ITL (ms):                            27.41
+==================================================
+```
+
+- Medium Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  162.33
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total input vision tokens:               0
+Total generated tokens:                  41669
+Total generated tokens (retokenized):    41443
+Request throughput (req/s):              0.49
+Input token throughput (tok/s):          1848.27
+Output token throughput (tok/s):         256.70
+Peak output token throughput (tok/s):    467.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          2104.97
+Concurrency:                             14.52
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   29456.89
+Median E2E Latency (ms):                 27628.16
+---------------Time to First Token----------------
+Mean TTFT (ms):                          1784.30
+Median TTFT (ms):                        1347.21
+P99 TTFT (ms):                           5384.54
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          53.65
+Median TPOT (ms):                        52.09
+P99 TPOT (ms):                           74.39
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           53.23
+Median ITL (ms):                         34.52
+P95 ITL (ms):                            35.81
+P99 ITL (ms):                            513.25
+Max ITL (ms):                            2865.73
+==================================================
+```
+
+- High Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model deepseek-ai/DeepSeek-V3.1 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  282.55
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total input vision tokens:               0
+Total generated tokens:                  170000
+Total generated tokens (retokenized):   169081
+Request throughput (req/s):              1.13
+Input token throughput (tok/s):          4508.6
+Output token throughput (tok/s):         601.67
+Peak output token throughput (tok/s):   1216
+Peak concurrent requests:                68
+Total token throughput (tok/s):         5110.27
+Concurrency:                            59.81
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                  52810.32
+Median E2E Latency (ms):                50981.81
+---------------Time to First Token----------------
+Mean TTFT (ms):                         786.69
+Median TTFT (ms):                       499.38
+P99 TTFT (ms):                          2925.98
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                         97.93
+Median TPOT (ms):                       103.45
+P99 TPOT (ms):                          157.84
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                          98.11
+Median ITL (ms):                        55.7
+P95 ITL (ms):                           240.71
+P99 ITL (ms):                          1114.36
+==================================================
+```
+
+#### 5.1.5 Understanding the Results
+
+**Key Metrics:**
+
+- **Request Throughput (req/s)**: Number of requests processed per second
+- **Output Token Throughput (tok/s)**: Total tokens generated per second
+- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
+- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
+- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
+
+**Why These Configurations Matter:**
+
+- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
+- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
+- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
+- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
+
+**Interpreting Results:**
+
+- Compare your results against baseline numbers for your hardware
+- Higher throughput at same latency = better performance
+- Lower TTFT = more responsive user experience
+- Lower TPOT = faster generation speed
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py \
+  --num-shots 8 \
+  --num-questions 1316 \
+  --parallel 1316
+```
+
+**Test Results:**
+
+```text Output
+Accuracy: 0.959
+Invalid: 0.000
+Latency: 29.185 s
+Output throughput: 4854.672 token/s
+```

docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx

@@ -0,0 +1,827 @@
+---
+title: DeepSeek-V3.2
+metatags:
+    description: "Deploy DeepSeek-V3.2 with SGLang - featuring DeepSeek Sparse Attention for efficient long-context processing and deep reasoning capabilities."
+---
+
+## 1. Model Introduction
+
+The DeepSeek-V3.2 series includes three model variants, each optimized for different use cases:
+
+**[DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)** is an upgraded version of DeepSeek-V3.1-Terminus, introducing the DeepSeek Sparse Attention (DSA) mechanism through continued training. DSA is a fine-grained sparse attention mechanism powered by a lightning indexer, enabling DeepSeek-V3.2-Exp to achieve significant efficiency improvements in long-context scenarios. Recommended for general conversations, long-context processing, and efficient inference.
+
+**[DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)** is the standard version suitable for general tasks and conversational scenarios. For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top_p = 0.95. Recommended for standard conversations and general tasks.
+
+**[DeepSeek-V3.2-Speciale](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale)** is a special variant designed exclusively for deep reasoning tasks. This model is specifically optimized for scenarios requiring complex logical reasoning and deep thinking. However this model does not support tool calls (see below). For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top_p = 0.95. Recommended for deep reasoning tasks, complex logical problems, and mathematical reasoning.
+
+**[DeepSeek-V3.2-NVFP4](https://huggingface.co/nvidia/DeepSeek-V3.2-NVFP4)** is an NVIDIA-optimized NVFP4-quantized variant of DeepSeek-V3.2 for Blackwell devices. It uses ModelOpt FP4 quantization with a choice of MoE runner backends (`flashinfer_trtllm` (recommended), `flashinfer_cutlass`, or `flashinfer_cutedsl`), enabling efficient deployment with lower tensor parallelism (TP=4). It supports the same features as DeepSeek-V3.2 including tool calling, reasoning, and speculative decoding (MTP).
+
+**[DeepSeek-V3.2-MXFP4](https://huggingface.co/amd/DeepSeek-V3.2-mxfp4)** is an OCP-MXFP4 optimized variant for DeepSeek-V3.2 for AMD MI300X/MI355X devices. It uses OCP MXFP4 quantization with a triton mxfp4 backend (the same backend for gptoss-120B), enabling efficient deployment with lower tensor parallelism (TP=8) in a single node. It includes the same features as DeepSeek-V3.2 including tool calling, reasoning, fp8-kv, CP, TP and speculative decoding MTP.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. SGLang supports serving DeepSeek V3.2 on NVIDIA H200, B200, and AMD MI300X/MI355X GPUs.
+
+import { DeepSeekV32Deployment } from "/src/snippets/autoregressive/deepseek-v32-deployment.jsx";
+
+<DeepSeekV32Deployment />
+
+### 3.2 Configuration Tips
+For more detailed configuration tips, please refer to [DeepSeek-V3.2 Usage](../../../docs/basic_usage/deepseek_v32).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [Basic API Usage](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+DeepSeek-V3.2 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+sglang serve \
+  --model deepseek-ai/DeepSeek-V3.2-Exp \
+  --reasoning-parser deepseek-v3 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3.2-Exp",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    extra_body = {"chat_template_kwargs": {"thinking": True}},
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To solve this problem, I need to calculate 15% of 240.
+Step 1: Convert 15% to decimal: 15% = 0.15
+Step 2: Multiply 240 by 0.15
+Step 3: 240 × 0.15 = 36
+=============== Content =================
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+DeepSeek-V3.2 and DeepSeek-V3.2-Exp support tool calling capabilities. But they use different parameters. Enable the tool call parser:
+
+**Note:** DeepSeek-V3.2-Speciale does **NOT** support tool calling. It is designed exclusively for deep reasoning tasks.
+
+**Deployment Command:**
+
+For DeepSeek-V3.2-Exp:
+
+```shell Command
+sglang serve \
+  --model deepseek-ai/DeepSeek-V3.2-Exp \
+  --tool-call-parser deepseekv31 \
+  --reasoning-parser deepseek-v3 \
+  --chat-template ./examples/chat_template/tool_chat_template_deepseekv32.jinja \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+For DeepSeek-V3.2, use `--tool-call-parser deepseekv32` and remove `--chat-template`.
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3.2-Exp",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    extra_body = {"chat_template_kwargs": {"thinking": True}},
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="deepseek-ai/DeepSeek-V3.2-Exp",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+#### 4.2.3 Enabling PP, CP and TP with FP8 KV cache
+
+We suggested `DP2` + `MTP` for local deployment of agentic workflow with DeepSeek V3.2 on Hopper platform:
+
+```shell Command
+export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32
+export SGLANG_SET_CPU_AFFINITY=1
+
+# Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970
+#   dp 2 : 5019.54  toks/sec, MAX ITL 7233
+#   dp 4 : 4942.82  toks/sec, MAX ITL 35654
+#   dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081
+sglang_args=$(echo serve \
+  --model-path $MAPPED_MODEL_PATH \
+  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
+  --dp 2 --enable-dp-attention --page-size 64 \
+  --trust-remote-code --host "0.0.0.0" --port 30000 \
+  --log-requests \
+  --context-length 65536 --max-running-requests 128 \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
+  --allow-auto-truncate --enable-metrics \
+  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
+  --served-model-name DeepSeek-V3.2-Opt-dp2-mtp
+)
+
+sglang_args=($sglang_args)
+
+sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
+```
+
+**CP + PP + EP + DP**
+
+`CP` is currently enabled with `PP=2` on Hopper platform and we can reduce TP=16 to TP=8 from standalone deployment:
+
+```shell Command
+# verified on Hopper platform
+sglang_args=$(echo serve \
+  --model-path $MAPPED_MODEL_PATH \
+  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --dp 1 --enable-dp-attention \
+  --moe-a2a-backend deepep --ep-size 16  \
+  --page-size 128 \
+  --chunked-prefill-size 16384 \
+  --attention-backend nsa \
+  --nsa-prefill-backend flashmla_sparse \
+  --nsa-decode-backend flashmla_sparse \
+  --enable-nsa-prefill-context-parallel \
+  --nsa-prefill-cp-mode round-robin-split \
+  --cuda-graph-max-bs 128 \
+  --max-running-requests 128 \
+  --trust-remote-code --host "0.0.0.0" --port 30000 \
+  --log-requests \
+  --context-length 65536 \
+  --allow-auto-truncate --enable-metrics \
+  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
+  --served-model-name DeepSeek-V3.2-nsa-pp-cp-ep-dp
+)
+
+sglang_args=($sglang_args)
+
+sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
+```
+
+**fp8 KV + CP + PP**
+
+With FP8 KV, we can have less memory footprint. This can be combined with various parallel schemes:
+
+```shell Command
+# verified in Hopper platform
+dp=1
+
+dp_config=" \
+  --dp 1 --enable-dp-attention \
+"
+
+cp_config=" \
+  --enable-nsa-prefill-context-parallel \
+"
+
+if [ "$dp" -eq 1 ]; then
+
+cp_config=" \
+  $cp_config \
+  --nsa-prefill-cp-mode round-robin-split \
+"
+
+else
+cp_config=" \
+  $cp_config \
+  --nsa-prefill-cp-mode in-seq-split \
+"
+fi
+
+# see discussion : https://github.com/sgl-project/sglang/pull/12065
+sglang_args=$(echo serve \
+  --model-path $MAPPED_MODEL_PATH \
+  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --pp-async-batch-depth 1 \
+  $dp_config \
+  --trust-remote-code --host "0.0.0.0" --port 30000 \
+  --log-requests \
+  --context-length 65536 --max-running-requests 128 \
+  $cp_config \
+  --kv-cache-dtype fp8_e4m3 \
+  --allow-auto-truncate --enable-metrics \
+  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
+  --served-model-name DeepSeek-V3.2-Opt-fp8kv-pp2-cp4
+)
+
+sglang_args=($sglang_args)
+
+sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark on Blackwell
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (8x)
+- Model: DeepSeek-V3.2-Exp
+- Tensor Parallelism: 8
+- sglang version: 0.5.6
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+sglang serve \
+  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
+  --tp 8 \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-V3.2-Exp \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  29.11
+Total input tokens:                      1972
+Total input text tokens:                 1972
+Total input vision tokens:               0
+Total generated tokens:                  2784
+Total generated tokens (retokenized):    2777
+Request throughput (req/s):              0.34
+Input token throughput (tok/s):          67.73
+Output token throughput (tok/s):         95.62
+Peak output token throughput (tok/s):    157.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          163.36
+Concurrency:                             1.00
+Accept length:                           2.46
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   2909.74
+Median E2E Latency (ms):                 3088.27
+P90 E2E Latency (ms):                    4200.62
+P99 E2E Latency (ms):                    5588.52
+---------------Time to First Token----------------
+Mean TTFT (ms):                          317.58
+Median TTFT (ms):                        191.31
+P99 TTFT (ms):                           740.79
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          9.09
+Median TPOT (ms):                        9.25
+P99 TPOT (ms):                           11.73
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           9.35
+Median ITL (ms):                         7.64
+P95 ITL (ms):                            22.81
+P99 ITL (ms):                            23.33
+Max ITL (ms):                            31.45
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+sglang serve \
+  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
+  --tp 8 \
+  --ep 8 \
+  --dp 8 \
+  --enable-dp-attention \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-V3.2-Exp \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  219.09
+Total input tokens:                      301701
+Total input text tokens:                 301701
+Total input vision tokens:               0
+Total generated tokens:                  188375
+Total generated tokens (retokenized):    187443
+Request throughput (req/s):              4.56
+Input token throughput (tok/s):          1377.06
+Output token throughput (tok/s):         859.80
+Peak output token throughput (tok/s):    2465.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          2236.86
+Concurrency:                             88.05
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   19291.23
+Median E2E Latency (ms):                 11927.39
+---------------Time to First Token----------------
+Mean TTFT (ms):                          530.36
+Median TTFT (ms):                        444.00
+P99 TTFT (ms):                           1504.78
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          106.16
+Median TPOT (ms):                        106.69
+P99 TPOT (ms):                           221.12
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           100.46
+Median ITL (ms):                         41.73
+P95 ITL (ms):                            225.67
+P99 ITL (ms):                            392.37
+Max ITL (ms):                            975.03
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 30000
+```
+
+- **Test Results**:
+  - DeepSeek-V3.2-Exp
+    ```
+    Accuracy: 0.980
+    Invalid: 0.000
+    Latency: 19.128 s
+    Output throughput: 965.919 token/s
+    ```
+
+#### 5.2.2 MMLU Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+cd sglang
+bash benchmark/mmlu/download_data.sh
+python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 30000
+```
+
+- **Test Results**:
+  - DeepSeek-V3.2-Exp
+    ```
+    subject: abstract_algebra, #q:100, acc: 0.780
+    subject: anatomy, #q:135, acc: 0.874
+    subject: astronomy, #q:152, acc: 0.961
+    subject: business_ethics, #q:100, acc: 0.860
+    subject: clinical_knowledge, #q:265, acc: 0.925
+    subject: college_biology, #q:144, acc: 0.972
+    subject: college_chemistry, #q:100, acc: 0.660
+    subject: college_computer_science, #q:100, acc: 0.880
+    subject: college_mathematics, #q:100, acc: 0.840
+    subject: college_medicine, #q:173, acc: 0.879
+    Total latency: 7.961
+    Average accuracy: 0.879
+    ```
+
+### 5.3 Speed Benchmark on Hopper
+
+**Test Environment:**
+
+- Hardware: NVIDIA H800 GPU (16x)
+- Model: DeepSeek-V3.2
+- Tensor Parallelism: 16
+- sglang version: 0.5.9
+
+#### 5.3.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32
+export SGLANG_SET_CPU_AFFINITY=1
+
+# Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970
+#   dp 2 : 5019.54  toks/sec, MAX ITL 7233
+#   dp 4 : 4942.82  toks/sec, MAX ITL 35654
+#   dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081
+sglang_args=$(echo serve \
+  --model-path $MAPPED_MODEL_PATH \
+  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
+  --dp 2 --enable-dp-attention --page-size 64 \
+  --trust-remote-code --host "0.0.0.0" --port 30000 \
+  --log-requests \
+  --context-length 65536 --max-running-requests 128 \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
+  --allow-auto-truncate --enable-metrics \
+  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
+  --served-model-name DeepSeek-V3.2-Opt-dp2-mtp
+)
+
+sglang_args=($sglang_args)
+
+sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host $MASTER_ADDR \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-V3.2 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    64.0
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  48.96
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4217
+Request throughput (req/s):              0.20
+Input token throughput (tok/s):          124.62
+Output token throughput (tok/s):         86.20
+Peak output token throughput (tok/s):    113.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          210.81
+Concurrency:                             1.00
+Accept length:                           3.27
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   4893.12
+Median E2E Latency (ms):                 3742.47
+P90 E2E Latency (ms):                    8877.37
+P99 E2E Latency (ms):                    10769.85
+---------------Time to First Token----------------
+Mean TTFT (ms):                          199.88
+Median TTFT (ms):                        176.15
+P99 TTFT (ms):                           272.49
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.99
+Median TPOT (ms):                        10.88
+P99 TPOT (ms):                           13.93
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.15
+Median ITL (ms):                         8.86
+P95 ITL (ms):                            17.29
+P99 ITL (ms):                            33.71
+Max ITL (ms):                            36.84
+==================================================
+```
+
+#### 5.3.2 Throughput-Sensitive Benchmark
+
+We simply use the same deployment method and vary the throughput by maximizing concurrencies:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host $MASTER_ADDR \
+  --port 30000 \
+  --model deepseek-ai/DeepSeek-V3.2 \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 2048 \
+  --max-concurrency 1024 # see picture below why we use 1024 for concurrency, hence num prompts 2048
+```
+
+DeepSeek 3.2 can steadily support concurrency up to `1024` and when concurrency is greater than `128`, the TTFT increase sharply:
+
+![DeepSeek V3.2 Concurrency ISL/OSL=1024/128](https://github.com/user-attachments/assets/d5c9c9fb-44f3-4793-a0fd-f8fa954546f5)
+
+
+Performance record:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    64.0
+Max request concurrency:                 1024
+Successful requests:                     2048
+Benchmark duration (s):                  408.09
+Total input tokens:                      1048992
+Total input text tokens:                 1048992
+Total generated tokens:                  1032734
+Total generated tokens (retokenized):    1031817
+Request throughput (req/s):              5.02
+Input token throughput (tok/s):          2570.50
+Output token throughput (tok/s):         2530.66
+Peak output token throughput (tok/s):    5092.00
+Peak concurrent requests:                1035
+Total token throughput (tok/s):          5101.16
+Concurrency:                             763.41
+Accept length:                           3.26
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   152117.70
+Median E2E Latency (ms):                 181704.84
+P90 E2E Latency (ms):                    215924.77
+P99 E2E Latency (ms):                    231679.59
+---------------Time to First Token----------------
+Mean TTFT (ms):                          127729.28
+Median TTFT (ms):                        170098.94
+P99 TTFT (ms):                           185705.73
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          49.18
+Median TPOT (ms):                        48.48
+P99 TPOT (ms):                           77.24
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           48.46
+Median ITL (ms):                         52.11
+P95 ITL (ms):                            110.26
+P99 ITL (ms):                            200.63
+Max ITL (ms):                            2666.37
+==================================================
+```
+
+By adding `--random-range-ratio 1`, we could get even higher statistical numbers:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    64.0
+Max request concurrency:                 1024
+Successful requests:                     2048
+Benchmark duration (s):                  612.87
+Total input tokens:                      2097152
+Total input text tokens:                 2097152
+Total generated tokens:                  2097152
+Total generated tokens (retokenized):    2096201
+Request throughput (req/s):              3.34
+Input token throughput (tok/s):          3421.84
+Output token throughput (tok/s):         3421.84
+Peak output token throughput (tok/s):    9077.00
+Peak concurrent requests:                1039
+Total token throughput (tok/s):          6843.68
+Concurrency:                             772.66
+Accept length:                           3.26
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   231222.27
+Median E2E Latency (ms):                 289846.24
+P90 E2E Latency (ms):                    314480.41
+P99 E2E Latency (ms):                    320392.27
+---------------Time to First Token----------------
+Mean TTFT (ms):                          194081.02
+Median TTFT (ms):                        252945.22
+P99 TTFT (ms):                           279637.50
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          36.31
+Median TPOT (ms):                        36.73
+P99 TPOT (ms):                           46.33
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           36.31
+Median ITL (ms):                         23.18
+P95 ITL (ms):                            96.79
+P99 ITL (ms):                            135.81
+Max ITL (ms):                            3121.00
+==================================================
+```

docs_new/cookbook/autoregressive/Ernie/Ernie4.5-VL.mdx

@@ -0,0 +1,28 @@
+---
+title: Ernie4.5-VL
+metatags:
+    description: "Deploy Ernie4.5-VL vision-language model with SGLang - community contribution guide for Baidu's multimodal model."
+---
+
+## 📝 Community Contribution Welcome
+
+This guide is currently under development. We welcome community contributions!
+
+If you have experience deploying **Ernie4.5-VL** with SGLang, please help us complete this documentation.
+
+## 🚀 How to Contribute
+
+```shell Command
+git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
+cd sglang-cookbook
+git checkout -b add-ernie4-5-vl-guide
+# Edit this file and submit a PR
+```
+
+## 📚 Reference
+
+- [GLM-4.6V](../GLM/GLM-4.6V)
+
+---
+
+**Let's build this together!** 🌟

docs_new/cookbook/autoregressive/Ernie/Ernie4.5.mdx

@@ -0,0 +1,696 @@
+---
+title: Ernie4.5
+metatags:
+    description: "Deploy Ernie4.5 with SGLang - community contribution guide for Baidu's Ernie 4.5 model deployment."
+---
+
+import { Ernie45Deployment } from '/src/snippets/autoregressive/ernie-45-deployment.jsx';
+
+## 1. Model Introduction
+
+The **ERNIE-4.5** series is a family of large language models developed by Baidu. ERNIE (Enhanced Representation through Knowledge Integration) 4.5 represents an advanced version of the ERNIE series, optimized for general-purpose tasks and conversational scenarios.
+
+ERNIE-4.5 delivers advanced features as below:
+- **Heterogeneous Modality Structure**: MoE architecture that supports parameter sharing across modalities while allowing dedicated parameters for each individual modality, enhancing multimodal understanding without compromising, and even improving, performance on text-related tasks.
+- **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing.
+- **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations.
+- **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions.
+- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
+
+<Ernie45Deployment />
+
+## 4. API Usage
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+The following example demonstrates deployment using ERNIE-4.5-21B-A3B-PT.
+
+```shell Command
+python -m sglang.launch_server \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --tp 1
+```
+
+**Basic Python Client Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="baidu/ERNIE-4.5-21B-A3B-PT",
+    messages=[
+        {"role": "user", "content": "What is artificial intelligence?"}
+    ],
+    temperature=1.0,
+    top_p=0.95,
+    max_tokens=1024
+)
+
+print(response.choices[0].message.content)
+```
+
+**Output Example:**
+```text Output
+**Artificial Intelligence (AI)** is the simulation of human intelligence processes by machines, particularly computer systems. These processes include **learning** (acquiring information and rules for using the information), **reasoning** (using rules to reach approximate or definite conclusions), and **self-correction**. AI encompasses a wide range of techniques, algorithms, and methodologies designed to enable machines to perform tasks that typically require human intelligence.
+
+### Key Characteristics of AI:
+...
+
+### In Summary:
+AI represents a transformative force with the potential to revolutionize industries and enhance human capabilities. However, its development requires careful consideration of ethical, legal, and social implications to ensure that it benefits society as a whole. As AI continues to evolve, ongoing dialogue among stakeholders will be crucial to balancing innovation with responsibility.
+```
+
+**Streaming Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="baidu/ERNIE-4.5-21B-A3B-PT",
+    messages=[
+        {"role": "user", "content": "Explain quantum computing in simple terms."}
+    ],
+    temperature=1.0,
+    top_p=0.95,
+    max_tokens=2048,
+    stream=True
+)
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+Sure! Here’s a simple explanation of quantum computing:
+
+### **Quantum Computing: Making Computers Super Fast (But Weird) Using Quantum Rules**
+
+1. **Classic vs. Quantum Computers**
+   - **Normal computers** use **bits** (0s and 1s) to store and process information.
+   - **Quantum computers** use **qubits** (short for quantum bits). Unlike bits, qubits can be **0, 1, or both at the same time** (this is called **superposition**).
+
+2. **Superposition: The Magic Behind Speed**
+   - A single qubit can represent **0 and 1 simultaneously**, like a coin spinning in the air.
+   - Many qubits working together (in something called **quantum parallelism**) can **check multiple possibilities at once**, making quantum computers much faster for certain problems.
+
+3. **Entanglement: Making Qubits Link**
+   - When qubits are **entangled**, their states are linked—changing one instantly affects the other, no matter how far apart they are (this is called **spooky action at a distance** by Einstein).
+   - Entanglement allows quantum computers to process information in **very efficient ways**.
+
+4. **What Quantum Computers Are Good At**
+   - **Cracking encryption** (like RSA).
+   - **Factoring large numbers** (used in encryption and cryptography).
+   - **Searching unsorted databases** (way faster than classical computers).
+   - **Simulating quantum systems** (like molecules for drug discovery).
+   - **Optimizing problems** (like logistics or finance).
+
+5. **Challenges & Current State**
+   - Qubits are **fragile** and easily disturbed (called **decoherence**).
+   - Engineers are working to keep qubits stable long enough to do useful calculations.
+   - Today’s quantum computers are **small and experimental**, but the goal is to build powerful ones that outperform classical supercomputers.
+
+### **Final Thought**
+Quantum computing isn’t just a faster calculator—it’s a **new way of thinking about problems** using the weird laws of physics. While still new, it has the potential to revolutionize fields like medicine, AI, and cybersecurity.
+
+Would you like an example of how a quantum computer might solve a problem? 😊
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X GPU (1x)
+- Model: ERNIE-4.5-21B-A3B-PT
+- Tensor Parallelism: 1
+- SGLang Version: 0.5.7
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Scenario Benchmark
+
+- Model Deployment Command:
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path baidu/ERNIE-4.5-21B-A3B-PT \
+  --tp 1
+```
+
+##### 5.1.1.1 Low Concurrency (Latency-Optimized)
+- Benchmark Command:
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  58.72
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4219
+Request throughput (req/s):              0.17
+Input token throughput (tok/s):          103.90
+Output token throughput (tok/s):         71.87
+Peak output token throughput (tok/s):    245.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          175.77
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5869.86
+Median E2E Latency (ms):                 1870.80
+---------------Time to First Token----------------
+Mean TTFT (ms):                          4152.58
+Median TTFT (ms):                        36.81
+P99 TTFT (ms):                           37498.23
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          4.07
+Median TPOT (ms):                        4.09
+P99 TPOT (ms):                           4.09
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           4.08
+Median ITL (ms):                         4.08
+P95 ITL (ms):                            4.14
+P99 ITL (ms):                            4.20
+Max ITL (ms):                            4.67
+==================================================
+```
+
+##### 5.1.1.2 Medium Concurrency (Balanced)
+- Benchmark Command:
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  34.30
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40773
+Request throughput (req/s):              2.33
+Input token throughput (tok/s):          1156.62
+Output token throughput (tok/s):         1189.77
+Peak output token throughput (tok/s):    1392.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          2346.39
+Concurrency:                             14.14
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6060.62
+Median E2E Latency (ms):                 6496.70
+---------------Time to First Token----------------
+Mean TTFT (ms):                          78.90
+Median TTFT (ms):                        45.90
+P99 TTFT (ms):                           234.33
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          11.99
+Median TPOT (ms):                        12.16
+P99 TPOT (ms):                           14.81
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.75
+Median ITL (ms):                         11.48
+P95 ITL (ms):                            12.24
+P99 ITL (ms):                            34.85
+Max ITL (ms):                            105.01
+==================================================
+```
+
+##### 5.1.1.3 High Concurrency (Throughput-Optimized)
+- Benchmark Command:
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  66.63
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    252449
+Request throughput (req/s):              7.50
+Input token throughput (tok/s):          3749.79
+Output token throughput (tok/s):         3792.28
+Peak output token throughput (tok/s):    4902.00
+Peak concurrent requests:                113
+Total token throughput (tok/s):          7542.06
+Concurrency:                             90.33
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   12036.90
+Median E2E Latency (ms):                 11782.16
+---------------Time to First Token----------------
+Mean TTFT (ms):                          104.86
+Median TTFT (ms):                        84.62
+P99 TTFT (ms):                           297.85
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          23.89
+Median TPOT (ms):                        24.62
+P99 TPOT (ms):                           26.91
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           23.66
+Median ITL (ms):                         20.48
+P95 ITL (ms):                            45.57
+P99 ITL (ms):                            54.31
+Max ITL (ms):                            185.12
+==================================================
+```
+
+#### 5.1.2 Reasoning Scenario Benchmark
+
+##### 5.1.2.1 Low Concurrency
+- Benchmark Command:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  185.11
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  44462
+Total generated tokens (retokenized):    44423
+Request throughput (req/s):              0.05
+Input token throughput (tok/s):          32.96
+Output token throughput (tok/s):         240.19
+Peak output token throughput (tok/s):    245.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          273.15
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   18508.84
+Median E2E Latency (ms):                 19866.81
+---------------Time to First Token----------------
+Mean TTFT (ms):                          32.59
+Median TTFT (ms):                        32.14
+P99 TTFT (ms):                           38.58
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          4.13
+Median TPOT (ms):                        4.13
+P99 TPOT (ms):                           4.20
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           4.16
+Median ITL (ms):                         4.12
+P95 ITL (ms):                            4.31
+P99 ITL (ms):                            4.36
+Max ITL (ms):                            7.28
+==================================================
+```
+
+##### 5.1.2.2 Medium Concurrency
+
+- Benchmark Command:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  263.48
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  318306
+Total generated tokens (retokenized):    317984
+Request throughput (req/s):              0.30
+Input token throughput (tok/s):          150.55
+Output token throughput (tok/s):         1208.09
+Peak output token throughput (tok/s):    1408.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          1358.64
+Concurrency:                             14.35
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   47249.55
+Median E2E Latency (ms):                 47828.67
+---------------Time to First Token----------------
+Mean TTFT (ms):                          62.77
+Median TTFT (ms):                        57.10
+P99 TTFT (ms):                           93.70
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          11.92
+Median TPOT (ms):                        12.09
+P99 TPOT (ms):                           12.50
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.86
+Median ITL (ms):                         12.04
+P95 ITL (ms):                            12.68
+P99 ITL (ms):                            13.61
+Max ITL (ms):                            39.94
+==================================================
+```
+
+##### 5.1.2.3 High Concurrency
+- Benchmark Command:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  428.30
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total input vision tokens:               0
+Total generated tokens:                  1301025
+Total generated tokens (retokenized):    1299877
+Request throughput (req/s):              0.75
+Input token throughput (tok/s):          371.09
+Output token throughput (tok/s):         3037.63
+Peak output token throughput (tok/s):    3880.00
+Peak concurrent requests:                69
+Total token throughput (tok/s):          3408.73
+Concurrency:                             57.08
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   76392.58
+Median E2E Latency (ms):                 79698.73
+---------------Time to First Token----------------
+Mean TTFT (ms):                          92.79
+Median TTFT (ms):                        78.71
+P99 TTFT (ms):                           168.89
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          18.81
+Median TPOT (ms):                        19.15
+P99 TPOT (ms):                           19.81
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           18.77
+Median ITL (ms):                         18.77
+P95 ITL (ms):                            19.86
+P99 ITL (ms):                            42.08
+Max ITL (ms):                            74.36
+==================================================
+```
+
+#### 5.1.3 Summarization Scenario Benchmark
+
+##### 5.1.3.1 Low Concurrency
+- Benchmark Command:
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  18.59
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4216
+Request throughput (req/s):              0.54
+Input token throughput (tok/s):          2256.43
+Output token throughput (tok/s):         227.04
+Peak output token throughput (tok/s):    245.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          2483.46
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   1856.72
+Median E2E Latency (ms):                 1513.87
+---------------Time to First Token----------------
+Mean TTFT (ms):                          86.66
+Median TTFT (ms):                        72.30
+P99 TTFT (ms):                           167.13
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          4.19
+Median TPOT (ms):                        4.22
+P99 TPOT (ms):                           4.30
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           4.20
+Median ITL (ms):                         4.23
+P95 ITL (ms):                            4.34
+P99 ITL (ms):                            4.42
+Max ITL (ms):                            5.68
+==================================================
+```
+
+##### 5.1.3.2 Medium Concurrency
+- Benchmark Command:
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  40.25
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total input vision tokens:               0
+Total generated tokens:                  41669
+Total generated tokens (retokenized):    41646
+Request throughput (req/s):              1.99
+Input token throughput (tok/s):          7454.72
+Output token throughput (tok/s):         1035.37
+Peak output token throughput (tok/s):    1310.00
+Peak concurrent requests:                20
+Total token throughput (tok/s):          8490.09
+Concurrency:                             14.37
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   7229.56
+Median E2E Latency (ms):                 7578.95
+---------------Time to First Token----------------
+Mean TTFT (ms):                          137.38
+Median TTFT (ms):                        122.59
+P99 TTFT (ms):                           485.34
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          14.04
+Median TPOT (ms):                        14.24
+P99 TPOT (ms):                           20.77
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           13.64
+Median ITL (ms):                         12.36
+P95 ITL (ms):                            14.72
+P99 ITL (ms):                            57.39
+Max ITL (ms):                            411.31
+==================================================
+```
+
+##### 5.1.3.3 High Concurrency
+
+- Benchmark Command:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model baidu/ERNIE-4.5-21B-A3B-PT \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  78.33
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total input vision tokens:               0
+Total generated tokens:                  170000
+Total generated tokens (retokenized):    169888
+Request throughput (req/s):              4.09
+Input token throughput (tok/s):          16262.33
+Output token throughput (tok/s):         2170.20
+Peak output token throughput (tok/s):    3005.00
+Peak concurrent requests:                73
+Total token throughput (tok/s):          18432.53
+Concurrency:                             58.79
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   14392.52
+Median E2E Latency (ms):                 14460.70
+---------------Time to First Token----------------
+Mean TTFT (ms):                          184.82
+Median TTFT (ms):                        155.24
+P99 TTFT (ms):                           379.82
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          26.97
+Median TPOT (ms):                        28.31
+P99 TPOT (ms):                           33.61
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           26.79
+Median ITL (ms):                         20.55
+P95 ITL (ms):                            47.55
+P99 ITL (ms):                            145.64
+Max ITL (ms):                            287.62
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command:
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py \
+  --num-shots 8 \
+  --num-questions 1316 \
+  --parallel 1316
+```
+
+- Test Results:
+  - ERNIE-4.5-21B-A3B-PT
+  ```
+  Accuracy: 0.865
+  Invalid: 0.000
+  Latency: 21.669 s
+  Output throughput: 10359.790 token/s
+  ```

docs_new/cookbook/autoregressive/FlashLabs/Chroma1.0.mdx

@@ -0,0 +1,193 @@
+---
+title: Chroma-1.0
+metatags:
+    description: "Deploy Chroma-1.0 end-to-end speech conversation model with SGLang - real-time speech generation, voice cloning, and speech reasoning."
+tag: NEW
+---
+
+## 1. Model Introduction
+
+[Chroma-1.0](https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma) is an open-source end-to-end speech conversation model developed by FlashLabs, focusing on the following core capabilities:
+
+- **Real-time Speech Generation**: Supports low-latency speech synthesis, suitable for real-time conversational scenarios.
+- **Customized Voice Cloning**: Capable of cloning and replicating specific speaker voice characteristics.
+- **End-to-End Architecture**: Provides a complete processing workflow from speech to speech.
+- **Speech Reasoning**: Equipped with reasoning capabilities to understand and process speech content.
+
+## 2. Architecture Overview
+
+**Chroma-1.0** utilizes a hybrid serving architecture rather than a direct SGLang deployment. This design choice is driven by:
+
+1. **Complex Model Architecture**: The end-to-end speech processing pipeline involves specialized components that go beyond standard text generation loops.
+2. **KV Cache & State Management**: The model requires custom handling of KV caches that differs from standard implementations.
+3. **Batching Limitations**: The current implementation supports a batch size of 1, meaning SGLang's advanced continuous batching capabilities are not yet fully applicable.
+
+Therefore, you will start the **FlashLabs Server**, which manages the overall workflow and selectively leverages SGLang for specific inference components where supported.
+
+- **Outer Layer**: FlashLabs Server (Handles Audio I/O, State, and Model Logic)
+- **Inner Engine**: SGLang Instance (Utilized for specific acceleration where applicable)
+
+## 3. Installation & Setup
+
+We recommend following these steps to set up the environment and prepare the model.
+
+### Step 1: Get the Docker Image
+
+Pull the official pre-built image from Docker Hub to ensure all dependencies are correctly configured.
+
+```bash Command
+docker pull flashlabs/chroma:latest
+```
+
+### Step 2: Download Model Weights
+
+Download the **Chroma-4B** weights from Hugging Face. You can choose one of the following methods:
+
+**Method 1: Using Python (Recommended)**
+
+```bash Command
+huggingface-cli download FlashLabs/Chroma-4B --local-dir Chroma-4B
+```
+
+**Method 2: Using Git Clone**
+
+Make sure you have Git LFS installed before cloning.
+
+```bash Command
+# Install Git LFS first
+git lfs install
+
+# Clone the repository
+git clone https://huggingface.co/FlashLabs/Chroma-4B Chroma-4B
+```
+
+### Step 3: Download Chroma Codes (SGLang version)
+
+```bash Command
+git clone https://github.com/FlashLabs-AI-Corp/Chroma-SGLang.git
+
+cd Chroma-SGLang
+```
+
+### Step 4: Run the Server
+
+```bash Command
+docker run -d \
+  --gpus all \
+  -p 8000:8000 \
+  -w /app/Chroma-SGLang \
+  -v "your_Chroma-SGLang_path":/app/Chroma-SGLang \
+  -v "your_chroma_path":/model \
+  -e CHROMA_MODEL_PATH=/model \
+  -e DP_SIZE="1" \
+  flashlabs/chroma:latest \
+  /opt/conda/bin/python -m uvicorn api_server:app \
+  --host 0.0.0.0 \
+  --port 8000 \
+  --workers 1
+```
+
+or run simply the following one line command
+
+```bash Command
+docker-compose up -d
+```
+
+## 5. Client Usage Example
+
+Once the server is running, you can interact with it using HTTP requests.
+
+### Python Client
+
+```python Example
+import requests
+import base64
+
+url = "http://localhost:8000/v1/chat/completions"
+headers = {"Content-Type": "application/json"}
+
+payload = {
+    "model": "chroma",
+    "messages": [
+        {
+            "role": "system",
+            "content": "You are Chroma, a voice agent developed by FlashLabs."
+        },
+        {
+            "role": "user",
+            "content": [
+                {"type": "audio", "audio": "assets/question_audio.wav"}
+            ]
+        }
+    ],
+    "max_tokens": 1000,
+    "return_audio": True
+}
+
+response = requests.post(url, json=payload, headers=headers)
+result = response.json()
+
+if result.get("audio"):
+    audio_data = base64.b64decode(result["audio"])
+    with open("output.wav", "wb") as f:
+        f.write(audio_data)
+    print("Audio saved to output.wav")
+```
+
+### OpenAI SDK Compatible Example
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="dummy",
+    base_url="http://localhost:8000/v1"
+)
+
+response = client.chat.completions.create(
+    model="chroma",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {
+            "role": "user",
+            "content": [
+                {"type": "audio", "audio": "assets/question_audio.wav"}
+            ]
+        }
+    ],
+    extra_body={
+        "prompt_text": "I have not... I'm so exhausted, I haven't slept in a very long time. It could be because... Well, I used our... Uh, I'm, I just use... This is what I use every day. I use our cleanser every day, I use serum in the morning and then the moistu- daily moisturizer. That's what I use every morning.",
+        "prompt_audio": "assets/ref_audio.wav",
+        "return_audio": True
+    }
+)
+
+print(response)
+```
+
+### CLI (cURL)
+
+```bash Command
+curl -X POST http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "chroma",
+    "messages": [
+      {
+        "role": "system",
+        "content": "You are Chroma, a voice agent developed by FlashLabs."
+      },
+      {
+        "role": "user",
+        "content": [
+          {
+            "type": "audio",
+            "audio": "assets/question_audio.wav"
+          }
+        ]
+      }
+    ],
+    "max_tokens": 1000,
+    "return_audio": true
+  }' | jq -r '.audio' | base64 -d > output.wav
+```

docs_new/cookbook/autoregressive/GLM/GLM-4.5.mdx

@@ -0,0 +1,490 @@
+---
+title: GLM-4.5
+metatags:
+    description: "Deploy GLM-4.5 with SGLang on AMD GPUs - advanced reasoning, function calling, BF16/FP8 quantization options."
+---
+
+## 1. Model Introduction
+
+[GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
+
+**Key Features:**
+
+- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
+- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
+- **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
+- **High Performance**: Optimized for both throughput and latency scenarios
+
+**Available Models:**
+
+- **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X
+- **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X
+
+**License:**
+
+Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
+
+import { GLM45Deployment } from "/src/snippets/autoregressive/glm-45-deployment.jsx";
+
+<GLM45Deployment />
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [GLM-4.5/GLM-4.6 Usage](../../../docs/basic_usage/glm45).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.5 \
+  --reasoning-parser glm45 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To solve this problem, I need to calculate 15% of 240.
+Step 1: Convert 15% to decimal: 15% = 0.15
+Step 2: Multiply 240 by 0.15
+Step 3: 240 × 0.15 = 36
+=============== Content =================
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+GLM-4.5 supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.5 \
+  --reasoning-parser glm45 \
+  --tool-call-parser glm45 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
+- Model: GLM-4.5
+- Tensor Parallelism: 8
+- SGLang Version: 0.5.6.post1
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Test Scenarios
+
+Three core scenarios reflect real-world usage patterns:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 Concurrency Levels
+
+Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
+
+- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
+- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
+- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
+
+#### 5.1.3 Number of Prompts
+
+For each concurrency level, configure `num_prompts` to simulate realistic user loads:
+
+- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
+- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
+- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
+
+---
+
+#### 5.1.4 Benchmark Commands
+
+**Scenario 1: Chat (1K/1K) - Most Important**
+
+- **Model Deployment**
+```bash Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.5 \
+  --tp 8
+```
+
+
+- Low Concurrency (Latency-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Medium Concurrency (Balanced)
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- High Concurrency (Throughput-Optimized)
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+**Scenario 2: Reasoning (1K/8K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Medium Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- High Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+**Scenario 3: Summarization (8K/1K)**
+
+- Low Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Medium Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- High Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.5 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+#### 5.1.5 Understanding the Results
+
+**Key Metrics:**
+
+- **Request Throughput (req/s)**: Number of requests processed per second
+- **Output Token Throughput (tok/s)**: Total tokens generated per second
+- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
+- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
+- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
+
+**Why These Configurations Matter:**
+
+- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
+- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
+- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
+- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
+
+**Interpreting Results:**
+
+- Compare your results against baseline numbers for your hardware
+- Higher throughput at same latency = better performance
+- Lower TTFT = more responsive user experience
+- Lower TPOT = faster generation speed
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+```bash Command
+python -m sglang.test.few_shot_gsm8k \
+  --num-questions 200 \
+  --port 30000
+```

docs_new/cookbook/autoregressive/GLM/GLM-4.5V.mdx

@@ -0,0 +1,533 @@
+---
+title: GLM-4.5V
+metatags:
+    description: "Deploy GLM-4.5V vision-language model with SGLang - SOTA multimodal performance, 64K context, image reasoning and video understanding."
+---
+
+## 1. Model Introduction
+
+[GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types.
+
+**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
+
+GLM-4.5V introduces several key features:
+
+- **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection.
+- **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences.
+- **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces.
+- **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context.
+- **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
+
+**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
+
+import { GLM45VDeployment } from "/src/snippets/autoregressive/glm-45v-deployment.jsx";
+
+<GLM45VDeployment />
+
+### 3.2 Configuration Tips
+- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
+- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`.
+- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
+- For more detailed configuration tips, please refer to [GLM-4.5V/GLM-4.6V Usage](../../../docs/basic_usage/glmv).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Multi-Modal Inputs
+
+GLM-4.5V supports both image and video inputs. Here's a basic example with image input:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Describe this image in detail."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5V",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 3.37s
+Generated text: Auntie Anne's
+
+CINNAMON SUGAR
+1 x 17,000                    17,000
+
+SUB TOTAL                    17,000
+
+GRAND TOTAL                  17,000
+
+CASH IDR                     20,000
+
+CHANGE DUE                  3,000
+```
+
+**Multi-Image Input Example:**
+
+GLM-4.5V can process multiple images in a single request for comparison or analysis:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
+                }
+            },
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5V",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 3.86s
+Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot).
+```
+
+**Video Input Example:**
+
+GLM-4.5V supports video understanding by processing video URLs:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "video_url",
+                "video_url": {
+                    "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Describe what happens in this video."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5V",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Note:**
+
+- For video processing, ensure you have sufficient context length configured (up to 64K tokens)
+- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
+- You can also provide local file paths using `file://` protocol
+
+**Example Output:**
+
+```text Output
+Response costs: 3.89s
+Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
+```
+
+#### 4.2.2 Thinking Mode
+
+GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path zai-org/GLM-4.5V \
+  --reasoning-parser glm45 \
+  --tp 4 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5V",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+**Disable Thinking Mode:**
+
+To disable thinking mode for a specific request:
+
+```python Example
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5V",
+    messages=[{"role": "user", "content": "What is the capital of France?"}],
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+```
+
+#### 4.2.3 Tool Calling
+
+GLM-4.5V supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path zai-org/GLM-4.5V \
+  --reasoning-parser glm45 \
+  --tool-call-parser glm45 \
+  --tp 4 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.5V",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="zai-org/GLM-4.5V",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+### 5.1 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.1.1 MMMU Benchmark
+
+- Benchmark Command
+
+```bash Command
+python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64
+```
+
+- Test Result
+
+```text Output
+Benchmark time: 616.6163094160147
+answers saved to: ./answer_sglang.json
+Evaluating...
+answers saved to: ./answer_sglang.json
+{'Accounting': {'acc': 0.867, 'num': 30},
+ 'Agriculture': {'acc': 0.567, 'num': 30},
+ 'Architecture_and_Engineering': {'acc': 0.667, 'num': 30},
+ 'Art': {'acc': 0.667, 'num': 30},
+ 'Art_Theory': {'acc': 0.9, 'num': 30},
+ 'Basic_Medical_Science': {'acc': 0.8, 'num': 30},
+ 'Biology': {'acc': 0.6, 'num': 30},
+ 'Chemistry': {'acc': 0.533, 'num': 30},
+ 'Clinical_Medicine': {'acc': 0.667, 'num': 30},
+ 'Computer_Science': {'acc': 0.8, 'num': 30},
+ 'Design': {'acc': 0.867, 'num': 30},
+ 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30},
+ 'Economics': {'acc': 0.833, 'num': 30},
+ 'Electronics': {'acc': 0.433, 'num': 30},
+ 'Energy_and_Power': {'acc': 0.733, 'num': 30},
+ 'Finance': {'acc': 0.767, 'num': 30},
+ 'Geography': {'acc': 0.667, 'num': 30},
+ 'History': {'acc': 0.8, 'num': 30},
+ 'Literature': {'acc': 0.9, 'num': 30},
+ 'Manage': {'acc': 0.733, 'num': 30},
+ 'Marketing': {'acc': 0.9, 'num': 30},
+ 'Materials': {'acc': 0.567, 'num': 30},
+ 'Math': {'acc': 0.8, 'num': 30},
+ 'Mechanical_Engineering': {'acc': 0.767, 'num': 30},
+ 'Music': {'acc': 0.3, 'num': 30},
+ 'Overall': {'acc': 0.732, 'num': 900},
+ 'Overall-Art and Design': {'acc': 0.683, 'num': 120},
+ 'Overall-Business': {'acc': 0.82, 'num': 150},
+ 'Overall-Health and Medicine': {'acc': 0.787, 'num': 150},
+ 'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120},
+ 'Overall-Science': {'acc': 0.707, 'num': 150},
+ 'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210},
+ 'Pharmacy': {'acc': 0.9, 'num': 30},
+ 'Physics': {'acc': 0.933, 'num': 30},
+ 'Psychology': {'acc': 0.767, 'num': 30},
+ 'Public_Health': {'acc': 0.9, 'num': 30},
+ 'Sociology': {'acc': 0.667, 'num': 30}}
+eval out saved to ./val_sglang.json
+Overall accuracy: 0.732
+```

docs_new/cookbook/autoregressive/GLM/GLM-4.6.mdx

@@ -0,0 +1,888 @@
+---
+title: GLM-4.6
+metatags:
+    description: "Deploy GLM-4.6 with SGLang - 200K context window, superior coding, advanced reasoning, and enhanced agentic capabilities."
+---
+
+## 1. Model Introduction
+
+[GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
+
+As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows:
+
+- **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks.
+- **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages.
+- **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability.
+- **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks.
+- **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios.
+
+For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
+
+import { GLM46Deployment } from "/src/snippets/autoregressive/glm-46-deployment.jsx";
+
+<GLM46Deployment />
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [GLM-4.5/GLM-4.6 Usage](../../../docs/basic_usage/glm45).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.6 \
+  --reasoning-parser glm45 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.6",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To solve this problem, I need to calculate 15% of 240.
+Step 1: Convert 15% to decimal: 15% = 0.15
+Step 2: Multiply 240 by 0.15
+Step 3: 240 × 0.15 = 36
+=============== Content =================
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+GLM-4.6 supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.6 \
+  --reasoning-parser glm45 \
+  --tool-call-parser glm45 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.6",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"🔧 Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="zai-org/GLM-4.6",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
+- Model: GLM-4.6
+- Tensor Parallelism: 8
+- SGLang Version: 0.5.6.post1
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Test Scenarios
+
+Three core scenarios reflect real-world usage patterns:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 Concurrency Levels
+
+Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
+
+- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
+- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
+- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
+
+#### 5.1.3 Number of Prompts
+
+For each concurrency level, configure `num_prompts` to simulate realistic user loads:
+
+- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
+- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
+- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
+
+---
+
+#### 5.1.4 Benchmark Commands
+
+**Scenario 1: Chat (1K/1K) - Most Important**
+
+- **Model Deployment**
+```bash Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.6 \
+  --tp 8
+```
+
+
+- Low Concurrency (Latency-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  63.82
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4210
+Total generated tokens (retokenized):    4209
+Request throughput (req/s):              0.16
+Input token throughput (tok/s):          95.60
+Output token throughput (tok/s):         65.97
+Peak output token throughput (tok/s):    68.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          161.57
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6379.24
+Median E2E Latency (ms):                 5085.00
+---------------Time to First Token----------------
+Mean TTFT (ms):                          155.57
+Median TTFT (ms):                        149.79
+P99 TTFT (ms):                           207.69
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          14.81
+Median TPOT (ms):                        14.80
+P99 TPOT (ms):                           14.84
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           14.82
+Median ITL (ms):                         14.82
+P95 ITL (ms):                            15.17
+P99 ITL (ms):                            15.36
+Max ITL (ms):                            25.05
+==================================================
+```
+
+
+- Medium Concurrency (Balanced)
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+```text Output
+
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  72.06
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40725
+Total generated tokens (retokenized):    40672
+Request throughput (req/s):              1.11
+Input token throughput (tok/s):          550.47
+Output token throughput (tok/s):         565.14
+Peak output token throughput (tok/s):    752.00
+Peak concurrent requests:                20
+Total token throughput (tok/s):          1115.61
+Concurrency:                             13.71
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   12348.93
+Median E2E Latency (ms):                 13164.81
+---------------Time to First Token----------------
+Mean TTFT (ms):                          196.08
+Median TTFT (ms):                        155.22
+P99 TTFT (ms):                           377.98
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          24.24
+Median TPOT (ms):                        24.55
+P99 TPOT (ms):                           30.42
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           23.92
+Median ITL (ms):                         21.40
+P95 ITL (ms):                            22.49
+P99 ITL (ms):                            123.83
+Max ITL (ms):                            486.54
+==================================================
+```
+
+
+- High Concurrency (Throughput-Optimized)
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  138.50
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252162
+Total generated tokens (retokenized):    251841
+Request throughput (req/s):              3.61
+Input token throughput (tok/s):          1803.78
+Output token throughput (tok/s):         1820.61
+Peak output token throughput (tok/s):    2900.00
+Peak concurrent requests:                107
+Total token throughput (tok/s):          3624.40
+Concurrency:                             90.91
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   25183.97
+Median E2E Latency (ms):                 23968.49
+---------------Time to First Token----------------
+Mean TTFT (ms):                          337.77
+Median TTFT (ms):                        180.65
+P99 TTFT (ms):                           906.14
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          49.97
+Median TPOT (ms):                        52.20
+P99 TPOT (ms):                           61.81
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           49.36
+Median ITL (ms):                         35.05
+P95 ITL (ms):                            124.91
+P99 ITL (ms):                            187.69
+Max ITL (ms):                            440.34
+==================================================
+```
+
+**Scenario 2: Reasoning (1K/8K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  666.64
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  44452
+Total generated tokens (retokenized):    44387
+Request throughput (req/s):              0.02
+Input token throughput (tok/s):          9.15
+Output token throughput (tok/s):         66.68
+Peak output token throughput (tok/s):    68.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          75.83
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   66661.35
+Median E2E Latency (ms):                 71902.36
+---------------Time to First Token----------------
+Mean TTFT (ms):                          160.21
+Median TTFT (ms):                        140.32
+P99 TTFT (ms):                           295.56
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          14.92
+Median TPOT (ms):                        14.94
+P99 TPOT (ms):                           15.02
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           14.96
+Median ITL (ms):                         14.96
+P95 ITL (ms):                            15.36
+P99 ITL (ms):                            15.57
+Max ITL (ms):                            19.06
+==================================================
+```
+
+- Medium Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  503.30
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  318226
+Total generated tokens (retokenized):    318025
+Request throughput (req/s):              0.16
+Input token throughput (tok/s):          78.82
+Output token throughput (tok/s):         632.28
+Peak output token throughput (tok/s):    752.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          711.09
+Concurrency:                             13.88
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   87349.22
+Median E2E Latency (ms):                 88248.04
+---------------Time to First Token----------------
+Mean TTFT (ms):                          228.54
+Median TTFT (ms):                        142.78
+P99 TTFT (ms):                           569.84
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          21.97
+Median TPOT (ms):                        22.14
+P99 TPOT (ms):                           22.47
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           21.91
+Median ITL (ms):                         21.80
+P95 ITL (ms):                            22.30
+P99 ITL (ms):                            22.78
+Max ITL (ms):                            137.19
+==================================================
+```
+
+- High Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  772.28
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total input vision tokens:               0
+Total generated tokens:                  1300705
+Total generated tokens (retokenized):    1299924
+Request throughput (req/s):              0.41
+Input token throughput (tok/s):          205.80
+Output token throughput (tok/s):         1684.24
+Peak output token throughput (tok/s):    2112.00
+Peak concurrent requests:                68
+Total token throughput (tok/s):          1890.05
+Concurrency:                             56.17
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   135563.36
+Median E2E Latency (ms):                 140888.88
+---------------Time to First Token----------------
+Mean TTFT (ms):                          232.45
+Median TTFT (ms):                        145.59
+P99 TTFT (ms):                           576.49
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          33.47
+Median TPOT (ms):                        34.02
+P99 TPOT (ms):                           35.10
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           33.30
+Median ITL (ms):                         32.63
+P95 ITL (ms):                            34.27
+P99 ITL (ms):                            104.39
+Max ITL (ms):                            155.65
+==================================================
+```
+
+**Scenario 3: Summarization (8K/1K)**
+
+- Low
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  65.11
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total input vision tokens:               0
+Total generated tokens:                  4210
+Total generated tokens (retokenized):    4210
+Request throughput (req/s):              0.15
+Input token throughput (tok/s):          644.17
+Output token throughput (tok/s):         64.66
+Peak output token throughput (tok/s):    68.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          708.83
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6508.31
+Median E2E Latency (ms):                 5263.36
+---------------Time to First Token----------------
+Mean TTFT (ms):                          189.48
+Median TTFT (ms):                        159.23
+P99 TTFT (ms):                           304.09
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          15.02
+Median TPOT (ms):                        15.03
+P99 TPOT (ms):                           15.27
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           15.04
+Median ITL (ms):                         15.03
+P95 ITL (ms):                            15.46
+P99 ITL (ms):                            15.65
+Max ITL (ms):                            24.20
+==================================================
+```
+
+- Medium Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  76.43
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total input vision tokens:               0
+Total generated tokens:                  41589
+Total generated tokens (retokenized):    41577
+Request throughput (req/s):              1.05
+Input token throughput (tok/s):          3925.47
+Output token throughput (tok/s):         544.15
+Peak output token throughput (tok/s):    752.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          4469.62
+Concurrency:                             13.95
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   13329.63
+Median E2E Latency (ms):                 14141.09
+---------------Time to First Token----------------
+Mean TTFT (ms):                          339.88
+Median TTFT (ms):                        252.75
+P99 TTFT (ms):                           906.54
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          25.37
+Median TPOT (ms):                        25.73
+P99 TPOT (ms):                           30.94
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           25.04
+Median ITL (ms):                         21.68
+P95 ITL (ms):                            22.69
+P99 ITL (ms):                            146.98
+Max ITL (ms):                            483.14
+==================================================
+```
+
+
+- High Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.6 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  136.24
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total input vision tokens:               0
+Total generated tokens:                  169680
+Total generated tokens (retokenized):    169452
+Request throughput (req/s):              2.35
+Input token throughput (tok/s):          9350.32
+Output token throughput (tok/s):         1245.44
+Peak output token throughput (tok/s):    1984.00
+Peak concurrent requests:                69
+Total token throughput (tok/s):          10595.77
+Concurrency:                             58.46
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   24889.40
+Median E2E Latency (ms):                 25123.37
+---------------Time to First Token----------------
+Mean TTFT (ms):                          355.82
+Median TTFT (ms):                        268.84
+P99 TTFT (ms):                           858.64
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          46.62
+Median TPOT (ms):                        49.04
+P99 TPOT (ms):                           58.88
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           46.36
+Median ITL (ms):                         32.46
+P95 ITL (ms):                            135.23
+P99 ITL (ms):                            204.27
+Max ITL (ms):                            508.14
+==================================================
+```
+
+#### 5.1.5 Understanding the Results
+
+**Key Metrics:**
+
+- **Request Throughput (req/s)**: Number of requests processed per second
+- **Output Token Throughput (tok/s)**: Total tokens generated per second
+- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
+- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
+- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
+
+**Why These Configurations Matter:**
+
+- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
+- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
+- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
+- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
+
+**Interpreting Results:**
+
+- Compare your results against baseline numbers for your hardware
+- Higher throughput at same latency = better performance
+- Lower TTFT = more responsive user experience
+- Lower TPOT = faster generation speed
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+```bash Command
+python -m sglang.test.few_shot_gsm8k \
+  --num-questions 200 \
+  --port 30000
+```
+
+- Test Result
+```text Output
+Accuracy: 0.975
+Invalid: 0.000
+Latency: 16.574 s
+Output throughput: 1194.637 token/s
+```

docs_new/cookbook/autoregressive/GLM/GLM-4.6V.mdx

@@ -0,0 +1,382 @@
+---
+title: GLM-4.6V
+metatags:
+    description: "Deploy GLM-4.6V vision-language model with SGLang - native function calling, 128K context, multimodal document understanding and frontend replication."
+---
+
+## 1. Model Introduction
+
+GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios.
+
+Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features:
+
+- **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#tool-call-example).
+- **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content.
+- **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text.
+- **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+### 2.1 Docker Installation (Recommended)
+
+```shell Command
+docker pull lmsysorg/sglang:latest
+```
+
+**Advantages:**
+
+- Ready to use out of the box, no manual environment configuration needed
+- Avoids dependency conflict issues
+- Easy to migrate between different environments
+
+### 2.2 Build from Source
+
+If you need to use the latest development version or require custom modifications, you can build from source:
+
+```bash Command
+# Install SGLang using UV (recommended)
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+uv venv
+source .venv/bin/activate
+uv pip install -e "python[all]" --index-url=https://pypi.org/simple
+pip install nvidia-cudnn-cu12==9.16.0.29
+# Install ffmpeg to support video input
+sudo apt update
+sudo apt install ffmpeg
+```
+
+**Use Cases:**
+
+- Need to customize and modify SGLang source code
+- Want to use the latest development features
+- Participate in SGLang project development
+
+For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/installation).
+
+## 3. Model Deployment
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
+
+import { GLM46VDeployment } from "/src/snippets/autoregressive/glm-46v-deployment.jsx";
+
+<GLM46VDeployment />
+
+### 3.2 Configuration Tips
+- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
+- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically).
+- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
+- For more detailed configuration tips, please refer to [GLM-4.5V/GLM-4.6V Usage](../../../docs/basic_usage/glmv).
+
+## 4. Example APIs
+
+### Image Input Example
+
+#### API Payload
+```python Example
+curl_command = f"""
+curl -s http://localhost:{30000}/v1/chat/completions \\
+  -H "Content-Type: application/json" \\
+  -d '{{
+    "model": "default",
+    "messages": [
+      {{
+        "role": "user",
+        "content": [
+          {{
+            "type": "image_url",
+            "image_url": {{
+              "url": "/home/jobuser/sgl_logo.png"
+            }}
+          }},
+          {{
+            "type": "text",
+            "text": "What is the image"
+          }}
+        ]
+      }}
+    ],
+    "temperature": "0",
+    "max_completion_tokens": "1000",
+    "max_tokens": "1000"
+  }}'
+"""
+
+response = subprocess.check_output(curl_command, shell=True).decode()
+print(response)
+```
+
+#### API Response
+```text Output
+{"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"</>\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"</>\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
+```
+
+### Video Input Example
+
+#### API Payload
+```python Example
+curl_command = f"""
+curl -s http://localhost:{30000}/v1/chat/completions \\
+  -H "Content-Type: application/json" \\
+  -d '{{
+    "model": "default",
+    "messages": [
+      {{
+        "role": "user",
+        "content": [
+          {{
+            "type": "video_url",
+            "video_url": {{
+              "url": "/home/jobuser/jobs_presenting_ipod.mp4"
+            }}
+          }},
+          {{
+            "type": "text",
+            "text": "What is the image"
+          }}
+        ]
+      }}
+    ],
+    "temperature": "0",
+    "max_completion_tokens": "1000",
+    "max_tokens": "1000"
+  }}'
+"""
+
+response = subprocess.check_output(curl_command, shell=True).decode()
+print(response)
+```
+
+#### API Response
+```text Output
+{"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n*   **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n*   **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n*   **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n*   **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
+```
+
+### Tool Call Example
+
+#### API Payload
+```python Example
+from openai import OpenAI
+import argparse
+import sys
+import base64
+
+def image_to_base64(image_path):
+    """Convert image file to base64 data URL format for OpenAI API"""
+    with open(image_path, 'rb') as image_file:
+        image_data = image_file.read()
+        base64_string = base64.b64encode(image_data).decode('utf-8')
+        return f"data:image/png;base64,{base64_string}"
+
+openai_api_key = "EMPTY"
+openai_api_base = "http://127.0.0.1:30000/v1"
+client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)
+
+
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get current temperature for a given location.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "City and country e.g. Beijing, China",
+                    }
+                },
+                "required": ["location"],
+                "additionalProperties": False,
+            },
+        },
+    }
+]
+
+
+messages = [
+    {
+        "role": "user",
+        "content": "Please help me check today’s weather in Beijing, and tell me whether the tool returned an image."
+    },
+    {
+        "role": "assistant",
+        "tool_calls": [
+            {
+                "id": "call_bk32t88BGpSdbtDgzT044Rh4",
+                "type": "function",
+                "function": {
+                    "name": 'get_weather',
+                    "arguments": '{"location":"Beijing, China"}'
+                }
+            }
+        ]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4",
+        "content": [
+            {
+                "type": "text",
+                "text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C."
+            },
+            {
+                "type": "image_url",
+                "image_url": {
+                     "url": "/home/jobuser/sgl_logo.png"
+                }
+            }
+        ]
+    },
+]
+
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.6V",
+    messages=messages,
+    timeout=900,
+    tools=tools
+)
+print(response.choices[0].message.content.strip())
+```
+
+#### Output
+
+```text Output
+The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C.
+
+Yes, the tool returned an image (the SGL logo).
+```
+
+## 5. Benchmark
+
+### 5.1. Text Benchmark: Latency, Throughput and Accuracy
+
+#### Command
+```shell Command
+python3 ./benchmark/gsm8k/bench_sglang.py
+```
+#### Result Output
+```text Output
+Accuracy: 0.925
+Invalid: 0.000
+Latency: 15.327 s
+Output throughput: 1788.375 token/s
+```
+
+### 5.2. Multimodal Benchmark - Latency and Throughput
+
+#### Command
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --port 30000 \
+  --model zai-org/GLM-4.6V \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 128 \
+  --max-concurrency 8
+```
+
+#### Result Output
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 8
+Successful requests:                     128
+Benchmark duration (s):                  89.27
+Total input tokens:                      315390
+Total input text tokens:                 8702
+Total input vision tokens:               306688
+Total generated tokens:                  66020
+Total generated tokens (retokenized):    31037
+Request throughput (req/s):              1.43
+Input token throughput (tok/s):          3533.17
+Output token throughput (tok/s):         739.59
+Peak output token throughput (tok/s):    823.00
+Peak concurrent requests:                12
+Total token throughput (tok/s):          4272.76
+Concurrency:                             7.67
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5349.20
+Median E2E Latency (ms):                 5380.98
+---------------Time to First Token----------------
+Mean TTFT (ms):                          1724.04
+Median TTFT (ms):                        1688.16
+P99 TTFT (ms):                           6152.34
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          8.15
+Median TPOT (ms):                        7.77
+P99 TPOT (ms):                           23.97
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           10.00
+Median ITL (ms):                         8.44
+P95 ITL (ms):                            9.23
+P99 ITL (ms):                            116.02
+Max ITL (ms):                            173.48
+==================================================
+```
+
+
+### 5.3. Multimodal Accuracy Benchmark - MMMU
+
+#### Command
+```shell Command
+python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}'
+```
+
+#### Result Output
+```text Output
+Benchmark time: 487.2229107860476
+answers saved to: ./answer_sglang.json
+Evaluating...
+answers saved to: ./answer_sglang.json
+{'Accounting': {'acc': 0.962, 'num': 26},
+ 'Agriculture': {'acc': 0.5, 'num': 30},
+ 'Architecture_and_Engineering': {'acc': 0.733, 'num': 15},
+ 'Art': {'acc': 0.833, 'num': 30},
+ 'Art_Theory': {'acc': 0.9, 'num': 30},
+ 'Basic_Medical_Science': {'acc': 0.733, 'num': 30},
+ 'Biology': {'acc': 0.586, 'num': 29},
+ 'Chemistry': {'acc': 0.654, 'num': 26},
+ 'Clinical_Medicine': {'acc': 0.633, 'num': 30},
+ 'Computer_Science': {'acc': 0.76, 'num': 25},
+ 'Design': {'acc': 0.867, 'num': 30},
+ 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30},
+ 'Economics': {'acc': 0.862, 'num': 29},
+ 'Electronics': {'acc': 0.5, 'num': 18},
+ 'Energy_and_Power': {'acc': 0.875, 'num': 16},
+ 'Finance': {'acc': 0.857, 'num': 28},
+ 'Geography': {'acc': 0.714, 'num': 28},
+ 'History': {'acc': 0.767, 'num': 30},
+ 'Literature': {'acc': 0.897, 'num': 29},
+ 'Manage': {'acc': 0.759, 'num': 29},
+ 'Marketing': {'acc': 1.0, 'num': 26},
+ 'Materials': {'acc': 0.833, 'num': 18},
+ 'Math': {'acc': 0.76, 'num': 25},
+ 'Mechanical_Engineering': {'acc': 0.619, 'num': 21},
+ 'Music': {'acc': 0.286, 'num': 28},
+ 'Overall': {'acc': 0.761, 'num': 803},
+ 'Overall-Art and Design': {'acc': 0.729, 'num': 118},
+ 'Overall-Business': {'acc': 0.884, 'num': 138},
+ 'Overall-Health and Medicine': {'acc': 0.773, 'num': 150},
+ 'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118},
+ 'Overall-Science': {'acc': 0.728, 'num': 136},
+ 'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143},
+ 'Pharmacy': {'acc': 0.933, 'num': 30},
+ 'Physics': {'acc': 0.929, 'num': 28},
+ 'Psychology': {'acc': 0.733, 'num': 30},
+ 'Public_Health': {'acc': 0.933, 'num': 30},
+ 'Sociology': {'acc': 0.724, 'num': 29}}
+eval out saved to ./val_sglang.json
+Overall accuracy: 0.761
+```

docs_new/cookbook/autoregressive/GLM/GLM-4.7-Flash.mdx

@@ -0,0 +1,931 @@
+---
+title: GLM-4.7-Flash
+metatags:
+    description: "Deploy GLM-4.7-Flash 30B-A3B MoE model with SGLang - lightweight, efficient inference optimized for single-GPU deployment."
+---
+
+## 1. Model Introduction
+
+[GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment.
+
+As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency:
+
+- **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference
+- **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance
+- **Superior Coding**: Strong code generation and understanding capabilities
+- **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows
+- **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios
+
+For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
+
+**Key Features:**
+
+- **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off
+- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
+- **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs
+- **High Performance**: Optimized for both throughput and latency scenarios
+
+**Available Models:**
+
+- **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
+
+**License:**
+
+Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
+
+import { GLM47FlashDeployment } from "/src/snippets/autoregressive/glm-47-flash-deployment.jsx";
+
+<GLM47FlashDeployment />
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [GLM-4.7 Usage](../../../docs/basic_usage/glm45).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.7-Flash \
+  --reasoning-parser glm45 \
+  --attention-backend triton \
+  --tp 1 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.7-Flash",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To solve this problem, I need to calculate 15% of 240.
+Step 1: Convert 15% to decimal: 15% = 0.15
+Step 2: Multiply 240 by 0.15
+Step 3: 240 × 0.15 = 36
+=============== Content =================
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.7-Flash \
+  --reasoning-parser glm45 \
+  --tool-call-parser glm47 \
+  --attention-backend triton \
+  --tp 1 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.7-Flash",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls (tool call deltas may stream in multiple chunks)
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+if tool_calls_accumulator:
+    print("\n=============== Tool Calls =================", flush=True)
+    for index, tool_call in sorted(tool_calls_accumulator.items()):
+        print(f"Tool Call: {tool_call['name']}")
+        print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the
+ user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you.
+=============== Tool Calls =================
+Tool Call: get_weather
+   Arguments: {"location": "Beijing"}
+
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="zai-org/GLM-4.7-Flash",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 (1x)
+- Model: GLM-4.7-Flash
+- Tensor Parallelism: 1
+- SGLang Version: 0.5.7
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Test Scenarios
+
+Three core scenarios reflect real-world usage patterns:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 Concurrency Levels
+
+Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
+
+- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
+- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
+- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
+
+#### 5.1.3 Number of Prompts
+
+For each concurrency level, configure `num_prompts` to simulate realistic user loads:
+
+- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
+- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
+- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
+
+---
+
+#### 5.1.4 Benchmark Commands
+
+**Scenario 1: Chat (1K/1K) - Most Important**
+
+- **Model Deployment**
+
+```bash Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.7-Flash \
+  --attention-backend triton \
+  --tp 1
+```
+
+- Low Concurrency (Latency-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  38.94
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4220
+Request throughput (req/s):              0.26
+Input token throughput (tok/s):          156.67
+Output token throughput (tok/s):         108.37
+Peak output token throughput (tok/s):    125.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          265.03
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3891.12
+Median E2E Latency (ms):                 3061.48
+P90 E2E Latency (ms):                    7172.25
+P99 E2E Latency (ms):                    9042.62
+---------------Time to First Token----------------
+Mean TTFT (ms):                          131.36
+Median TTFT (ms):                        94.55
+P99 TTFT (ms):                           435.93
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          8.75
+Median TPOT (ms):                        8.82
+P99 TPOT (ms):                           9.39
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           8.93
+Median ITL (ms):                         8.98
+P95 ITL (ms):                            9.83
+P99 ITL (ms):                            10.20
+Max ITL (ms):                            18.50
+==================================================
+```
+
+- Medium Concurrency (Balanced)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  52.73
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40775
+Request throughput (req/s):              1.52
+Input token throughput (tok/s):          752.27
+Output token throughput (tok/s):         773.83
+Peak output token throughput (tok/s):    1040.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          1526.10
+Concurrency:                             13.98
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   9217.90
+Median E2E Latency (ms):                 9642.50
+P90 E2E Latency (ms):                    15147.02
+P99 E2E Latency (ms):                    18237.06
+---------------Time to First Token----------------
+Mean TTFT (ms):                          299.02
+Median TTFT (ms):                        105.98
+P99 TTFT (ms):                           1109.29
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          18.03
+Median TPOT (ms):                        18.00
+P99 TPOT (ms):                           26.51
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           17.52
+Median ITL (ms):                         16.07
+P95 ITL (ms):                            18.14
+P99 ITL (ms):                            89.43
+Max ITL (ms):                            763.13
+==================================================
+```
+
+- High Concurrency (Throughput-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  91.48
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    250941
+Request throughput (req/s):              5.47
+Input token throughput (tok/s):          2730.87
+Output token throughput (tok/s):         2761.82
+Peak output token throughput (tok/s):    4199.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          5492.69
+Concurrency:                             90.54
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   16566.04
+Median E2E Latency (ms):                 16134.36
+P90 E2E Latency (ms):                    30167.60
+P99 E2E Latency (ms):                    34034.04
+---------------Time to First Token----------------
+Mean TTFT (ms):                          433.94
+Median TTFT (ms):                        123.26
+P99 TTFT (ms):                           1760.09
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          32.26
+Median TPOT (ms):                        33.56
+P99 TPOT (ms):                           38.78
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           31.99
+Median ITL (ms):                         24.06
+P95 ITL (ms):                            79.62
+P99 ITL (ms):                            103.03
+Max ITL (ms):                            1369.20
+==================================================
+```
+
+
+**Scenario 2: Reasoning (1K/8K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  525.43
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  44462
+Total generated tokens (retokenized):    44451
+Request throughput (req/s):              0.02
+Input token throughput (tok/s):          11.61
+Output token throughput (tok/s):         84.62
+Peak output token throughput (tok/s):    125.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          96.23
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   52540.19
+Median E2E Latency (ms):                 53694.45
+P90 E2E Latency (ms):                    94742.08
+P99 E2E Latency (ms):                    101224.18
+---------------Time to First Token----------------
+Mean TTFT (ms):                          97.45
+Median TTFT (ms):                        95.28
+P99 TTFT (ms):                           105.64
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.94
+Median TPOT (ms):                        11.25
+P99 TPOT (ms):                           13.09
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.80
+Median ITL (ms):                         11.51
+P95 ITL (ms):                            15.83
+P99 ITL (ms):                            16.86
+Max ITL (ms):                            19.96
+==================================================
+```
+
+- Medium Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  473.92
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total generated tokens:                  318306
+Total generated tokens (retokenized):    317860
+Request throughput (req/s):              0.17
+Input token throughput (tok/s):          83.70
+Output token throughput (tok/s):         671.65
+Peak output token throughput (tok/s):    1040.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          755.35
+Concurrency:                             13.80
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   81746.73
+Median E2E Latency (ms):                 78508.54
+P90 E2E Latency (ms):                    155292.49
+P99 E2E Latency (ms):                    166769.99
+---------------Time to First Token----------------
+Mean TTFT (ms):                          117.50
+Median TTFT (ms):                        101.97
+P99 TTFT (ms):                           182.88
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          20.36
+Median TPOT (ms):                        20.48
+P99 TPOT (ms):                           22.63
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           20.52
+Median ITL (ms):                         20.42
+P95 ITL (ms):                            23.41
+P99 ITL (ms):                            26.29
+Max ITL (ms):                            90.48
+==================================================
+```
+
+- High Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  714.72
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total generated tokens:                  1301025
+Total generated tokens (retokenized):    1289431
+Request throughput (req/s):              0.45
+Input token throughput (tok/s):          222.38
+Output token throughput (tok/s):         1820.33
+Peak output token throughput (tok/s):    3200.00
+Peak concurrent requests:                68
+Total token throughput (tok/s):          2042.71
+Concurrency:                             55.68
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   124364.58
+Median E2E Latency (ms):                 129250.98
+P90 E2E Latency (ms):                    219175.80
+P99 E2E Latency (ms):                    247741.77
+---------------Time to First Token----------------
+Mean TTFT (ms):                          149.40
+Median TTFT (ms):                        114.78
+P99 TTFT (ms):                           288.60
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          30.51
+Median TPOT (ms):                        31.75
+P99 TPOT (ms):                           33.32
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           30.56
+Median ITL (ms):                         30.82
+P95 ITL (ms):                            33.20
+P99 ITL (ms):                            80.54
+Max ITL (ms):                            117.72
+==================================================
+```
+
+**Scenario 3: Summarization (8K/1K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  58.27
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4220
+Request throughput (req/s):              0.17
+Input token throughput (tok/s):          719.73
+Output token throughput (tok/s):         72.42
+Peak output token throughput (tok/s):    112.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          792.15
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5825.08
+Median E2E Latency (ms):                 4624.26
+P90 E2E Latency (ms):                    12690.22
+P99 E2E Latency (ms):                    13177.96
+---------------Time to First Token----------------
+Mean TTFT (ms):                          296.01
+Median TTFT (ms):                        195.59
+P99 TTFT (ms):                           717.88
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          12.63
+Median TPOT (ms):                        13.07
+P99 TPOT (ms):                           16.68
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           13.13
+Median ITL (ms):                         13.17
+P95 ITL (ms):                            17.02
+P99 ITL (ms):                            17.47
+Max ITL (ms):                            19.84
+==================================================
+```
+
+- Medium Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  89.59
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total generated tokens:                  41669
+Total generated tokens (retokenized):    41656
+Request throughput (req/s):              0.89
+Input token throughput (tok/s):          3348.77
+Output token throughput (tok/s):         465.10
+Peak output token throughput (tok/s):    752.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          3813.87
+Concurrency:                             14.39
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   16120.74
+Median E2E Latency (ms):                 16246.55
+P90 E2E Latency (ms):                    27279.72
+P99 E2E Latency (ms):                    34577.93
+---------------Time to First Token----------------
+Mean TTFT (ms):                          1943.94
+Median TTFT (ms):                        382.19
+P99 TTFT (ms):                           8980.41
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          27.87
+Median TPOT (ms):                        28.26
+P99 TPOT (ms):                           40.55
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           27.27
+Median ITL (ms):                         21.74
+P95 ITL (ms):                            23.32
+P99 ITL (ms):                            232.65
+Max ITL (ms):                            4282.01
+==================================================
+```
+
+- High Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7-Flash \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  167.01
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total generated tokens:                  170000
+Total generated tokens (retokenized):    169226
+Request throughput (req/s):              1.92
+Input token throughput (tok/s):          7627.82
+Output token throughput (tok/s):         1017.93
+Peak output token throughput (tok/s):    1984.00
+Peak concurrent requests:                69
+Total token throughput (tok/s):          8645.75
+Concurrency:                             59.68
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   31147.52
+Median E2E Latency (ms):                 30603.34
+P90 E2E Latency (ms):                    54889.44
+P99 E2E Latency (ms):                    67665.30
+---------------Time to First Token----------------
+Mean TTFT (ms):                          428.87
+Median TTFT (ms):                        441.69
+P99 TTFT (ms):                           1232.68
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          58.06
+Median TPOT (ms):                        62.79
+P99 TPOT (ms):                           82.23
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           57.93
+Median ITL (ms):                         33.30
+P95 ITL (ms):                            247.98
+P99 ITL (ms):                            409.63
+Max ITL (ms):                            1421.21
+==================================================
+```
+
+#### 5.1.5 Understanding the Results
+
+**Key Metrics:**
+
+- **Request Throughput (req/s)**: Number of requests processed per second
+- **Output Token Throughput (tok/s)**: Total tokens generated per second
+- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
+- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
+- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
+
+**Why These Configurations Matter:**
+
+- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
+- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
+- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
+- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
+
+**Interpreting Results:**
+
+- Compare your results against baseline numbers for your hardware
+- Higher throughput at same latency = better performance
+- Lower TTFT = more responsive user experience
+- Lower TPOT = faster generation speed
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+
+```bash Command
+python -m sglang.test.few_shot_gsm8k \
+  --num-questions 200 \
+  --port 30000
+```
+
+- Result
+
+```text Output
+Accuracy: 0.845
+Invalid: 0.000
+Latency: 8.431 s
+Output throughput: 2195.387 token/s
+```

docs_new/cookbook/autoregressive/GLM/GLM-4.7.mdx

@@ -0,0 +1,546 @@
+---
+title: GLM-4.7
+metatags:
+    description: "Deploy GLM-4.7 with SGLang on AMD GPUs - state-of-the-art reasoning, enhanced coding, and robust tool calling capabilities."
+---
+
+## 1. Model Introduction
+
+[GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is the latest and most powerful language model in the GLM series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and multi-modal understanding.
+
+As the newest iteration in the GLM series, GLM-4.7 achieves significant improvements across all domains:
+
+- **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations
+- **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing
+- **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance
+- **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows
+- **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms
+
+For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
+
+**Key Features:**
+
+- **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks
+- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
+- **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
+- **High Performance**: Optimized for both throughput and latency scenarios
+
+**Available Models:**
+
+- **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) - Recommended for MI300X/MI325X/MI355X
+- **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8) - Recommended for MI300X/MI325X/MI355X
+
+**License:**
+
+Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
+
+import { GLM47Deployment } from "/src/snippets/autoregressive/glm-47-deployment.jsx";
+
+<GLM47Deployment />
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [GLM-4.7 Usage](../../../docs/basic_usage/glm45).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.7 \
+  --reasoning-parser glm47 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.7",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To solve this problem, I need to calculate 15% of 240.
+Step 1: Convert 15% to decimal: 15% = 0.15
+Step 2: Multiply 240 by 0.15
+Step 3: 240 × 0.15 = 36
+=============== Content =================
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+GLM-4.7 supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.7 \
+  --reasoning-parser glm47 \
+  --tool-call-parser glm47 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/GLM-4.7",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="zai-org/GLM-4.7",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
+- Model: GLM-4.7
+- Tensor Parallelism: 8
+- SGLang Version: 0.5.6.post1
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Test Scenarios
+
+Three core scenarios reflect real-world usage patterns:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 Concurrency Levels
+
+Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
+
+- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
+- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
+- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
+
+#### 5.1.3 Number of Prompts
+
+For each concurrency level, configure `num_prompts` to simulate realistic user loads:
+
+- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
+- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
+- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
+
+---
+
+#### 5.1.4 Benchmark Commands
+
+**Scenario 1: Chat (1K/1K) - Most Important**
+
+- **Model Deployment**
+```bash Command
+python -m sglang.launch_server \
+  --model zai-org/GLM-4.7 \
+  --tp 8
+```
+
+
+- Low Concurrency (Latency-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Medium Concurrency (Balanced)
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- High Concurrency (Throughput-Optimized)
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+**Scenario 2: Reasoning (1K/8K)**
+
+- Low Concurrency
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Medium Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- High Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+**Scenario 3: Summarization (8K/1K)**
+
+- Low Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Medium Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- High Concurrency
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-4.7 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+
+#### 5.1.5 Understanding the Results
+
+**Key Metrics:**
+
+- **Request Throughput (req/s)**: Number of requests processed per second
+- **Output Token Throughput (tok/s)**: Total tokens generated per second
+- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
+- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
+- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
+
+**Why These Configurations Matter:**
+
+- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
+- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
+- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
+- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
+
+**Interpreting Results:**
+
+- Compare your results against baseline numbers for your hardware
+- Higher throughput at same latency = better performance
+- Lower TTFT = more responsive user experience
+- Lower TPOT = faster generation speed
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+```bash Command
+python -m sglang.test.few_shot_gsm8k \
+  --num-questions 200 \
+  --port 30000
+```

docs_new/cookbook/autoregressive/GLM/GLM-5.1.mdx

@@ -0,0 +1,641 @@
+---
+title: GLM-5.1
+metatags:
+    description: "Deploy GLM-5.1 with SGLang on NVIDIA H100/H200/B200/GB300 and AMD MI300X/MI325X/MI355X."
+tag: NEW
+---
+
+## 1. Model Introduction
+
+**Available Models:**
+
+- **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1)
+- **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8)
+
+**License:** MIT
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B200, GB300, and AMD MI300X/MI325X/MI355X GPUs.
+
+import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment.jsx'
+
+<GLM51Deployment />
+
+### 3.2 Configuration Tips
+
+- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
+- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
+- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
+- BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=16</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=32</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB300</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
+    </tr>
+  </tbody>
+</table>
+
+- **AMD GPUs**: Both BF16 and FP8 checkpoints are supported on MI300X/MI325X/MI355X at tp=8. Use `--nsa-prefill-backend tilelang --nsa-decode-backend tilelang` for the NSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is not currently supported on AMD for GLM-5.1.
+- **GB300**: Only the FP8 checkpoint is recommended on GB300, with `tp=4`. For high-throughput DP attention on GB300, use `--dp 4`.
+- For other configuration tips, please refer to [DeepSeek V3.2 documentation](../../../docs/basic_usage/deepseek_v32). GLM-5.1 and DeepSeek V3.2 share the same model structure, so the optimization techniques between these two models are also common (MTP, DSA kernel, Context Parallel...).
+- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5.1-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
+
+## 4. Model Invocation
+
+Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled):
+
+```shell Command
+SGLANG_ENABLE_SPEC_V2=1 sglang serve \
+  --model-path zai-org/GLM-5.1-FP8 \
+  --tp 8 \
+  --tool-call-parser glm47 \
+  --reasoning-parser glm45 \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4 \
+  --mem-fraction-static 0.85 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command
+
+The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above.
+
+#### FP8 (Recommended)
+
+```shell Command
+sglang serve \
+  --model-path zai-org/GLM-5.1-FP8 \
+  --tp 8 \
+  --trust-remote-code \
+  --tool-call-parser glm47 \
+  --reasoning-parser glm45 \
+  --nsa-prefill-backend tilelang \
+  --nsa-decode-backend tilelang \
+  --chunked-prefill-size 131072 \
+  --mem-fraction-static 0.80 \
+  --watchdog-timeout 1200 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+#### BF16
+
+```shell Command
+sglang serve \
+  --model-path zai-org/GLM-5.1 \
+  --tp 8 \
+  --trust-remote-code \
+  --nsa-prefill-backend tilelang \
+  --nsa-decode-backend tilelang \
+  --chunked-prefill-size 131072 \
+  --mem-fraction-static 0.80 \
+  --watchdog-timeout 1200 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.2 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.3 Advanced Usage
+
+#### 4.3.1 Reasoning Parser
+
+GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
+
+To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
+
+- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
+- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
+
+**Example 1: Thinking Mode (Default)**
+
+Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Thinking mode is enabled by default, no extra parameters needed
+response = client.chat.completions.create(
+    model="zai-org/GLM-5.1-FP8",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+1.  **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step.
+
+2.  **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$.
+
+3.  **Step-by-Step Breakdown:**
+    *   *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$.
+    *   *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$.
+    *   *Step 3: Perform the calculation.*
+        *   $0.15 \times 240$
+        *   I can break this down further to make it easy to follow:
+            *   $0.10 \times 240 = 24$ (which is 10%)
+            *   $0.05 \times 240 = 12$ (which is 5%, half of 10%)
+            *   $24 + 12 = 36$
+        *   Alternatively, standard multiplication:
+            *   $240 \times 15 = 3600$
+            *   Move decimal two places left -> $36$
+    *   *Step 4: State the final answer clearly.*
+
+4.  **Draft the Response (incorporating the steps clearly):**
+    *   *Introduction:* State the problem clearly.
+    *   *Step 1:* Explain how to convert 15% to a decimal.
+    *   *Step 2:* Explain the multiplication step.
+    *   *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value).
+    *   *Conclusion:* Give the final answer.
+
+5.  **Refine the Output (Self-Correction/Polishing during drafting):**
+    *   *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15.
+    *   *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240.
+    *   *Drafting Step 3 (Standard way):* 0.15 × 240 = 36.
+    *   *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%.
+        *   10% of 240 = 24 (move the decimal point one place to the left)
+        *   5% is half of 10%, so half of 24 = 12
+        *   Add them together: 24 + 12 = 36.
+    *   *Final Answer:* 15% of 240 is 36.
+
+6.  **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output).
+=============== Content =================
+Here is the step-by-step solution to find 15% of 240:
+
+**Step 1: Convert the percentage to a decimal.**
+To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left).
+* 15% = 15 ÷ 100 = **0.15**
+
+**Step 2: Multiply the decimal by the number.**
+Now, multiply the decimal (0.15) by the number you are finding the percentage of (240).
+* 0.15 × 240 = **36**
+
+*(Alternative mental math method for Step 2)*:
+If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%:
+* **10% of 240** = 24 (just move the decimal point one place to the left)
+* **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2)
+* **Add them together**: 24 + 12 = **36**
+
+**Answer:**
+15% of 240 is **36**.
+```
+
+**Example 2: Instruct Mode (Thinking Off)**
+
+To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Disable thinking mode via chat_template_kwargs
+response = client.chat.completions.create(
+    model="zai-org/GLM-5.1-FP8",
+    messages=[
+        {"role": "user", "content": "What is 15% of 240?"}
+    ],
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+    max_tokens=2048,
+    stream=True
+)
+
+# In Instruct mode, the model responds directly without reasoning_content
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+15% of 240 is 36.
+
+Here is how to calculate it:
+1. Convert the percentage to a decimal: 15% = 0.15
+2. Multiply the decimal by the number: 0.15 × 240 = 36
+```
+
+#### 4.3.2 Tool Calling
+
+GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/GLM-5.1-FP8",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location.
+=============== Content =================
+Tool Call: get_weather
+   Arguments:
+Tool Call: None
+   Arguments: {
+Tool Call: None
+   Arguments: "location": "Be
+Tool Call: None
+   Arguments: ijing"
+Tool Call: None
+   Arguments: }
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: H200 (8x)
+- Model: GLM-5.1-FP8
+- Tensor Parallelism: 8
+- SGLang Version: commit 947927bdb
+
+#### 5.1.1 Latency Benchmark
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-5.1-FP8 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  35.78
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4213
+Request throughput (req/s):              0.28
+Input token throughput (tok/s):          170.54
+Output token throughput (tok/s):         117.96
+Peak output token throughput (tok/s):    148.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          288.50
+Concurrency:                             1.00
+Accept length:                           3.48
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3576.31
+Median E2E Latency (ms):                 2935.97
+P90 E2E Latency (ms):                    5908.97
+P99 E2E Latency (ms):                    8588.08
+---------------Time to First Token----------------
+Mean TTFT (ms):                          290.88
+Median TTFT (ms):                        282.34
+P99 TTFT (ms):                           332.27
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          7.54
+Median TPOT (ms):                        6.97
+P99 TPOT (ms):                           9.04
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           7.80
+Median ITL (ms):                         6.81
+P95 ITL (ms):                            13.51
+P99 ITL (ms):                            26.99
+Max ITL (ms):                            29.50
+==================================================
+```
+
+#### 5.1.2 Throughput Benchmark
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-5.1-FP8 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 1000 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  411.74
+Total input tokens:                      502493
+Total input text tokens:                 502493
+Total generated tokens:                  500251
+Total generated tokens (retokenized):    499614
+Request throughput (req/s):              2.43
+Input token throughput (tok/s):          1220.41
+Output token throughput (tok/s):         1214.97
+Peak output token throughput (tok/s):    2648.00
+Peak concurrent requests:                105
+Total token throughput (tok/s):          2435.38
+Concurrency:                             96.30
+Accept length:                           3.50
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   39648.76
+Median E2E Latency (ms):                 39058.12
+P90 E2E Latency (ms):                    57009.82
+P99 E2E Latency (ms):                    68880.33
+---------------Time to First Token----------------
+Mean TTFT (ms):                          20613.80
+Median TTFT (ms):                        21429.21
+P99 TTFT (ms):                           29543.17
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          38.73
+Median TPOT (ms):                        36.52
+P99 TPOT (ms):                           67.09
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           38.13
+Median ITL (ms):                         16.57
+P95 ITL (ms):                            86.01
+P99 ITL (ms):                            164.88
+Max ITL (ms):                            1307.02
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --port 30000
+```
+
+- Test Result
+```text Output
+Accuracy: 0.955
+Invalid: 0.000
+Latency: 32.470 s
+Output throughput: 642.044 token/s
+```
+
+#### 5.2.2 MMLU Benchmark
+
+- Benchmark Command
+```bash Command
+python3 benchmark/mmlu/bench_sglang.py --port 30000
+```
+
+- Test Result
+```text Output
+subject: abstract_algebra, #q:100, acc: 0.860
+subject: anatomy, #q:135, acc: 0.874
+subject: astronomy, #q:152, acc: 0.941
+subject: business_ethics, #q:100, acc: 0.880
+subject: clinical_knowledge, #q:265, acc: 0.932
+subject: college_biology, #q:144, acc: 0.972
+subject: college_chemistry, #q:100, acc: 0.640
+subject: college_computer_science, #q:100, acc: 0.900
+subject: college_mathematics, #q:100, acc: 0.810
+subject: college_medicine, #q:173, acc: 0.873
+subject: college_physics, #q:102, acc: 0.912
+subject: computer_security, #q:100, acc: 0.880
+subject: conceptual_physics, #q:235, acc: 0.928
+subject: econometrics, #q:114, acc: 0.807
+subject: electrical_engineering, #q:145, acc: 0.897
+subject: elementary_mathematics, #q:378, acc: 0.937
+subject: formal_logic, #q:126, acc: 0.778
+subject: global_facts, #q:100, acc: 0.710
+subject: high_school_biology, #q:310, acc: 0.961
+subject: high_school_chemistry, #q:203, acc: 0.847
+subject: high_school_computer_science, #q:100, acc: 0.960
+subject: high_school_european_history, #q:165, acc: 0.891
+subject: high_school_geography, #q:198, acc: 0.960
+subject: high_school_government_and_politics, #q:193, acc: 0.984
+subject: high_school_macroeconomics, #q:390, acc: 0.923
+subject: high_school_mathematics, #q:270, acc: 0.696
+subject: high_school_microeconomics, #q:238, acc: 0.962
+subject: high_school_physics, #q:151, acc: 0.821
+subject: high_school_psychology, #q:545, acc: 0.956
+subject: high_school_statistics, #q:216, acc: 0.889
+subject: high_school_us_history, #q:204, acc: 0.941
+subject: high_school_world_history, #q:237, acc: 0.945
+subject: human_aging, #q:223, acc: 0.857
+subject: human_sexuality, #q:131, acc: 0.908
+subject: international_law, #q:121, acc: 0.934
+subject: jurisprudence, #q:108, acc: 0.907
+subject: logical_fallacies, #q:163, acc: 0.933
+subject: machine_learning, #q:112, acc: 0.830
+subject: management, #q:103, acc: 0.942
+subject: marketing, #q:234, acc: 0.940
+subject: medical_genetics, #q:100, acc: 0.990
+subject: miscellaneous, #q:783, acc: 0.959
+subject: moral_disputes, #q:346, acc: 0.873
+subject: moral_scenarios, #q:895, acc: 0.837
+subject: nutrition, #q:306, acc: 0.922
+subject: philosophy, #q:311, acc: 0.897
+subject: prehistory, #q:324, acc: 0.929
+subject: professional_accounting, #q:282, acc: 0.844
+subject: professional_law, #q:1534, acc: 0.714
+subject: professional_medicine, #q:272, acc: 0.941
+subject: professional_psychology, #q:612, acc: 0.913
+subject: public_relations, #q:110, acc: 0.791
+subject: security_studies, #q:245, acc: 0.878
+subject: sociology, #q:201, acc: 0.940
+subject: us_foreign_policy, #q:100, acc: 0.920
+subject: virology, #q:166, acc: 0.596
+subject: world_religions, #q:171, acc: 0.936
+Total latency: 165.275
+Average accuracy: 0.877
+```
+
+### 5.3 AMD GPU Benchmarks
+
+#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
+
+- MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang NSA backends)
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
+```
+
+```text Output
+Accuracy: 0.970
+Invalid: 0.000
+```
+
+Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).

docs_new/cookbook/autoregressive/GLM/GLM-5.mdx

@@ -0,0 +1,667 @@
+---
+title: GLM-5
+metatags:
+    description: "Deploy GLM-5 with SGLang on NVIDIA H100/H200/B200 and AMD MI300X/MI325X/MI355X — state-of-the-art reasoning, enhanced coding, and robust tool calling capabilities."
+tag: NEW
+---
+
+## 1. Model Introduction
+
+[GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity.
+
+With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks.
+
+**Key Features:**
+
+- **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks
+- **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2)
+- **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity
+- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
+- **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency
+
+**Available Models:**
+
+- **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5)
+- **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8)
+
+**License:** MIT
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs.
+
+import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.jsx'
+
+<GLM5Deployment />
+
+### 3.2 Configuration Tips
+
+- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
+- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
+- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
+- BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=16</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=32</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
+    </tr>
+  </tbody>
+</table>
+
+- **B200 (FP8)**: Use `--ep 1 --attention-backend nsa --nsa-decode-backend trtllm --nsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized NSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization.
+
+- **AMD GPUs**: Use `--nsa-prefill-backend tilelang --nsa-decode-backend tilelang` for the NSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.
+- For other configuration tips, please refer to [DeepSeek V3.2 documentation](../../../docs/basic_usage/deepseek_v32). GLM-5 and DeepSeek V3.2 share the same model structure, so the optimization techniques between these two models are also common (MTP, DSA kernel, Context Parallel...).
+- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
+
+<Warning>
+**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off.
+</Warning>
+
+## 4. Model Invocation
+
+Deploy GLM-5 with the following command (FP8 on H200, all features enabled):
+
+```shell Command
+sglang serve \
+  --model zai-org/GLM-5-FP8 \
+  --tp 8 \
+  --tool-call-parser glm47 \
+  --reasoning-parser glm45 \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4 \
+  --mem-fraction-static 0.85 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command
+
+The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above.
+
+```shell Command
+sglang serve \
+  --model zai-org/GLM-5 \
+  --tp 8 \
+  --trust-remote-code \
+  --nsa-prefill-backend tilelang \
+  --nsa-decode-backend tilelang \
+  --chunked-prefill-size 131072 \
+  --mem-fraction-static 0.80 \
+  --watchdog-timeout 1200 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.2 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.3 Advanced Usage
+
+#### 4.3.1 Reasoning Parser
+
+GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
+
+To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
+
+- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
+- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
+
+**Example 1: Thinking Mode (Default)**
+
+Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Thinking mode is enabled by default, no extra parameters needed
+response = client.chat.completions.create(
+    model="zai-org/GLM-5-FP8",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user wants me to solve a math problem: "What is 15% of 240?".
+
+Step 1: Understand the problem. I need to calculate a percentage of a number.
+Formula: Percentage × Number = Result.
+
+Step 2: Convert the percentage to a decimal or fraction.
+15% = 15/100 or 0.15.
+
+Step 3: Perform the multiplication.
+Method A: Decimal multiplication.
+0.15 × 240.
+Break it down:
+10% of 240 = 24.
+5% is half of 10%, so 12.
+15% = 10% + 5% = 24 + 12 = 36.
+
+Method B: Fraction multiplication.
+15/100 × 240.
+Simplify 240/100 = 2.4.
+15 × 2.4.
+10 × 2.4 = 24.
+5 × 2.4 = 12.
+24 + 12 = 36.
+
+Method C: Direct multiplication.
+240 × 0.15.
+240 × 0.10 = 24.
+240 × 0.05 = 12.
+24 + 12 = 36.
+
+Step 4: Final Verification.
+Is 36 reasonable?
+10% is 24. 20% is 48.
+15% is halfway between 10% and 20%.
+Halfway between 24 and 48 is 36.
+The result is correct.
+
+Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%).
+=============== Content =================
+Here is the step-by-step solution:
+
+**Step 1: Convert the percentage to a decimal.**
+To convert 15% to a decimal, divide by 100.
+$$15\% = \frac{15}{100} = 0.15$$
+
+**Step 2: Multiply the decimal by the number.**
+Now, multiply 0.15 by 240.
+$$0.15 \times 240$$
+
+**Step 3: Perform the calculation.**
+You can break this down to make it easier:
+$$0.15 = 0.10 + 0.05$$
+
+*   First, find 10% of 240:
+    $$0.10 \times 240 = 24$$
+*   Next, find 5% (which is half of 10%):
+    $$\frac{24}{2} = 12$$
+*   Add the two results together:
+    $$24 + 12 = 36$$
+
+**Answer:**
+15% of 240 is **36**.
+```
+
+**Example 2: Instruct Mode (Thinking Off)**
+
+To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Disable thinking mode via chat_template_kwargs
+response = client.chat.completions.create(
+    model="zai-org/GLM-5-FP8",
+    messages=[
+        {"role": "user", "content": "What is 15% of 240?"}
+    ],
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+    max_tokens=2048,
+    stream=True
+)
+
+# In Instruct mode, the model responds directly without reasoning_content
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+To find **15% of 240**, follow these steps:
+
+### Step 1: Convert the Percentage to a Decimal
+First, convert the percentage to a decimal by dividing by 100.
+
+\[
+15\% = \frac{15}{100} = 0.15
+\]
+
+### Step 2: Multiply by the Number
+Next, multiply the decimal by the number you want to find the percentage of.
+
+\[
+0.15 \times 240
+\]
+
+### Step 3: Perform the Multiplication
+Calculate the multiplication:
+
+\[
+0.15 \times 240 = 36
+\]
+
+### Final Answer
+\[
+\boxed{36}
+\]
+```
+
+#### 4.3.2 Tool Calling
+
+GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/GLM-5-FP8",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required:
+
+- location: required, should be "Beijing"
+- unit: optional (not in required array), can be "celsius" or "fahrenheit"
+
+Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you.
+=============== Content =================
+Tool Call: get_weather
+   Arguments:
+Tool Call: None
+   Arguments: {
+Tool Call: None
+   Arguments: "location": "Be
+Tool Call: None
+   Arguments: ijing"
+Tool Call: None
+   Arguments: }
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: H200 (8x)
+- Model: GLM-5-FP8
+- Tensor Parallelism: 8
+- SGLang Version: commit 947927bdb
+
+#### 5.1.1 Latency Benchmark
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-5-FP8 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  35.78
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4213
+Request throughput (req/s):              0.28
+Input token throughput (tok/s):          170.54
+Output token throughput (tok/s):         117.96
+Peak output token throughput (tok/s):    148.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          288.50
+Concurrency:                             1.00
+Accept length:                           3.48
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3576.31
+Median E2E Latency (ms):                 2935.97
+P90 E2E Latency (ms):                    5908.97
+P99 E2E Latency (ms):                    8588.08
+---------------Time to First Token----------------
+Mean TTFT (ms):                          290.88
+Median TTFT (ms):                        282.34
+P99 TTFT (ms):                           332.27
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          7.54
+Median TPOT (ms):                        6.97
+P99 TPOT (ms):                           9.04
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           7.80
+Median ITL (ms):                         6.81
+P95 ITL (ms):                            13.51
+P99 ITL (ms):                            26.99
+Max ITL (ms):                            29.50
+==================================================
+```
+
+#### 5.1.2 Throughput Benchmark
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/GLM-5-FP8 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 1000 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  411.74
+Total input tokens:                      502493
+Total input text tokens:                 502493
+Total generated tokens:                  500251
+Total generated tokens (retokenized):    499614
+Request throughput (req/s):              2.43
+Input token throughput (tok/s):          1220.41
+Output token throughput (tok/s):         1214.97
+Peak output token throughput (tok/s):    2648.00
+Peak concurrent requests:                105
+Total token throughput (tok/s):          2435.38
+Concurrency:                             96.30
+Accept length:                           3.50
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   39648.76
+Median E2E Latency (ms):                 39058.12
+P90 E2E Latency (ms):                    57009.82
+P99 E2E Latency (ms):                    68880.33
+---------------Time to First Token----------------
+Mean TTFT (ms):                          20613.80
+Median TTFT (ms):                        21429.21
+P99 TTFT (ms):                           29543.17
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          38.73
+Median TPOT (ms):                        36.52
+P99 TPOT (ms):                           67.09
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           38.13
+Median ITL (ms):                         16.57
+P95 ITL (ms):                            86.01
+P99 ITL (ms):                            164.88
+Max ITL (ms):                            1307.02
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --port 30000
+```
+
+- Test Result
+```text Output
+Accuracy: 0.955
+Invalid: 0.000
+Latency: 32.470 s
+Output throughput: 642.044 token/s
+```
+
+#### 5.2.2 MMLU Benchmark
+
+- Benchmark Command
+```bash Command
+python3 benchmark/mmlu/bench_sglang.py --port 30000
+```
+
+- Test Result
+```text Output
+subject: abstract_algebra, #q:100, acc: 0.860
+subject: anatomy, #q:135, acc: 0.874
+subject: astronomy, #q:152, acc: 0.941
+subject: business_ethics, #q:100, acc: 0.880
+subject: clinical_knowledge, #q:265, acc: 0.932
+subject: college_biology, #q:144, acc: 0.972
+subject: college_chemistry, #q:100, acc: 0.640
+subject: college_computer_science, #q:100, acc: 0.900
+subject: college_mathematics, #q:100, acc: 0.810
+subject: college_medicine, #q:173, acc: 0.873
+subject: college_physics, #q:102, acc: 0.912
+subject: computer_security, #q:100, acc: 0.880
+subject: conceptual_physics, #q:235, acc: 0.928
+subject: econometrics, #q:114, acc: 0.807
+subject: electrical_engineering, #q:145, acc: 0.897
+subject: elementary_mathematics, #q:378, acc: 0.937
+subject: formal_logic, #q:126, acc: 0.778
+subject: global_facts, #q:100, acc: 0.710
+subject: high_school_biology, #q:310, acc: 0.961
+subject: high_school_chemistry, #q:203, acc: 0.847
+subject: high_school_computer_science, #q:100, acc: 0.960
+subject: high_school_european_history, #q:165, acc: 0.891
+subject: high_school_geography, #q:198, acc: 0.960
+subject: high_school_government_and_politics, #q:193, acc: 0.984
+subject: high_school_macroeconomics, #q:390, acc: 0.923
+subject: high_school_mathematics, #q:270, acc: 0.696
+subject: high_school_microeconomics, #q:238, acc: 0.962
+subject: high_school_physics, #q:151, acc: 0.821
+subject: high_school_psychology, #q:545, acc: 0.956
+subject: high_school_statistics, #q:216, acc: 0.889
+subject: high_school_us_history, #q:204, acc: 0.941
+subject: high_school_world_history, #q:237, acc: 0.945
+subject: human_aging, #q:223, acc: 0.857
+subject: human_sexuality, #q:131, acc: 0.908
+subject: international_law, #q:121, acc: 0.934
+subject: jurisprudence, #q:108, acc: 0.907
+subject: logical_fallacies, #q:163, acc: 0.933
+subject: machine_learning, #q:112, acc: 0.830
+subject: management, #q:103, acc: 0.942
+subject: marketing, #q:234, acc: 0.940
+subject: medical_genetics, #q:100, acc: 0.990
+subject: miscellaneous, #q:783, acc: 0.959
+subject: moral_disputes, #q:346, acc: 0.873
+subject: moral_scenarios, #q:895, acc: 0.837
+subject: nutrition, #q:306, acc: 0.922
+subject: philosophy, #q:311, acc: 0.897
+subject: prehistory, #q:324, acc: 0.929
+subject: professional_accounting, #q:282, acc: 0.844
+subject: professional_law, #q:1534, acc: 0.714
+subject: professional_medicine, #q:272, acc: 0.941
+subject: professional_psychology, #q:612, acc: 0.913
+subject: public_relations, #q:110, acc: 0.791
+subject: security_studies, #q:245, acc: 0.878
+subject: sociology, #q:201, acc: 0.940
+subject: us_foreign_policy, #q:100, acc: 0.920
+subject: virology, #q:166, acc: 0.596
+subject: world_religions, #q:171, acc: 0.936
+Total latency: 165.275
+Average accuracy: 0.877
+```
+
+### 5.3 AMD GPU Benchmarks
+
+#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
+
+- MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang NSA backends)
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
+```
+
+```text Output
+Accuracy: 0.970
+Invalid: 0.000
+```
+
+Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).

docs_new/cookbook/autoregressive/GLM/GLM-Glyph.mdx

@@ -0,0 +1,829 @@
+---
+title: GLM Glyph
+metatags:
+    description: "Deploy GLM-Glyph with SGLang - community contribution guide for Zhipu AI's GLM Glyph model deployment."
+---
+
+import { GLMGlyphDeployment } from '/src/snippets/autoregressive/glm-glyph-deployment.jsx';
+
+## 1. Model Introduction
+
+[Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
+
+**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
+
+**Key Features:**
+
+- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
+- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
+- **High Performance**: Optimized for both throughput and latency scenarios
+
+**Available Models:**
+
+- **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph)
+- **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8)
+
+**License:**
+
+Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options.
+
+<GLMGlyphDeployment />
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [GLM-4.5/GLM-4.6 Usage](../../../docs/basic_usage/glm45).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Thinking Mode
+
+Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path zai-org/Glyph \
+  --reasoning-parser glm45 \
+  --tp 4
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="zai-org/Glyph",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+**Disable Thinking Mode:**
+
+To disable thinking mode for a specific request:
+
+```python Example
+response = client.chat.completions.create(
+    model="zai-org/Glyph",
+    messages=[{"role": "user", "content": "What is the capital of France?"}],
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+```
+
+#### 4.2.2 Tool Calling
+
+Glyph supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path zai-org/Glyph \
+  --reasoning-parser glm45 \
+  --tool-call-parser glm45 \
+  --tp 4
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="zai-org/Glyph",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="zai-org/Glyph",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Model: Glyph
+- SGLang Version: 0.5.6.post1
+
+**Benchmark Methodology:**
+
+We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
+
+#### 5.1.1 Standard Scenario Benchmark
+
+- **Model Deployment**
+```bash Command
+python -m sglang.launch_server \
+  --model zai-org/Glyph \
+  --tp 2
+```
+
+##### 5.1.1.1 Low Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  17.03
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4220
+Request throughput (req/s):              0.59
+Input token throughput (tok/s):          358.17
+Output token throughput (tok/s):         247.74
+Peak output token throughput (tok/s):    251.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          605.91
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   1702.14
+Median E2E Latency (ms):                 1361.72
+---------------Time to First Token----------------
+Mean TTFT (ms):                          22.35
+Median TTFT (ms):                        22.61
+P99 TTFT (ms):                           23.76
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          3.99
+Median TPOT (ms):                        3.99
+P99 TPOT (ms):                           4.01
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           3.99
+Median ITL (ms):                         3.99
+P95 ITL (ms):                            4.03
+P99 ITL (ms):                            4.12
+Max ITL (ms):                            7.46
+==================================================
+```
+
+##### 5.1.1.2 Medium Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  16.27
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40804
+Request throughput (req/s):              4.92
+Input token throughput (tok/s):          2438.06
+Output token throughput (tok/s):         2507.94
+Peak output token throughput (tok/s):    3069.00
+Peak concurrent requests:                26
+Total token throughput (tok/s):          4946.00
+Concurrency:                             13.44
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   2733.43
+Median E2E Latency (ms):                 2892.98
+---------------Time to First Token----------------
+Mean TTFT (ms):                          33.10
+Median TTFT (ms):                        27.73
+P99 TTFT (ms):                           49.34
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          5.33
+Median TPOT (ms):                        5.39
+P99 TPOT (ms):                           5.86
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           5.30
+Median ITL (ms):                         4.89
+P95 ITL (ms):                            5.54
+P99 ITL (ms):                            21.17
+Max ITL (ms):                            25.14
+==================================================
+```
+
+##### 5.1.1.3 High Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  25.67
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    252657
+Request throughput (req/s):              19.48
+Input token throughput (tok/s):          9733.69
+Output token throughput (tok/s):         9843.99
+Peak output token throughput (tok/s):    13398.00
+Peak concurrent requests:                127
+Total token throughput (tok/s):          19577.68
+Concurrency:                             89.49
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   4593.75
+Median E2E Latency (ms):                 4431.03
+---------------Time to First Token----------------
+Mean TTFT (ms):                          48.66
+Median TTFT (ms):                        35.88
+P99 TTFT (ms):                           120.61
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          9.10
+Median TPOT (ms):                        9.55
+P99 TPOT (ms):                           11.00
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           9.01
+Median ITL (ms):                         6.51
+P95 ITL (ms):                            23.19
+P99 ITL (ms):                            25.54
+Max ITL (ms):                            52.93
+==================================================
+```
+
+#### 5.1.2 Reasoning Scenario Benchmark
+
+##### 5.1.2.1 Low Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  201.53
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  44462
+Total generated tokens (retokenized):    44455
+Request throughput (req/s):              0.05
+Input token throughput (tok/s):          30.27
+Output token throughput (tok/s):         220.63
+Peak output token throughput (tok/s):    251.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          250.90
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   20151.45
+Median E2E Latency (ms):                 21576.31
+---------------Time to First Token----------------
+Mean TTFT (ms):                          2362.23
+Median TTFT (ms):                        23.03
+P99 TTFT (ms):                           21310.14
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          4.00
+Median TPOT (ms):                        4.00
+P99 TPOT (ms):                           4.01
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           4.00
+Median ITL (ms):                         4.00
+P95 ITL (ms):                            4.05
+P99 ITL (ms):                            4.08
+Max ITL (ms):                            5.67
+==================================================
+```
+
+##### 5.1.2.2 Medium Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  118.67
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  318306
+Total generated tokens (retokenized):    318270
+Request throughput (req/s):              0.67
+Input token throughput (tok/s):          334.27
+Output token throughput (tok/s):         2682.26
+Peak output token throughput (tok/s):    3264.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          3016.53
+Concurrency:                             13.74
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   20387.23
+Median E2E Latency (ms):                 20466.09
+---------------Time to First Token----------------
+Mean TTFT (ms):                          132.47
+Median TTFT (ms):                        27.19
+P99 TTFT (ms):                           583.15
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          5.09
+Median TPOT (ms):                        5.13
+P99 TPOT (ms):                           5.19
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           5.09
+Median ITL (ms):                         5.08
+P95 ITL (ms):                            5.18
+P99 ITL (ms):                            5.57
+Max ITL (ms):                            522.26
+==================================================
+```
+
+##### 5.1.2.3 High Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  150.00
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total input vision tokens:               0
+Total generated tokens:                  1301025
+Total generated tokens (retokenized):    1300901
+Request throughput (req/s):              2.13
+Input token throughput (tok/s):          1059.59
+Output token throughput (tok/s):         8673.49
+Peak output token throughput (tok/s):    11899.00
+Peak concurrent requests:                71
+Total token throughput (tok/s):          9733.09
+Concurrency:                             54.71
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   25645.42
+Median E2E Latency (ms):                 26913.26
+---------------Time to First Token----------------
+Mean TTFT (ms):                          163.75
+Median TTFT (ms):                        93.67
+P99 TTFT (ms):                           426.19
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          6.27
+Median TPOT (ms):                        6.39
+P99 TPOT (ms):                           6.59
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.27
+Median ITL (ms):                         0.17
+P95 ITL (ms):                            32.94
+P99 ITL (ms):                            67.89
+Max ITL (ms):                            136.00
+==================================================
+```
+
+#### 5.1.3 Summarization Scenario Benchmark
+
+#### 5.1.3.1 Low Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  17.44
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4220
+Request throughput (req/s):              0.57
+Input token throughput (tok/s):          2405.19
+Output token throughput (tok/s):         242.00
+Peak output token throughput (tok/s):    250.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          2647.19
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   1742.54
+Median E2E Latency (ms):                 1412.47
+---------------Time to First Token----------------
+Mean TTFT (ms):                          53.48
+Median TTFT (ms):                        45.05
+P99 TTFT (ms):                           98.57
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          4.01
+Median TPOT (ms):                        4.01
+P99 TPOT (ms):                           4.03
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           4.01
+Median ITL (ms):                         4.01
+P95 ITL (ms):                            4.06
+P99 ITL (ms):                            4.09
+Max ITL (ms):                            4.95
+==================================================
+```
+
+##### 5.1.3.2 Medium Concurrency
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  16.90
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total input vision tokens:               0
+Total generated tokens:                  41669
+Total generated tokens (retokenized):    41668
+Request throughput (req/s):              4.73
+Input token throughput (tok/s):          17753.58
+Output token throughput (tok/s):         2465.75
+Peak output token throughput (tok/s):    3005.00
+Peak concurrent requests:                25
+Total token throughput (tok/s):          20219.33
+Concurrency:                             13.68
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   2890.33
+Median E2E Latency (ms):                 3069.55
+---------------Time to First Token----------------
+Mean TTFT (ms):                          41.46
+Median TTFT (ms):                        31.75
+P99 TTFT (ms):                           93.18
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          5.52
+Median TPOT (ms):                        5.58
+P99 TPOT (ms):                           6.14
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           5.48
+Median ITL (ms):                         5.13
+P95 ITL (ms):                            5.93
+P99 ITL (ms):                            20.76
+Max ITL (ms):                            36.01
+==================================================
+```
+
+##### 5.1.3.3 High Concurrency
+
+- **Benchmark Command**:
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model zai-org/Glyph \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64 \
+  --request-rate inf
+```
+- **Test Results**:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  35.54
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total input vision tokens:               0
+Total generated tokens:                  170000
+Total generated tokens (retokenized):    169994
+Request throughput (req/s):              9.01
+Input token throughput (tok/s):          35848.57
+Output token throughput (tok/s):         4783.96
+Peak output token throughput (tok/s):    8396.00
+Peak concurrent requests:                80
+Total token throughput (tok/s):          40632.53
+Concurrency:                             59.26
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6580.96
+Median E2E Latency (ms):                 6248.74
+---------------Time to First Token----------------
+Mean TTFT (ms):                          345.27
+Median TTFT (ms):                        96.06
+P99 TTFT (ms):                           2823.92
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          12.26
+Median TPOT (ms):                        12.53
+P99 TPOT (ms):                           23.58
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.76
+Median ITL (ms):                         6.57
+P95 ITL (ms):                            27.66
+P99 ITL (ms):                            91.24
+Max ITL (ms):                            2609.64
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+
+```bash Command
+python -m sglang.test.few_shot_gsm8k \
+  --num-questions 200
+```
+
+- Test Result
+
+```text Output
+Accuracy: 0.890
+Invalid: 0.000
+Latency: 3.718 s
+Output throughput: 5245.606 token/s
+```

docs_new/cookbook/autoregressive/GLM/GLM-OCR.mdx

@@ -0,0 +1,227 @@
+---
+title: GLM-OCR
+metatags:
+    description: "Deploy GLM-OCR with SGLang - state-of-the-art OCR performance for complex document understanding."
+---
+
+## 1. Model Introduction
+
+[GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization.
+
+The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.
+
+**Hardware Support:** NVIDIA B200/H100/H200
+
+**Key Features:**
+
+- **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction.
+- **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts.
+- **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments.
+- **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems.
+
+For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding.
+
+import { GLMOCRDeployment } from '/src/snippets/autoregressive/glm-ocr-deployment.jsx'
+
+<GLMOCRDeployment />
+
+### 3.2 Configuration Tips
+
+- **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT.
+- **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency.
+- **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 OCR Image Processing
+
+GLM-OCR supports OCR tasks on various document types. Here's a basic example:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Please extract all text from this image."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="zai-org/GLM-OCR",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 2.29s
+Generated text: CINNAMON SUGAR
+1 x 17,000 17,000
+
+SUB TOTAL 17,000
+
+GRAND TOTAL 17,000
+
+CASH IDR 20,000
+
+CHANGE DUE 3,000
+
+```
+
+#### 4.2.2 Complex Document Processing
+
+GLM-OCR excels at processing complex documents including:
+
+- **Tables**: Accurate extraction of tabular data with structure preservation
+- **Formulas**: Mathematical formula recognition
+- **Code Documents**: Source code extraction from screenshots
+- **Seals and Stamps**: Recognition of seals and stamps in documents
+- **Multi-layout Documents**: Mixed content with text, images, and tables
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+# Example: Processing a document with tables
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "YOUR_DOCUMENT_IMAGE_URL"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Please extract the table content from this document and format it as markdown."
+            }
+        ]
+    }
+]
+
+response = client.chat.completions.create(
+    model="zai-org/GLM-OCR",
+    messages=messages,
+    max_tokens=4096
+)
+print(response.choices[0].message.content)
+```
+
+## 5. Benchmark
+
+### 5.1 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.1.1 OCRBench Benchmark
+
+- Benchmark Command
+
+```bash Command
+python3 -m lmms_eval \
+  --model openai_compatible \
+  --model_args "model_version=zai-org/GLM-OCR" \
+  --tasks ocrbench \
+  --batch_size 128 \
+  --log_samples \
+  --log_samples_suffix "openai_compatible" \
+  --output_path ./logs
+```
+
+- Test Result
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "12.5%"}} />
+    <col style={{width: "12.5%"}} />
+    <col style={{width: "12.5%"}} />
+    <col style={{width: "12.5%"}} />
+    <col style={{width: "12.5%"}} />
+    <col style={{width: "12.5%"}} />
+    <col style={{width: "12.5%"}} />
+    <col style={{width: "12.5%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Stderr</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yaml</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench_accuracy</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.806</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 OmniDocBench V1.5
+
+GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks.

docs_new/cookbook/autoregressive/InclusionAI/LLaDA-2.1.mdx

@@ -0,0 +1,701 @@
+---
+title: LLaDA 2.1
+metatags:
+    description: "Deploy LLaDA 2.1 with SGLang - large-scale discrete diffusion language model with parallel token generation, iterative denoising, MoE architecture, and reinforcement learning for reasoning."
+tag: NEW
+---
+
+import { LLaDA21Deployment } from '/src/snippets/autoregressive/llada-21-deployment.jsx';
+
+## 1. Model Introduction
+
+[LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation.
+
+**Key Features:**
+
+- **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight
+- **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores
+- **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling
+- **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following
+- **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model
+
+**Available Models:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "20.0%"}} />
+    <col style={{width: "20.0%"}} />
+    <col style={{width: "20.0%"}} />
+    <col style={{width: "20.0%"}} />
+    <col style={{width: "20.0%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameters</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Architecture</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFace</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-mini**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE (20 layers, 16 attention heads)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-flash**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100B</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)</td>
+    </tr>
+  </tbody>
+</table>
+
+**License:**
+
+Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details.
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs.
+
+<LLaDA21Deployment />
+
+### 3.2 Configuration Tips
+
+**dLLM-Specific Parameters:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diffusion decoding algorithm</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`JointThreshold`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for LLaDA model loading</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.8`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--max-running-requests`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum concurrent requests</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1` (for best quality)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention computation backend</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
+    </tr>
+  </tbody>
+</table>
+
+**Decoding Mode Comparison:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Mode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Threshold</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Speed</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Quality</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Best For</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quality Mode (Q)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Conservative</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Moderate</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Higher benchmark scores</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Accuracy-critical tasks</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Speed Mode (S)**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Aggressive</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Very fast, relies on T2T editing</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Slightly lower</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Throughput-critical tasks</td>
+    </tr>
+  </tbody>
+</table>
+
+**Hardware Requirements:**
+
+- **LLaDA2.1-mini (16B)**: ~47 GB VRAM, runs on a single GPU (TP=1)
+- **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200)
+
+## 4. Model Invocation
+
+### 4.1 Deployment
+
+Start the server using the command generated above, for example:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path inclusionAI/LLaDA2.1-mini \
+  --dllm-algorithm JointThreshold \
+  --tp 1 \
+  --trust-remote-code \
+  --mem-fraction-static 0.8 \
+  --max-running-requests 1 \
+  --attention-backend flashinfer \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+### 4.2 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+**Simple Completion Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="inclusionAI/LLaDA2.1-mini",
+    messages=[
+        {"role": "user", "content": "Explain what a diffusion language model is in simple terms."}
+    ],
+    max_tokens=1024
+)
+
+print(response.choices[0].message.content)
+```
+
+**Output Example:**
+
+```text Output
+Sure! Let's break it down in simple terms.
+
+A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text.
+
+Here’s how it works, using a simple real-life analogy:
+
+Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together.
+
+Now, think of the process like this:
+
+1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper).
+2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book.
+3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start.
+4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word.
+
+So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story.
+
+In short:
+A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned.
+```
+
+### 4.3 Advanced Usage
+
+#### 4.3.1 Streaming
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="inclusionAI/LLaDA2.1-mini",
+    messages=[
+        {"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+````text Output
+Here are several ways to implement the Fibonacci sequence in Python:
+
+## 1. Recursive Approach (Simple but Inefficient)
+
+```python
+def fibonacci_recursive(n):
+    """
+    Compute the nth Fibonacci number using recursion.
+
+    Args:
+        n (int): The position in the Fibonacci sequence (0-indexed)
+
+    Returns:
+        int: The nth Fibonacci number
+
+    Raises:
+        ValueError: If n is negative
+    """
+    if n < 0:
+        raise ValueError("n must be non-negative")
+
+    if n <= 1:
+        return n
+
+    return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)
+
+# Example usage
+print(fibonacci_recursive(10))  # Output: 55
+```
+
+## 2. Iterative Approach (Efficient)
+...
+````
+
+#### 4.3.2 Code Generation
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="inclusionAI/LLaDA2.1-mini",
+    messages=[
+        {"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."}
+    ],
+    max_tokens=2048
+)
+
+print(response.choices[0].message.content)
+```
+
+**Output Example:**
+
+````text Output
+```python
+def is_palindrome(s):
+    """
+    Check if a string is a palindrome.
+
+    A palindrome is a word, phrase, or sequence that reads the same backward as forward.
+    This function ignores case, spaces, punctuation, and non characters characters.
+
+    Args:
+        s (str): The string to check
+
+    Returns:
+        bool: True if the string is a palindrome, False otherwise
+
+    Examples:
+        >>> is_palindrome("racecar")
+        True
+        >>> is_palindrome("A man a plan a canal Panama")
+        True
+        >>> is_palindrome("race a car")
+        False
+        >>> is_palindrome("")
+        True
+        >>> is_palindrome("a")
+        True
+    """
+    # Remove non-alphanumeric characters and convert to lowercase
+    cleaned = ''.join(char.lower() for char in s if char.isalnum())
+
+    # Check if the cleaned string reads the same forwards and backwards
+    return cleaned == cleaned[::-1]
+
+# Test cases
+def test_is_palindrome():
+    """Test the is_palindrome function with various inputs."""
+
+    # Test basic palindromes
+    assert is_palindrome("racecar") == True
+    assert is_palindrome("level") == True
+    assert is_palindrome("madam") == True
+    assert is_palindrome("radar") == True
+
+    # Test palindromes with spaces and punctuation
+    assert is_palindrome("A man a plan a canal Panama") == True
+    assert is_palindrome("race a car") == False
+    assert is_palindrome("Was it a car or a cat I saw?") == True
+    assert is_palindrome("Madam, I'm Adam") == True
+
+    # Test edge cases
+    assert is_palindrome("") == True
+    assert is_palindrome("a") == True
+    assert is_palindrome("A") == True
+    assert is_palindrome("Aa") == True
+
+    # Test non-palindromes
+    assert is_palindrome("hello") == False
+    assert is_palindrome("world") == False
+    assert is_palindrome("python") == False
+
+    # Test single characters
+    assert is_palindrome("1") == True
+    assert is_palindrome("1") == True
+
+    print("All tests passed!")
+
+# Run the tests
+if __name__ == "__main__":
+    # Example usage
+    print("Testing isalindrome function:")
+    print(f"'racecar' {is_palindrome('racecar')}")
+    print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}")
+    print(f"'race a car': {is_palindrome('race a car')}")
+    print(f"'hello': {is_palindrome('hello')}")
+
+    # Run tests
+    test_is_palindrome()
+```
+
+This implementation includes:
+
+1. **Comprehensive function** `is_palindrome()` that:
+   - Ignores case by converting to lowercase
+   - Removes all non-alphanumeric characters (spaces, punctuation, etc.)
+   - Uses string slicing (`[::-1]`) to reverse the string
+
+2. **Detailed docstring** explaining:
+   - What the function does
+   - How it works
+   - Return value
+   - Examples of usage
+
+3. **Extensive test cases** covering:
+   - Basic palindromes
+   - Palindromes with spaces and punctuation
+   - Edge cases (empty string, single character)
+   - Non-palindromes
+   - Mixed case scenarios
+
+4. **Test function** that uses assertions to verify the function works correctly
+
+The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama".
+````
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 (4x)
+- SGLang Version: 0.5.8+
+
+#### 5.1.1 LLaDA2.1-mini
+
+**Model Deployment:**
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path inclusionAI/LLaDA2.1-mini \
+  --dllm-algorithm JointThreshold \
+  --tp 1 \
+  --trust-remote-code \
+  --mem-fraction-static 0.8 \
+  --max-running-requests 1 \
+  --attention-backend flashinfer
+```
+
+- Latency Benchmark
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model inclusionAI/LLaDA2.1-mini \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- **Latency Result**:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  9.90
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    3433
+Request throughput (req/s):              1.01
+Input token throughput (tok/s):          616.26
+Output token throughput (tok/s):         426.26
+Peak output token throughput (tok/s):    1010.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          1042.53
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   988.87
+Median E2E Latency (ms):                 655.27
+P90 E2E Latency (ms):                    1952.50
+P99 E2E Latency (ms):                    2932.19
+---------------Time to First Token----------------
+Mean TTFT (ms):                          152.74
+Median TTFT (ms):                        150.37
+P99 TTFT (ms):                           229.78
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          2.16
+Median TPOT (ms):                        2.08
+P99 TPOT (ms):                           3.72
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           2.10
+Median ITL (ms):                         1.99
+P95 ITL (ms):                            4.03
+P99 ITL (ms):                            6.34
+Max ITL (ms):                            26.59
+==================================================
+```
+
+- Throughput Benchmark
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model inclusionAI/LLaDA2.1-mini \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+- **Throughput Result**:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  467.74
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    189717
+Request throughput (req/s):              1.07
+Input token throughput (tok/s):          534.12
+Output token throughput (tok/s):         540.17
+Peak output token throughput (tok/s):    1753.00
+Peak concurrent requests:                105
+Total token throughput (tok/s):          1074.30
+Concurrency:                             90.77
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   84912.27
+Median E2E Latency (ms):                 86564.26
+P90 E2E Latency (ms):                    110567.26
+P99 E2E Latency (ms):                    114303.38
+---------------Time to First Token----------------
+Mean TTFT (ms):                          83920.39
+Median TTFT (ms):                        85669.54
+P99 TTFT (ms):                           112969.91
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          2.67
+Median TPOT (ms):                        1.65
+P99 TPOT (ms):                           4.43
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           1.69
+Median ITL (ms):                         1.46
+P95 ITL (ms):                            3.96
+P99 ITL (ms):                            4.84
+Max ITL (ms):                            92.08
+==================================================
+```
+
+#### 5.1.2 LLaDA2.1-flash
+
+**Model Deployment:**
+
+```bash Command
+python -m sglang.launch_server \
+  --model-path inclusionAI/LLaDA2.1-flash \
+  --dllm-algorithm JointThreshold \
+  --tp 4 \
+  --trust-remote-code \
+  --mem-fraction-static 0.8 \
+  --max-running-requests 1 \
+  --attention-backend flashinfer
+```
+
+- Latency Benchmark
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model inclusionAI/LLaDA2.1-flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- **Latency Result**:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  14.46
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    3276
+Request throughput (req/s):              0.69
+Input token throughput (tok/s):          421.79
+Output token throughput (tok/s):         291.75
+Peak output token throughput (tok/s):    676.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          713.53
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   1445.16
+Median E2E Latency (ms):                 968.06
+P90 E2E Latency (ms):                    3101.86
+P99 E2E Latency (ms):                    4208.49
+---------------Time to First Token----------------
+Mean TTFT (ms):                          231.63
+Median TTFT (ms):                        242.67
+P99 TTFT (ms):                           341.33
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          3.04
+Median TPOT (ms):                        2.79
+P99 TPOT (ms):                           5.33
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           3.05
+Median ITL (ms):                         2.41
+P95 ITL (ms):                            7.25
+P99 ITL (ms):                            8.27
+Max ITL (ms):                            29.27
+==================================================
+```
+
+- Throughput Benchmark
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model inclusionAI/LLaDA2.1-flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+- **Throughput Result**:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  671.85
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    177961
+Request throughput (req/s):              0.74
+Input token throughput (tok/s):          371.85
+Output token throughput (tok/s):         376.07
+Peak output token throughput (tok/s):    1521.00
+Peak concurrent requests:                103
+Total token throughput (tok/s):          747.92
+Concurrency:                             91.28
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   122658.36
+Median E2E Latency (ms):                 125265.55
+P90 E2E Latency (ms):                    159554.07
+P99 E2E Latency (ms):                    165174.88
+---------------Time to First Token----------------
+Mean TTFT (ms):                          121009.17
+Median TTFT (ms):                        124437.80
+P99 TTFT (ms):                           163579.29
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          4.73
+Median TPOT (ms):                        2.16
+P99 TPOT (ms):                           7.13
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           2.38
+Median ITL (ms):                         1.40
+P95 ITL (ms):                            6.89
+P99 ITL (ms):                            8.60
+Max ITL (ms):                            176.78
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+```bash Command
+python -m sglang.test.few_shot_gsm8k \
+  --num-questions 200 \
+  --port 8000
+```
+
+**Results:**
+
+```text Output
+Accuracy: 0.895
+Invalid: 0.000
+Latency: 100.552 s
+Output throughput: 262.094 token/s
+```

docs_new/cookbook/autoregressive/InclusionAI/Ling-2.5-1T.mdx

@@ -0,0 +1,221 @@
+---
+title: Ling-2.5-1T
+metatags:
+    description: "Deploy Ling-2.5-1T with SGLang - 1T parameter MoE model with 63B active parameters, trillion-scale context length up to 1M tokens, and agentic tool calling capabilities."
+tag: NEW
+---
+
+## 1. Model Introduction
+
+[Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality.
+
+**Key Features:**
+
+- **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens.
+- **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume ~4x the output tokens.
+- **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following.
+- **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4.
+- **Context Length**: 256K -> 1M (YaRN)
+
+**Available Models:**
+
+- **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T)
+
+**License:** MIT
+
+## 2. SGLang Installation
+
+Ling-2.5-1T requires a specific SGLang Docker image:
+
+```bash Command
+# For H200/B200
+docker pull lmsysorg/sglang:nightly-dev-20260213-a0ebaa64
+
+# For GB200/GB300
+docker pull lmsysorg/sglang:nightly-dev-cu13-20260213-a0ebaa64
+```
+
+For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
+
+Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/installation) guide for setup instructions.
+
+## 3. Model Deployment
+
+Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform.
+
+import { Ling251TDeployment } from '/src/snippets/autoregressive/ling-25-1t-deployment.jsx'
+
+<Ling251TDeployment />
+
+### Configuration Tips
+
+- The `--trust-remote-code` flag is required for this model due to custom modeling code.
+- `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes.
+- Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading.
+- On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For example, launch the server on 2 H200 nodes:
+
+```bash Command
+export MASTER_IP=10.10.0.1 # The IP of Node 0
+export PORT=30000
+export DIST_PORT=50000
+
+# Node 0:
+python3 -m sglang.launch_server \
+--model-path inclusionAI/Ling-2.5-1T \
+--trust-remote-code \
+--tp-size 8 \
+--pp-size 2 \
+--nnodes 2 \
+--node-rank 0 \
+--host 0.0.0.0 \
+--port ${PORT} \
+--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
+--tool-call-parser qwen \
+--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
+--mem-frac 0.95
+
+
+# Node 1:
+python3 -m sglang.launch_server \
+--model-path inclusionAI/Ling-2.5-1T \
+--trust-remote-code \
+--tp-size 8 \
+--pp-size 2 \
+--nnodes 2 \
+--node-rank 1 \
+--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
+--tool-call-parser qwen \
+--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
+--mem-frac 0.95
+```
+
+Once the server is running, send requests to the master node:
+
+```bash Command
+curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
+```
+Output:
+```json Config
+{
+  "id": "e82af153da844ee6aed7a27a3187f2f4",
+  "object": "chat.completion",
+  "created": 1771216764,
+  "model": "auto",
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": "The capital of France is **Paris**.\n\n**Additional details:**\n*   It is the largest city in France.\n*   It is located in the north-central part of the country along the Seine River.\n*   Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).",
+        "reasoning_content": null,
+        "tool_calls": null
+      },
+      "logprobs": null,
+      "finish_reason": "stop",
+      "matched_stop": 156895
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 25,
+    "total_tokens": 93,
+    "completion_tokens": 68,
+    "prompt_tokens_details": null,
+    "reasoning_tokens": 0
+  }
+}
+```
+
+For more API usage examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Tool Calling Example
+
+```bash Command
+curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "inclusionAI/Ling-2.5-1T",
+    "messages": [{"role": "user", "content": "Search for the latest news about AI"}],
+    "tools": [{
+      "type": "function",
+      "function": {
+        "name": "search",
+        "description": "Search for information on the internet",
+        "parameters": {
+          "type": "object",
+          "properties": {
+            "query": {"type": "string", "description": "The search query"}
+          },
+          "required": ["query"]
+        }
+      }
+    }],
+    "tool_choice": "auto"
+  }'
+```
+Output:
+```json Config
+{
+  "id": "b968e45c7d414f7482c8ffc0f9c6b688",
+  "object": "chat.completion",
+  "created": 1771216520,
+  "model": "inclusionAI/Ling-2.5-1T",
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": null,
+        "reasoning_content": null,
+        "tool_calls": [
+          {
+            "id": "call_e75f711d8ad840ed9d382c9e",
+            "index": 0,
+            "type": "function",
+            "function": {
+              "name": "search",
+              "arguments": "{\"query\": \"latest news about AI\"}"
+            }
+          }
+        ]
+      },
+      "logprobs": null,
+      "finish_reason": "tool_calls",
+      "matched_stop": null
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 173,
+    "total_tokens": 196,
+    "completion_tokens": 23,
+    "prompt_tokens_details": null,
+    "reasoning_tokens": 0
+  }
+}
+```
+
+## 5. Benchmark
+
+### GSM8K
+
+- Benchmark Command
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py
+```
+
+- Test Result
+```text Output
+Accuracy: 0.960
+Invalid: 0.000
+Latency: 45.410 s
+Output throughput: 560.642 token/s
+```

docs_new/cookbook/autoregressive/InclusionAI/Ring-2.5-1T.mdx

@@ -0,0 +1,266 @@
+---
+title: Ring-2.5-1T
+metatags:
+    description: "Deploy Ring-2.5-1T with SGLang - world's first open-source 1T parameter reasoning model with hybrid linear attention, deep reasoning, and agentic tool calling capabilities."
+tag: NEW
+---
+
+## 1. Model Introduction
+
+[Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities.
+
+**Key Features:**
+
+- **Trillion-Scale Model**: ~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention)
+- **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens
+- **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback
+- **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training
+- **Tool Calling**: Supports function calling with XML-style tool call format
+- **Context Length**: 128K -> 256K (YaRN)
+
+**Available Models:**
+
+- **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T)
+
+**License:** MIT
+
+## 2. SGLang Installation
+
+Ring-2.5-1T requires a specific SGLang Docker image:
+
+```bash Command
+# For H200/B200
+docker pull lmsysorg/sglang:nightly-dev-20260213-a0ebaa64
+
+# For GB200/GB300
+docker pull lmsysorg/sglang:nightly-dev-cu13-20260213-a0ebaa64
+
+# For MI300X/325X
+docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
+
+# For MI355X
+docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
+```
+
+For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
+
+import { Ring251TDeployment } from '/src/snippets/autoregressive/ring-25-1t-deployment.jsx'
+
+<Ring251TDeployment />
+
+### 3.2 Configuration Tips
+
+- The `--trust-remote-code` flag is required for this model due to custom modeling code.
+- The model uses FP8 quantization (compressed-tensors format).
+
+## 4. Model Invocation
+
+Deploy Ring-2.5-1T with the following command (on H200, all features enabled):
+
+```shell Command
+sglang serve \
+  --model-path inclusionAI/Ring-2.5-1T \
+  --tp 8 \
+  --trust-remote-code \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response.
+
+```shell Command
+sglang serve \
+  --model-path inclusionAI/Ring-2.5-1T \
+  --tp 8 \
+  --trust-remote-code \
+  --reasoning-parser deepseek-r1 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="inclusionAI/Ring-2.5-1T",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            print(delta.reasoning_content, end="", flush=True)
+
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+<details>
+<summary>Output Example</summary>
+
+````text Output
+We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution.
+
+We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36.
+
+But we need to present step by step. Also ensure it's clear.
+
+We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36.
+
+But any method is fine.
+
+We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)."
+
+We'll show:
+
+15% = 15/100 = 0.15
+
+Then 0.15 × 240 = 36.
+
+Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36.
+
+Finally, answer: 36.
+
+We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36."
+
+We'll present as a final answer: \boxed{36}.
+
+However, we need to provide step-by-step solution as per instructions. We'll write a full explanation.
+
+We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36.
+
+Alr.
+
+I think that's it.
+
+
+**Step 1:** Write 15% as a fraction or decimal.
+\[ 15\% = \frac{15}{100} = 0.15\]
+
+**Step 2:** Multiply the number (240) by this fraction/decimal.
+\[ 240 \times 0.15 = 36\]
+
+Alternatively, using the fraction:
+\[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\]
+
+**Conclusion:** 15% of 240 is 36.
+
+\[ \boxed{36} \]
+````
+
+</details>
+
+#### 4.2.2 Tool Calling
+
+To enable tool calling, add `--tool-call-parser qwen` when launching the server.
+
+```shell Command
+sglang serve \
+  --model-path inclusionAI/Ring-2.5-1T \
+  --tp 8 \
+  --trust-remote-code \
+  --tool-call-parser qwen \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+response = client.chat.completions.create(
+    model="inclusionAI/Ring-2.5-1T",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools
+)
+
+print(response.choices[0].message.tool_calls)
+```
+
+**Output Example:**
+
+```text Output
+[ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
+```
+
+## 5. Benchmark
+
+### GSM8K
+
+- Deployment Command
+```bash Command
+sglang serve \
+  --model-path inclusionAI/Ring-2.5-1T \
+  --tp-size 8 \
+  --trust-remote-code
+```
+
+- Benchmark Command
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --temperature 1.2 --top-p 0.8 --max-new-tokens 32768 --num-questions 200 --tokenizer-path inclusionAI/Ring-2.5-1T --enable-thinking
+```
+
+- Test Result
+```text Output
+Accuracy: 0.955
+Invalid: 0.010
+Latency: 615.833 s
+Output throughput: 412.360 token/s
+```

docs_new/cookbook/autoregressive/InternLM/Intern-S1.mdx

@@ -0,0 +1,28 @@
+---
+title: Intern-S1
+metatags:
+    description: "Deploy Intern-S1 with SGLang - community contribution guide for InternLM's Intern-S1 model deployment."
+---
+
+## 📝 Community Contribution Welcome
+
+This guide is currently under development. We welcome community contributions!
+
+If you have experience deploying **Intern-S1** with SGLang, please help us complete this documentation.
+
+## 🚀 How to Contribute
+
+```shell Command
+git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
+cd sglang-cookbook
+git checkout -b add-intern-s1-guide
+# Edit this file and submit a PR
+```
+
+## 📚 Reference
+
+- [GLM-4.6V](../GLM/GLM-4.6V)
+
+---
+
+**Let's build this together!** 🌟

docs_new/cookbook/autoregressive/InternVL/InternVL3.5.mdx

@@ -0,0 +1,29 @@
+---
+title: InternVL3.5
+metatags:
+    description: "Deploy InternVL3.5 vision-language model with SGLang - community contribution guide for OpenGVLab's multimodal model."
+---
+
+
+## 📝 Community Contribution Welcome
+
+This guide is currently under development. We welcome community contributions!
+
+If you have experience deploying **InternVL3.5** with SGLang, please help us complete this documentation.
+
+## 🚀 How to Contribute
+
+```shell Command
+git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
+cd sglang-cookbook
+git checkout -b add-internvl3-5-guide
+# Edit this file and submit a PR
+```
+
+## 📚 Reference
+
+- [GLM-4.6V](../GLM/GLM-4.6V)
+
+---
+
+**Let's build this together!** 🌟

docs_new/cookbook/autoregressive/Jina/Jina-reranker-m0.mdx

@@ -0,0 +1,28 @@
+---
+title: Jina-reranker-m0
+metatags:
+    description: "Deploy Jina-reranker-m0 with SGLang - community contribution guide for Jina AI's reranker model deployment."
+---
+
+## 📝 Community Contribution Welcome
+
+This guide is currently under development. We welcome community contributions!
+
+If you have experience deploying **Jina-reranker-m0** with SGLang, please help us complete this documentation.
+
+## 🚀 How to Contribute
+
+```shell Command
+git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
+cd sglang-cookbook
+git checkout -b add-jina-reranker-m0-guide
+# Edit this file and submit a PR
+```
+
+## 📚 Reference
+
+- [DeepSeek-V3.2](../DeepSeek/DeepSeek-V3_2.md)
+
+---
+
+**Let's build this together!** 🌟

docs_new/cookbook/autoregressive/Llama/Llama3.1.mdx

@@ -0,0 +1,644 @@
+---
+title: Llama-3.1
+metatags:
+    description: "Deploy Llama 3.1 (8B/70B/405B) with SGLang - 128K context, tool use, multilingual support, and speculative decoding optimization."
+---
+## 1. Model Introduction
+
+Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time.
+
+These models bring open intelligence to all, with several new features and improvements:
+
+- **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities.
+- **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning.
+- **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools.
+- **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai.
+
+For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models.
+
+import { Llama31Deployment } from "/src/snippets/autoregressive/llama31-deployment.jsx";
+
+<Llama31Deployment />
+### 3.2 Configuration Tips
+
+**Speculative Decoding (NVIDIA GPUs):**
+
+- Using Speculative Decoding for latency-sensitive scenarios:
+  - `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
+  - `--speculative-num-steps 3`: Number of speculative verification rounds
+  - `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
+  - `--speculative-num-draft-tokens 4`: Number of draft tokens per step
+  - `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B).
+
+**AMD GPU Deployment:**
+
+- **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB)
+- **Verified TP Configurations**:
+  - MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1)
+  - MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1)
+- **FP8 Model Variants**:
+  - 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8`
+  - 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV`
+- **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+SGLang exposes an OpenAI-compatible endpoint. First, start the server
+
+```shell Command
+sglang serve \
+  --model-path  Meta-Llama/Llama-3.1-405B-Instruct \
+  --tp 8
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY",
+)
+
+resp = client.chat.completions.create(
+    model="Meta-Llama/Llama-3.1-405B-Instruct",
+    messages=[
+        {"role": "system", "content": "You are a helpful coding assistant."},
+        {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
+    ],
+    temperature=0.2,
+    max_tokens=512,
+)
+
+print(resp.choices[0].message.content)
+```
+
+**Output Example:**
+
+````text Output
+**Exponential Backoff Retry Function in Python**
+=====================================================
+
+Below is a Python function that uses the `requests` library to retry a request with exponential backoff.
+
+```python
+import requests
+import time
+import random
+
+def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60):
+    """
+    Retry a request with exponential backoff.
+
+    Args:
+        url (str): The URL to make the request to.
+        method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.).
+        retries (int): The number of retries to attempt. Defaults to 3.
+        backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1.
+        max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60.
+
+    Returns:
+        The response object from the successful request.
+    """
+
+    delay = 1
+    for attempt in range(retries + 1):
+        try:
+            response = requests.request(method, url)
+            response.raise_for_status()  # Raise an exception for HTTP errors
+            return response
+        except requests.RequestException as e:
+            if attempt < retries:
+                # Calculate the delay for this retry
+                delay = min(delay * backoff_factor, max_delay)
+                # Add a random jitter to the delay to prevent thundering herd problem
+                delay += random.uniform(0, delay * 0.1)
+                # Wait for the calculated delay before retrying
+                time.sleep(delay)
+            else:
+                # If all retries have failed, raise the exception
+                raise e
+...
+````
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Tool Calling
+
+Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled:
+
+```shell Command
+sglang serve \
+  --model-path  Meta-Llama/Llama-3.1-405B-Instruct \
+  --tool-call-parser llama3 \
+  --tp 8
+```
+
+**Python Example**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1")
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the weather in a given location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "city": {
+                        "type": "string",
+                        "description": "The city to find the weather for, e.g. 'San Francisco'",
+                    },
+                    "unit": {
+                        "type": "string",
+                        "description": "The unit to fetch the temperature in",
+                        "enum": ["celsius", "fahrenheit"],
+                    },
+                },
+                "required": ["city", "unit"],
+            },
+        },
+    }
+]
+
+response = client.chat.completions.create(
+    model="meta-llama/Llama-3.1-405B-Instruct",
+    messages=[
+        {
+            "role": "user",
+            "content": "What's the weather like in Boston today?",
+        }
+    ],
+    temperature=0.7,
+    stream=True,
+    tools=tools,
+)
+
+
+arguments = []
+
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#OpenAI-Compatible-API)
+
+**Output Example**
+
+```text Output
+🔧 Tool Call: get_weather
+   Arguments: {"city": "Boston", "unit": "fahrenheit"}
+```
+
+**Handling Tool Call Results**
+After getting the tool call, you can execute the function:
+
+```python Example
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather like in Boston today?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Boston", "unit": "fahrenheit"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Boston", "fahrenheit")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="Meta-Llama/Llama-3.1-405B-Instruct",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside"
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA A100 GPU (8x)
+- Model: Meta-Llama/Llama-3.1-70B
+- Tensor Parallelism: 8
+- sglang version: 0.5.6
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
+
+#### 5.1.1 Standard Scenario Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+sglang serve \
+  --model-path Meta-Llama/Llama-3.1-70B \
+  --tp 8
+```
+
+##### 5.1.1.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+sglang serve \
+  --backend sglang \
+  --model Meta-Llama/Llama-3.1-70B \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  79.81
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4208
+Request throughput (req/s):              0.13
+Input token throughput (tok/s):          76.44
+Output token throughput (tok/s):         52.88
+Peak output token throughput (tok/s):    54.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          129.32
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   7977.81
+Median E2E Latency (ms):                 6373.48
+---------------Time to First Token----------------
+Mean TTFT (ms):                          131.61
+Median TTFT (ms):                        131.77
+P99 TTFT (ms):                           163.88
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          18.63
+Median TPOT (ms):                        18.63
+P99 TPOT (ms):                           18.65
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           18.64
+Median ITL (ms):                         18.64
+P95 ITL (ms):                            18.69
+P99 ITL (ms):                            18.74
+Max ITL (ms):                            21.95
+==================================================
+```
+
+##### 5.1.1.2 Medium Concurrency
+
+```shell Command
+sglang serve \
+  --backend sglang \
+  --model-path Meta-Llama/Llama-3.1-70B \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  79.47
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    38450
+Request throughput (req/s):              1.01
+Input token throughput (tok/s):          499.17
+Output token throughput (tok/s):         513.48
+Peak output token throughput (tok/s):    674.00
+Peak concurrent requests:                20
+Total token throughput (tok/s):          1012.65
+Concurrency:                             13.47
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   13376.67
+Median E2E Latency (ms):                 14130.48
+---------------Time to First Token----------------
+Mean TTFT (ms):                          264.84
+Median TTFT (ms):                        147.02
+P99 TTFT (ms):                           791.93
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          26.09
+Median TPOT (ms):                        26.08
+P99 TPOT (ms):                           34.65
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           25.76
+Median ITL (ms):                         23.95
+P95 ITL (ms):                            24.72
+P99 ITL (ms):                            98.32
+Max ITL (ms):                            478.92
+==================================================
+```
+
+##### 5.1.1.3 High Concurrency
+
+```shell Command
+sglang serve \
+  --backend sglang \
+  --model-path Meta-Llama/Llama-3.1-70B \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  131.64
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    243641
+Request throughput (req/s):              3.80
+Input token throughput (tok/s):          1897.87
+Output token throughput (tok/s):         1919.38
+Peak output token throughput (tok/s):    3100.00
+Peak concurrent requests:                107
+Total token throughput (tok/s):          3817.25
+Concurrency:                             89.70
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   23616.71
+Median E2E Latency (ms):                 22770.44
+---------------Time to First Token----------------
+Mean TTFT (ms):                          245.98
+Median TTFT (ms):                        184.22
+P99 TTFT (ms):                           1251.67
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          47.19
+Median TPOT (ms):                        48.67
+P99 TPOT (ms):                           56.37
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           46.34
+Median ITL (ms):                         33.46
+P95 ITL (ms):                            108.61
+P99 ITL (ms):                            166.11
+Max ITL (ms):                            1107.09
+==================================================
+```
+
+#### 5.1.2 Summarization Scenario Benchmark
+
+##### 5.1.2.1 Low Concurrency
+
+```shell Command
+sglang serve \
+  --backend sglang \
+  --model-path Meta-Llama/Llama-3.1-70B\
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  83.25
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4220
+Request throughput (req/s):              0.12
+Input token throughput (tok/s):          503.77
+Output token throughput (tok/s):         50.69
+Peak output token throughput (tok/s):    54.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          554.46
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   8322.45
+Median E2E Latency (ms):                 6873.36
+---------------Time to First Token----------------
+Mean TTFT (ms):                          395.25
+Median TTFT (ms):                        318.02
+P99 TTFT (ms):                           850.80
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          18.80
+Median TPOT (ms):                        18.81
+P99 TPOT (ms):                           19.03
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           18.83
+Median ITL (ms):                         18.81
+P95 ITL (ms):                            19.06
+P99 ITL (ms):                            19.08
+Max ITL (ms):                            23.08
+==================================================
+```
+
+##### 5.1.2.2 Medium Concurrency
+
+```shell Command
+sglang serve \
+  --backend sglang \
+  --model-path Meta-Llama/Llama-3.1-70B \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  107.12
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total input vision tokens:               0
+Total generated tokens:                  41669
+Total generated tokens (retokenized):    41603
+Request throughput (req/s):              0.75
+Input token throughput (tok/s):          2800.81
+Output token throughput (tok/s):         389.00
+Peak output token throughput (tok/s):    624.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          3189.81
+Concurrency:                             14.18
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   18988.30
+Median E2E Latency (ms):                 20290.66
+---------------Time to First Token----------------
+Mean TTFT (ms):                          603.42
+Median TTFT (ms):                        531.82
+P99 TTFT (ms):                           2607.95
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          36.94
+Median TPOT (ms):                        36.73
+P99 TPOT (ms):                           79.19
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           35.36
+Median ITL (ms):                         25.72
+P95 ITL (ms):                            27.07
+P99 ITL (ms):                            439.74
+Max ITL (ms):                            2529.51
+==================================================
+```
+
+##### 5.1.2.3 High Concurrency
+
+```shell Command
+sglang serve \
+  --backend sglang \
+  --model-path Meta-Llama/Llama-3.1-70B \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  215.66
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total input vision tokens:               0
+Total generated tokens:                  170000
+Total generated tokens (retokenized):    169035
+Request throughput (req/s):              1.48
+Input token throughput (tok/s):          5906.92
+Output token throughput (tok/s):         788.27
+Peak output token throughput (tok/s):    1920.00
+Peak concurrent requests:                69
+Total token throughput (tok/s):          6695.19
+Concurrency:                             60.01
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   40443.85
+Median E2E Latency (ms):                 39813.12
+---------------Time to First Token----------------
+Mean TTFT (ms):                          633.32
+Median TTFT (ms):                        616.38
+P99 TTFT (ms):                           1912.97
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          74.95
+Median TPOT (ms):                        82.85
+P99 TPOT (ms):                           118.46
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           75.08
+Median ITL (ms):                         34.12
+P95 ITL (ms):                            261.18
+P99 ITL (ms):                            828.12
+Max ITL (ms):                            1970.03
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+
+- **Results**:
+
+```text Output
+Accuracy: 0.830
+Invalid: 0.000
+Latency: 11.794 s
+Output throughput: 1406.961 token/s
+```

docs_new/cookbook/autoregressive/Llama/Llama3.3-70B.mdx

@@ -0,0 +1,229 @@
+---
+title: Llama-3.3-70B
+metatags:
+    description: "Deploy Llama-3.3-70B-Instruct with SGLang on AMD GPUs - 128K context, enhanced reasoning, tool calling, and multilingual support."
+---
+## 1. Model Introduction
+
+[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment.
+
+**Key Features:**
+
+- **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1
+- **Tool Calling**: Native support for function calling and tool use scenarios
+- **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai)
+- **Extended Context**: 128K token context window for processing long documents and complex tasks
+- **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X
+
+**License:**
+Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details.
+
+For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models).
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X).
+
+### 3.1 Interactive Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup.
+
+import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
+
+<Llama33Deployment />
+
+### 3.2 Configuration Tips
+
+**AMD GPU Deployment:**
+
+- All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants
+- **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV`
+- **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support
+- **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Tool Calling
+
+Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment:
+
+```shell Command
+python -m sglang.launch_server \
+  --model-path meta-llama/Llama-3.3-70B-Instruct \
+  --tool-call-parser llama3 \
+  --tp 1 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+**Python Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request
+response = client.chat.completions.create(
+    model="meta-llama/Llama-3.3-70B-Instruct",
+    messages=[
+        {"role": "user", "content": "What's the weather in Tokyo?"}
+    ],
+    tools=tools,
+    temperature=0.7
+)
+
+# Check for tool calls
+message = response.choices[0].message
+if message.tool_calls:
+    tool_call = message.tool_calls[0]
+    print(f"Function: {tool_call.function.name}")
+    print(f"Arguments: {tool_call.function.arguments}")
+```
+
+**Handling Tool Call Results:**
+
+```python Example
+# After executing the function, send the result back
+def get_weather(location, unit="celsius"):
+    # Your weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Build conversation with tool result
+messages = [
+    {"role": "user", "content": "What's the weather in Tokyo?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Tokyo", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Tokyo", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="meta-llama/Llama-3.3-70B-Instruct",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!"
+```
+
+#### 4.2.2 Long Context Processing
+
+Leverage the 128K context window for processing long documents:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Example with long document
+long_document = "..." * 10000  # Your long document here
+
+response = client.chat.completions.create(
+    model="meta-llama/Llama-3.3-70B-Instruct",
+    messages=[
+        {"role": "user", "content": f"Summarize this document:\n\n{long_document}"}
+    ],
+    temperature=0.7,
+    max_tokens=1000
+)
+
+print(response.choices[0].message.content)
+```
+
+## 5. Benchmarking
+
+Use the SGLang benchmarking suite to test model performance with different workload patterns:
+
+### 5.1 Basic Benchmark Command
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --dataset-name random \
+  --num-prompts 1000 \
+  --random-input 1024 \
+  --random-output 1024 \
+  --max-concurrency 16
+```
+
+### 5.2 Adjusting Benchmark Parameters
+
+**Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns:
+
+- Short conversations: `--random-input 1024 --random-output 1024`
+- Long outputs: `--random-input 1024 --random-output 8192`
+- Long inputs: `--random-input 8192 --random-output 1024`
+
+**Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios:
+
+- Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100`
+- Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000`
+- High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000`
+
+---
+
+## 📚 Additional Resources
+
+- [Meta Llama Models Repository](https://github.com/meta-llama/llama-models)
+- [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
+- [SGLang Documentation](/)
+- [AMD ROCm Documentation](https://rocm.docs.amd.com/)

docs_new/cookbook/autoregressive/Llama/Llama4.mdx

@@ -0,0 +1,474 @@
+---
+title: Llama 4
+metatags:
+    description: "Deploy Llama 4 Scout and Maverick with SGLang - Meta's latest generation open-source LLMs with industry-leading performance."
+---
+
+import { Llama4ScoutDeployment } from '/src/snippets/autoregressive/llama4-scout-deployment.jsx';
+import { Llama4MaverickDeployment } from '/src/snippets/autoregressive/llama4-maverick-deployment.jsx';
+
+## 1. Model Introduction
+
+[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance.
+
+SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).
+
+Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).
+
+This generation delivers comprehensive upgrades across the board:
+
+The highly capable Llama 4 Maverick with 17B active parameters out of ~400B total, with 128 experts.
+The efficient Llama 4 Scout also has 17B active parameters out of ~109B total, using just 16 experts.
+Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi).
+
+For more details, please refer to the official llama4 Repository:https://www.llama.com/models/llama-4/
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
+
+<Llama4ScoutDeployment />
+
+<Llama4MaverickDeployment />
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Launch the docker
+```shell Command
+docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
+```
+
+```shell Command
+docker run -d -it --ipc=host --network=host --privileged \
+  --cap-add=CAP_SYS_ADMIN \
+  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
+  --group-add video --cap-add=SYS_PTRACE \
+  --security-opt seccomp=unconfined \
+  -v /:/work \
+  -e SHELL=/bin/bash \
+  --name Llama4 \
+  lmsysorg/sglang:v0.5.9-rocm720-mi30x \
+  /bin/bash
+```
+
+#### 4.2.2 Launch the server
+
+### Llama-4-Scout
+8-GPU deployment command:
+
+```bash Command
+sglang serve \
+  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
+  --tp 8 \
+  --context-length 1000000 \
+  --trust-remote-code
+```
+
+### Llama-4-Maverick
+8-GPU deployment command:
+
+```bash Command
+sglang serve \
+  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
+  --tp 8 \
+  --context-length 1000000 \
+  --trust-remote-code
+```
+
+## 5. Benchmark
+### 5.1 Speed Benchmark
+Test Environment:
+
+Hardware: AMD MI300x GPU
+
+Model: Llama-4-Scout
+
+Tensor Parallelism: 8
+
+sglang version: 0.5.9
+
+- **Model Deployment**
+```bash Command
+sglang serve \
+  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
+  --tp 8 \
+  --context-length 1000000 \
+  --trust-remote-code
+```
+
+### 5.1.1 Low Concurrency (Latency-Optimized)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  74.62
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4211
+Request throughput (req/s):              0.14
+Input token throughput (tok/s):          82.88
+Output token throughput (tok/s):         57.42
+Peak output token throughput (tok/s):    146.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          140.20
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   7459.48
+Median E2E Latency (ms):                 4489.77
+---------------Time to First Token----------------
+Mean TTFT (ms):                          4246.98
+Median TTFT (ms):                        68.57
+P99 TTFT (ms):                           48091.05
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          7.49
+Median TPOT (ms):                        7.40
+P99 TPOT (ms):                           7.40
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           7.49
+Median ITL (ms):                         7.49
+P95 ITL (ms):                            7.47
+P99 ITL (ms):                            7.52
+Max ITL (ms):                            10.44
+==================================================
+```
+### 5.1.2 Medium Concurrency (Balanced)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  45.41
+Total input tokens:                      49668
+Total input text tokens:                 49668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40516
+Request throughput (req/s):              2.26
+Input token throughput (tok/s):          1120.46
+Output token throughput (tok/s):         1152.47
+Peak output token throughput (tok/s):    1520.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          2272.84
+Concurrency:                             14.76
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6089.22
+Median E2E Latency (ms):                 6568.80
+---------------Time to First Token----------------
+Mean TTFT (ms):                          124.44
+Median TTFT (ms):                        87.42
+P99 TTFT (ms):                           268.72
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          11.88
+Median TPOT (ms):                        12.00
+P99 TPOT (ms):                           15.49
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.72
+Median ITL (ms):                         10.54
+P95 ITL (ms):                            11.22
+P99 ITL (ms):                            67.88
+Max ITL (ms):                            74.05
+==================================================
+```
+### 5.1.3 High Concurrency (Throughput-Optimized)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  85.84
+Total input tokens:                      249841
+Total input text tokens:                 249841
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    250498
+Request throughput (req/s):              5.84
+Input token throughput (tok/s):          2910.84
+Output token throughput (tok/s):         2944.82
+Peak output token throughput (tok/s):    4100.00
+Peak concurrent requests:                110
+Total token throughput (tok/s):          5854.65
+Concurrency:                             92.24
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   15844.00
+Median E2E Latency (ms):                 15262.56
+---------------Time to First Token----------------
+Mean TTFT (ms):                          204.46
+Median TTFT (ms):                        129.96
+P99 TTFT (ms):                           528.54
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          41.56
+Median TPOT (ms):                        42.90
+P99 TPOT (ms):                           47.48
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           40.99
+Median ITL (ms):                         24.46
+P95 ITL (ms):                            84.46
+P99 ITL (ms):                            87.64
+Max ITL (ms):                            226.06
+==================================================
+```
+
+### 5.2 Speed Benchmark
+Test Environment:
+
+Hardware: AMD MI300x GPU
+
+Model: Llama-4-Maverick
+
+Tensor Parallelism: 8
+
+sglang version: 0.5.9
+
+- **Model Deployment**
+```bash Command
+sglang serve \
+  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
+  --tp 8 \
+  --context-length 1000000 \
+  --trust-remote-code
+```
+
+### 5.2.1 Low Concurrency (Latency-Optimized)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  68.08
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4202
+Request throughput (req/s):              0.15
+Input token throughput (tok/s):          89.62
+Output token throughput (tok/s):         61.99
+Peak output token throughput (tok/s):    168.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          151.61
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6805.62
+Median E2E Latency (ms):                 2733.91
+---------------Time to First Token----------------
+Mean TTFT (ms):                          4296.56
+Median TTFT (ms):                        57.45
+P99 TTFT (ms):                           38633.95
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          5.95
+Median TPOT (ms):                        5.96
+P99 TPOT (ms):                           5.97
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           5.96
+Median ITL (ms):                         5.96
+P95 ITL (ms):                            6.02
+P99 ITL (ms):                            6.08
+Max ITL (ms):                            7.02
+==================================================
+```
+### 5.2.2 Medium Concurrency (Balanced)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  30.72
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40923
+Request throughput (req/s):              2.60
+Input token throughput (tok/s):          1291.39
+Output token throughput (tok/s):         1328.41
+Peak output token throughput (tok/s):    1760.00
+Peak concurrent requests:                22
+Total token throughput (tok/s):          2619.80
+Concurrency:                             13.92
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5345.15
+Median E2E Latency (ms):                 5679.73
+---------------Time to First Token----------------
+Mean TTFT (ms):                          259.30
+Median TTFT (ms):                        72.60
+P99 TTFT (ms):                           1063.45
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.53
+Median TPOT (ms):                        10.22
+P99 TPOT (ms):                           20.27
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           9.99
+Median ITL (ms):                         9.10
+P95 ITL (ms):                            9.87
+P99 ITL (ms):                            55.62
+Max ITL (ms):                            868.54
+==================================================
+```
+### 5.2.3 High Concurrency (Throughput-Optimized)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  90.95
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    251625
+Request throughput (req/s):              5.50
+Input token throughput (tok/s):          2746.77
+Output token throughput (tok/s):         2777.90
+Peak output token throughput (tok/s):    3700.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          5524.67
+Concurrency:                             93.04
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   16924.17
+Median E2E Latency (ms):                 16294.85
+---------------Time to First Token----------------
+Mean TTFT (ms):                          188.19
+Median TTFT (ms):                        128.96
+P99 TTFT (ms):                           534.81
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          33.63
+Median TPOT (ms):                        35.37
+P99 TPOT (ms):                           38.26
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           33.19
+Median ITL (ms):                         27.66
+P95 ITL (ms):                            76.91
+P99 ITL (ms):                            78.82
+Max ITL (ms):                            268.17
+==================================================
+```
+### 5.3 Accuracy Benchmark
+
+#### 5.3.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+ - Llama-4-Scout-17B-16E-Instruct
+```text Output
+Accuracy: 0.945
+Invalid: 0.000
+Latency: 12.731 s
+Output throughput: 1595.418 token/s
+```
+ - Llama-4-Maverick-17B-128E-Instruct
+```text Output
+Accuracy: 0.895
+Invalid: 0.000
+Latency: 9.739 s
+Output throughput: 2405.505 token/s
+```

docs_new/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx

@@ -0,0 +1,723 @@
+---
+title: MiniMax-M2.7
+metatags:
+    description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA and AMD GPUs — model self-evolution, professional software engineering, and native agent teams."
+tag: NEW
+---
+
+## 1. Model Introduction
+
+[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search.
+
+Key highlights:
+
+- **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**.
+- **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems.
+- **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier).
+- **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making.
+
+For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en).
+
+**License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE)
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+**Docker Images by Hardware Platform:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA A100 / H100 / H200 / B200</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA B300 / GB300</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-cu130`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`</td>
+    </tr>
+  </tbody>
+</table>
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities.
+
+import { MiniMaxM27Deployment } from '/src/snippets/autoregressive/minimax-m27-deployment.jsx'
+
+<MiniMaxM27Deployment />
+
+### 3.2 Configuration Tips
+
+**Key Parameters:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tool-call-parser`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tool call parser for function calling support</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-m2`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning parser for thinking mode</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-append-think`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for MiniMax model loading</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.85`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tp`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor parallelism size</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2` / `4` / `8` depending on hardware</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--ep`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Expert parallelism size</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`8` (NVIDIA 8-GPU) or EP=TP (AMD)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--kv-cache-dtype`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>KV cache data type (AMD only)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`fp8_e4m3`</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention backend (AMD only)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
+    </tr>
+  </tbody>
+</table>
+
+**Hardware Requirements: NVIDIA**
+
+- **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4
+- **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8
+
+**Hardware Requirements: NVIDIA GB300**
+
+- **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2
+- **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput
+
+**Hardware Requirements: AMD**
+
+- **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2
+- **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
+- **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+**Deployment Command:**
+
+```bash Command
+sglang serve \
+  --model-path MiniMaxAI/MiniMax-M2.7 \
+  --tp 4 \
+  --tool-call-parser minimax-m2 \
+  --reasoning-parser minimax-append-think \
+  --trust-remote-code \
+  --mem-fraction-static 0.85
+```
+
+**Testing Deployment:**
+
+After startup, you can test the SGLang OpenAI-compatible API with the following command:
+
+```bash Command
+curl http://localhost:30000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "MiniMaxAI/MiniMax-M2.7",
+        "messages": [
+            {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
+            {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
+        ]
+    }'
+```
+
+**Simple Completion Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="MiniMaxAI/MiniMax-M2.7",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Who won the world series in 2020?"}
+    ],
+    max_tokens=1024
+)
+
+print(response.choices[0].message.content)
+```
+
+**Example Output**:
+```text Output
+<think>The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly.
+
+We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context.
+
+We should answer concisely.
+
+Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas.
+
+We must avoid disallowed content, no issue.
+
+Thus final.
+</think>
+
+The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic.
+```
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections:
+
+```bash Command
+sglang serve \
+  --model-path MiniMaxAI/MiniMax-M2.7 \
+  --tp 4 \
+  --reasoning-parser minimax-append-think \
+  --trust-remote-code \
+  --mem-fraction-static 0.85
+```
+
+**Streaming with Thinking Process**
+
+With `minimax-append-think`, the thinking content is wrapped in `<think>...</think>` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="MiniMaxAI/MiniMax-M2.7",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream, separating <think>...</think> from content
+in_think = False
+think_printed_header = False
+content_printed_header = False
+buffer = ""
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+        if delta.content:
+            buffer += delta.content
+
+            while buffer:
+                if in_think:
+                    # Look for closing </think> tag
+                    end_idx = buffer.find("</think>")
+                    if end_idx != -1:
+                        print(buffer[:end_idx], end="", flush=True)
+                        buffer = buffer[end_idx + len("</think>"):]
+                        in_think = False
+                    else:
+                        # Still in thinking, print what we have
+                        print(buffer, end="", flush=True)
+                        buffer = ""
+                else:
+                    # Look for opening <think> tag
+                    start_idx = buffer.find("<think>")
+                    if start_idx != -1:
+                        # Print any content before <think>
+                        before = buffer[:start_idx]
+                        if before:
+                            if not content_printed_header:
+                                print("=============== Content =================", flush=True)
+                                content_printed_header = True
+                            print(before, end="", flush=True)
+                        buffer = buffer[start_idx + len("<think>"):]
+                        in_think = True
+                        if not think_printed_header:
+                            print("=============== Thinking =================", flush=True)
+                            think_printed_header = True
+                    else:
+                        # No <think> tag, print as content
+                        if not content_printed_header and think_printed_header:
+                            print("\n=============== Content =================", flush=True)
+                            content_printed_header = True
+                        print(buffer, end="", flush=True)
+                        buffer = ""
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation.
+
+But also ensure we follow any policy? No issues. Just straightforward.
+
+I'll provide a step-by-step solution.
+
+Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.
+
+Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36.
+
+All good.
+
+=============== Content =================
+
+**Step‑by‑step solution**
+
+1. **Convert the percent to a decimal (or a fraction).**
+
+   15% = 15/100 = 0.15 = 3/20
+
+2. **Multiply the original number (240) by this decimal/fraction.**
+
+   Using the decimal:
+   240 × 0.15 = 36
+
+   Or using the fraction:
+   240 × 3/20 = (240/20) × 3 = 12 × 3 = 36
+
+3. **Result:**
+
+   15% of 240 = **36**
+
+*Check:*
+- 10% of 240 = 24
+- 5% of 240 = 12
+- Adding them: 24 + 12 = 36, which matches the calculation.
+```
+
+**Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `<think>...</think>` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately.
+
+#### 4.2.2 Tool Calling
+
+MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser:
+
+```bash Command
+sglang serve \
+  --model-path MiniMaxAI/MiniMax-M2.7 \
+  --tp 4 \
+  --tool-call-parser minimax-m2 \
+  --reasoning-parser minimax-append-think \
+  --trust-remote-code \
+  --mem-fraction-static 0.85
+```
+
+**Python Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Non-streaming request
+response = client.chat.completions.create(
+    model="MiniMaxAI/MiniMax-M2.7",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools
+)
+
+message = response.choices[0].message
+
+# Check for tool calls
+if message.tool_calls:
+    for tool_call in message.tool_calls:
+        print(f"Tool Call: {tool_call.function.name}")
+        print(f"   Arguments: {tool_call.function.arguments}")
+else:
+    print(message.content)
+```
+
+**Output Example**:
+```text Output
+Tool Call: get_weather
+   Arguments: {"location": "Beijing"}
+```
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="MiniMaxAI/MiniMax-M2.7",
+    messages=messages
+)
+
+print(final_response.choices[0].message.content)
+```
+
+**Output Example:**
+```text Output
+The weather in Beijing is currently 22°C and sunny.
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+**Test Environment**:
+
+- Hardware: 2× NVIDIA GB300 (275GB per die)
+- Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130`
+- Model: MiniMax-M2.7 (FP8)
+- Tensor Parallelism: 2
+- SGLang version: 0.5.10.post1
+
+### 5.1 Accuracy Benchmark
+
+**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills)
+
+**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True`
+
+#### 5.1.1 GPQA Diamond
+
+- Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
+- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking))
+- Evaluation command:
+```bash Command
+ns prepare_data gpqa
+
+ns eval \
+    --cluster=local \
+    --server_type=openai \
+    --model=MiniMaxAI/MiniMax-M2.7 \
+    --server_address=http://localhost:30000/v1 \
+    --output_dir=./m2.7-eval/ \
+    --benchmarks=gpqa:8 \
+    ++prompt_config=eval/aai/mcq-4choices \
+    ++inference.tokens_to_generate=120000 \
+    ++inference.temperature=0.6 \
+    ++inference.top_p=0.95 \
+    ++parse_reasoning=True
+```
+- Test Results:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>84.91%</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3.54%</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**88.89%**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>96.46%</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.2 AIME 2025
+
+- Dataset: AIME 2025 (30 problems)
+- Prompt: `generic/math` (boxed answer format)
+- Evaluation command:
+```bash Command
+ns prepare_data aime25
+
+ns eval \
+    --cluster=local \
+    --server_type=openai \
+    --model=MiniMaxAI/MiniMax-M2.7 \
+    --server_address=http://localhost:30000/v1 \
+    --output_dir=./m2.7-eval/ \
+    --benchmarks=aime25:8 \
+    ++inference.tokens_to_generate=120000 \
+    ++inference.temperature=0.6 \
+    ++inference.top_p=0.95 \
+    ++parse_reasoning=True
+```
+- Test Results:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.50% ± 5.56%</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>2.92%</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**97.08%**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100.00%</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
+    </tr>
+  </tbody>
+</table>
+
+#### 5.1.3 MMLU-Pro
+
+- Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
+- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice)
+- Evaluation command:
+```bash Command
+ns prepare_data mmlu-pro
+
+ns eval \
+    --cluster=local \
+    --server_type=openai \
+    --model=MiniMaxAI/MiniMax-M2.7 \
+    --server_address=http://localhost:30000/v1 \
+    --output_dir=./m2.7-eval/ \
+    --benchmarks=mmlu-pro \
+    ++prompt_config=eval/aai/mcq-10choices \
+    ++inference.tokens_to_generate=32768 \
+    ++inference.temperature=0.0 \
+    ++parse_reasoning=True
+```
+- Test Results:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (greedy)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>69.41%</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>18.75%</td>
+    </tr>
+  </tbody>
+</table>
+
+> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly.
+
+#### 5.1.4 GSM8K Benchmark
+- Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API
+- Test Results:
+```text Output
+GSM8K Results (8-shot CoT)
+Model: MiniMaxAI/MiniMax-M2.7
+Total: 1319
+Correct: 1218
+Accuracy: 92.34%
+```
+
+### 5.2 Speed Benchmark
+
+#### 5.2.1 Low Concurrency
+
+- Benchmark Command:
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model MiniMaxAI/MiniMax-M2.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  34.33
+Total input tokens:                      6101
+Total generated tokens:                  4220
+Request throughput (req/s):              0.29
+Input token throughput (tok/s):          177.71
+Output token throughput (tok/s):         122.92
+Total token throughput (tok/s):          300.63
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3431.21
+Median E2E Latency (ms):                 2742.57
+---------------Time to First Token----------------
+Mean TTFT (ms):                          50.28
+Median TTFT (ms):                        53.85
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          8.02
+Median TPOT (ms):                        8.01
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           8.03
+Median ITL (ms):                         8.02
+==================================================
+```
+
+#### 5.2.2 High Concurrency
+
+- Benchmark Command:
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model MiniMaxAI/MiniMax-M2.7 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  100.20
+Total input tokens:                      249831
+Total generated tokens:                  252662
+Request throughput (req/s):              4.99
+Input token throughput (tok/s):          2493.41
+Output token throughput (tok/s):         2521.66
+Total token throughput (tok/s):          5015.07
+Concurrency:                             90.19
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   18072.69
+Median E2E Latency (ms):                 17761.84
+---------------Time to First Token----------------
+Mean TTFT (ms):                          247.94
+Median TTFT (ms):                        92.05
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          35.75
+Median TPOT (ms):                        36.67
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           35.34
+Median ITL (ms):                         30.55
+==================================================
+```

docs_new/cookbook/autoregressive/MiniMax/MiniMax-M2.mdx

@@ -0,0 +1,541 @@
+---
+title: MiniMax-M2
+metatags:
+    description: "Deploy MiniMax-M2 with SGLang - community contribution guide for MiniMax M2 model deployment."
+---
+
+import { MiniMaxM2Deployment } from '/src/snippets/autoregressive/minimax-m2-deployment.jsx';
+
+## 1. Model Introduction
+
+[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence.
+
+This generation delivers comprehensive upgrades across the board:
+
+- **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally.
+
+- **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages.
+
+- **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps.
+
+- **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks.
+
+For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions. The AMD environment is currently available in SGLang via Docker image install.
+
+### 2.1 AMD Docker
+#### 2.1.1 Launch docker
+```shell Command
+docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
+```
+```shell Command
+docker run -d -it --ipc=host --network=host --privileged \
+  --cap-add=CAP_SYS_ADMIN \
+  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
+  --group-add video --cap-add=SYS_PTRACE \
+  --security-opt seccomp=unconfined \
+  -v /:/work \
+  -e SHELL=/bin/bash \
+  --name Minimax \
+ lmsysorg/sglang:v0.5.9-rocm720-mi30x \
+  /bin/bash
+```
+
+#### 2.1.2 Make modifications inside the docker
+
+```shell Command
+mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \
+   /sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py
+```
+
+#### 2.1.3 Fix torch compile
+Comment out the following line: @torch.compile(dynamic=True, backend=get_compiler_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax_m2.py
+```shell Command
+#@torch.compile(dynamic=True, backend=get_compiler_backend())
+```
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
+
+<MiniMaxM2Deployment />
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+Server Command:
+```shell Command
+    sglang serve \
+    --model-path MiniMaxAI/MiniMax-M2 \
+    --tp-size 4 \
+    --reasoning-parser minimax-append-think \
+    --trust-remote-code \
+    --mem-fraction-static 0.85
+```
+Test Code:
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="MiniMaxAI/MiniMax-M2",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.6,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+Output Example:
+```text Output
+<think>First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction.
+
+The problem is: What is 15% of 240?
+
+To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15.
+
+Therefore, 15% of 240 is 240 times 0.15.
+
+Let me calculate that: 240 × 0.15.
+
+I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100.
+
+Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600.
+
+Then, divide by 100: 3600 / 100 = 36.
+
+So, 15% of 240 is 36.
+
+I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer.
+
+Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36.
+
+All methods confirm it's 36.
+
+The user said "solve this problem step by step," so I should present the steps clearly.
+
+Step 1: Understand that "15% of 240" means 15 per hundred of 240.
+
+Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15.
+
+Step 3: Multiply the number by the decimal: 240 × 0.15.
+
+Step 4: Calculate the multiplication: 240 × 0.15 = 36.
+
+Step 5: Therefore, 15% of 240 is 36.
+
+I should also mention that percentage means per hundred, so it's straightforward.
+
+Finally, I need to box the answer as per the instruction.
+
+So, the final answer is 36.
+</think>
+
+To find 15% of 240, follow these steps:
+
+1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100.
+2. **Convert to a decimal**: 15/100 = 0.15.
+3. **Multiply by the number**: 240 × 0.15.
+4. **Calculate the result**:
+   - 240 × 0.15 = 36.
+
+Alternatively, you can break it down:
+- 10% of 240 is 24 (since 240 ÷ 10 = 24).
+- 5% of 240 is half of 10%, which is 12.
+- Therefore, 15% is 10% + 5% = 24 + 12 = 36.
+
+Both methods confirm the result.
+
+**Answer**: 36
+```
+
+### 4.2.2 Tool Calling
+
+Server Command:
+```shell Command
+sglang serve \
+    --model-path MiniMaxAI/MiniMax-M2 \
+    --tp-size 4 \
+    --tool-call-parser minimax-m2 \
+    --trust-remote-code \
+    --mem-fraction-static 0.85
+```
+Test Code:
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="MiniMaxAI/MiniMax-M2",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+Output Example:
+```text Output
+Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me.
+
+Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me.
+
+Looking at the tool's parameters, I can see it requires:
+1. location - which is required and should be a string representing the city name
+2. unit - which is optional and can be either "celsius" or "fahrenheit"
+
+For the location parameter, I'll use "Beijing" since that's what the user asked about.
+
+For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information.
+
+So I need to make a tool call to get_weather with the following parameters:
+- location: "Beijing"
+- unit: "celsius"
+
+This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions.
+</think>
+
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+## 5. Benchmark
+### 5.1 Speed Benchmark
+**Test Environment**:
+
+- Hardware: AMD MI300X GPU(4x)
+
+- Model: MiniMax-M2
+
+- Tensor Parallelism: 4
+
+- sglang version: 0.5.7
+
+**Model Deployment**:
+
+```bash Command
+sglang serve \
+    --model-path MiniMaxAI/MiniMax-M2 \
+    --tp-size 4 \
+    --trust-remote-code \
+    --mem-fraction-static 0.85
+```
+
+### 5.1.1 Low Concurrency (Latency-Optimized)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model MiniMaxAI/MiniMax-M2 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  138.91
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4220
+Request throughput (req/s):              0.07
+Input token throughput (tok/s):          43.92
+Output token throughput (tok/s):         30.38
+Peak output token throughput (tok/s):    46.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          74.30
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   13887.62
+Median E2E Latency (ms):                 10377.26
+---------------Time to First Token----------------
+Mean TTFT (ms):                          4528.94
+Median TTFT (ms):                        385.23
+P99 TTFT (ms):                           38338.51
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          22.21
+Median TPOT (ms):                        22.24
+P99 TPOT (ms):                           22.25
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           22.23
+Median ITL (ms):                         22.24
+P95 ITL (ms):                            22.35
+P99 ITL (ms):                            22.41
+Max ITL (ms):                            23.64
+==================================================
+```
+
+### 5.1.2 Medium Concurrency (Balanced)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model MiniMaxAI/MiniMax-M2 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  81.07
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40803
+Request throughput (req/s):              0.99
+Input token throughput (tok/s):          489.29
+Output token throughput (tok/s):         503.32
+Peak output token throughput (tok/s):    704.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          992.61
+Concurrency:                             13.74
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   13925.95
+Median E2E Latency (ms):                 14348.75
+---------------Time to First Token----------------
+Mean TTFT (ms):                          532.32
+Median TTFT (ms):                        147.69
+P99 TTFT (ms):                           1978.48
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          27.49
+Median TPOT (ms):                        26.56
+P99 TPOT (ms):                           46.52
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           26.31
+Median ITL (ms):                         23.47
+P95 ITL (ms):                            24.37
+P99 ITL (ms):                            125.10
+Max ITL (ms):                            1192.51
+==================================================
+```
+
+### 5.1.3 High Concurrency (Throughput-Optimized)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model MiniMaxAI/MiniMax-M2 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  153.71
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    250982
+Request throughput (req/s):              3.25
+Input token throughput (tok/s):          1625.33
+Output token throughput (tok/s):         1643.75
+Peak output token throughput (tok/s):    2597.00
+Peak concurrent requests:                107
+Total token throughput (tok/s):          3269.09
+Concurrency:                             91.14
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   28017.24
+Median E2E Latency (ms):                 26865.28
+---------------Time to First Token----------------
+Mean TTFT (ms):                          387.41
+Median TTFT (ms):                        183.90
+P99 TTFT (ms):                           1192.44
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          55.23
+Median TPOT (ms):                        57.84
+P99 TPOT (ms):                           70.23
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           54.79
+Median ITL (ms):                         39.01
+P95 ITL (ms):                            143.10
+P99 ITL (ms):                            150.46
+Max ITL (ms):                            986.14
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+#### 5.2.1 GSM8K Benchmark
+
+- **Server Command**:
+```shell Command
+sglang serve \
+    --model-path MiniMaxAI/MiniMax-M2 \
+    --tp-size 4 \
+    --trust-remote-code \
+    --mem-fraction-static 0.85
+```
+
+- **Benchmark Command**:
+```shell Command
+  python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+- **Result**:
+  - MiniMax-M2
+```text Output
+ Accuracy: 0.950
+ Invalid: 0.000
+ Latency: 15.120 s
+ Output throughput: 1306.711 token/s
+```

docs_new/cookbook/autoregressive/Mistral/Devstral-2.mdx

@@ -0,0 +1,518 @@
+---
+title: Devstral 2 (Mistral)
+metatags:
+    description: "Deploy Devstral 2 agentic coding models with SGLang - optimized for tool use, codebase exploration, and multi-file edits with 256K context."
+---
+
+## 1. Model Introduction
+
+**Devstral 2** is an agentic LLM family for software engineering tasks. It is designed for agentic workflows such as tool use, codebase exploration, and multi-file edits, and achieves strong performance on **SWE-bench**.
+
+The **Devstral 2 Instruct** checkpoints are instruction-tuned **FP8** models, making them a good fit for chat, tool-using agents, and instruction-following SWE workloads.
+
+**Key Features:**
+
+- **Agentic coding**: Optimized for tool-driven coding and software engineering agents
+- **Improved performance**: A step up compared to earlier Devstral models
+- **Better generalization**: More robust across diverse prompts and coding environments
+- **Long context**: Up to a **256K** context window
+
+**Use Cases:**
+AI code assistants, agentic coding, and software engineering tasks that require deep codebase understanding and tool integration.
+
+For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), please reach out to Mistral.
+
+**Models:**
+
+- **Collection**: [mistralai/devstral-2 (Hugging Face)](https://huggingface.co/collections/mistralai/devstral-2)
+- **FP8 Instruct**:
+  - **[mistralai/Devstral-2-123B-Instruct-2512](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512)**
+  - **[mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512)**
+
+---
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+<Warning title="Transformers version requirement">
+Devstral 2 requires a recent `transformers`. Please verify `transformers >= 5.0.0.rc`:
+
+```shell Command
+python -c "import transformers; print(transformers.__version__)"
+```
+
+If your version is lower, upgrade:
+
+```shell Command
+pip install -U --pre "transformers>=5.0.0rc0"
+```
+</Warning>
+
+---
+
+## 3. Model Deployment
+
+### 3.1 Basic configuration
+
+**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Devstral Small 2 (24B) or Devstral 2 (123B).
+
+<Note>
+The TP size is set to the minimum required for the selected model size.
+</Note>
+
+
+import { Devstral2Deployment } from "/src/snippets/autoregressive/devstral-2-deployment.jsx";
+
+<Devstral2Deployment />
+
+### 3.2 Configuration tips
+
+- **Context length vs memory**: Devstral 2 advertises a long context window; if you are memory-constrained, start by lowering `--context-length` (for example `32768`) and increase once things are stable.
+- **FP8 checkpoints**: Both Devstral Small 2 and Devstral 2 are published as **FP8** weights. If you hit kernel / dtype issues, try a newer SGLang build and recent CUDA drivers.
+
+---
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage (OpenAI-Compatible API)
+
+SGLang exposes an OpenAI-compatible endpoint. Example:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+resp = client.chat.completions.create(
+    model="mistralai/Devstral-Small-2-24B-Instruct-2512",
+    messages=[
+        {"role": "system", "content": "You are a helpful coding assistant."},
+        {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
+    ],
+    temperature=0.2,
+    max_tokens=512,
+)
+
+print(resp.choices[0].message.content)
+```
+
+**Output Example:**
+
+```text Output
+  Here's a Python function that implements exponential backoff for retrying a request. This function uses the `requests` library to make HTTP requests and includes error handling for common HTTP and connection errors.
+
+  ```python
+  import time
+  import requests
+  from requests.exceptions import RequestException
+
+  def retry_with_exponential_backoff(
+      url,
+      max_retries=3,
+      initial_delay=1,
+      backoff_factor=2,
+      method="GET",
+      **kwargs
+  ):
+      """
+      Retry a request with exponential backoff.
+
+      Parameters:
+      - url: The URL to request.
+      - max_retries: Maximum number of retry attempts (default: 3).
+      - initial_delay: Initial delay in seconds (default: 1).
+      - backoff_factor: Multiplier for the delay between retries (default: 2).
+      - method: HTTP method to use (default: "GET").
+      - **kwargs: Additional arguments to pass to the request function (e.g., headers, data, etc.).
+
+      Returns:
+      - Response object if the request succeeds.
+      - Raises an exception if all retries fail.
+      """
+      retry_count = 0
+      delay = initial_delay
+
+      while retry_count < max_retries:
+          try:
+              response = requests.request(method, url, **kwargs)
+              # Check if the response status code indicates success
+              if response.status_code < 400:
+                  return response
+              else:
+                  raise RequestException(f"HTTP {response.status_code}: {response.text}")
+
+          except RequestException as e:
+              if retry_count == max_retries - 1:
+                  raise Exception(f"All retries failed. Last error: {e}")
+
+              print(f"Attempt {retry_count + 1} failed. Retrying in {delay} seconds...")
+              time.sleep(delay)
+...
+```
+
+### 4.2 Tool calling (optional)
+
+Devstral 2 supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model mistralai/Devstral-2-123B-Instruct-2512 \
+  --tp 2 \
+  --tool-call-parser mistral
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="mistralai/Devstral-2-123B-Instruct-2512",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing"}
+```
+
+
+## AMD GPU Support
+
+## 1. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+
+### 1.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 1.2 Advanced Usage
+
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path mistralai/Devstral-2-123B-Instruct-2512 \
+  --tp 8 \
+  --trust-remote-code \
+  --port 8888
+```
+
+## 2.Benchmark
+
+### 5.1 Benchmark Commands
+
+**Scenario 1: Chat (1K/1K) - Most Important**
+
+- **Model Deployment**
+
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path mistralai/Devstral-2-123B-Instruct-2512 \
+  --tp 8 \
+  --trust-remote-code \
+  --port 8888
+```
+
+- Low Concurrency (Latency-Optimized)
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model mistralai/Devstral-2-123B-Instruct-2512 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf \
+  --port 8888
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  94.30
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4206
+Request throughput (req/s):              0.11
+Input token throughput (tok/s):          64.70
+Output token throughput (tok/s):         44.75
+Peak output token throughput (tok/s):    82.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          109.44
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   9427.59
+Median E2E Latency (ms):                 5637.23
+---------------Time to First Token----------------
+Mean TTFT (ms):                          4253.85
+Median TTFT (ms):                        116.95
+P99 TTFT (ms):                           37764.48
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          12.28
+Median TPOT (ms):                        12.29
+P99 TPOT (ms):                           12.30
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           12.29
+Median ITL (ms):                         12.29
+P95 ITL (ms):                            12.38
+P99 ITL (ms):                            12.42
+Max ITL (ms):                            12.90
+==================================================
+```
+
+- Medium Concurrency (Balanced)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model mistralai/Devstral-2-123B-Instruct-2512 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf \
+  --port 8888
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  52.11
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40761
+Request throughput (req/s):              1.54
+Input token throughput (tok/s):          761.31
+Output token throughput (tok/s):         783.13
+Peak output token throughput (tok/s):    1120.00
+Peak concurrent requests:                20
+Total token throughput (tok/s):          1544.44
+Concurrency:                             13.60
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   8856.19
+Median E2E Latency (ms):                 9314.71
+---------------Time to First Token----------------
+Mean TTFT (ms):                          398.80
+Median TTFT (ms):                        127.81
+P99 TTFT (ms):                           1500.32
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          17.32
+Median TPOT (ms):                        16.90
+P99 TPOT (ms):                           32.78
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           16.61
+Median ITL (ms):                         14.26
+P95 ITL (ms):                            15.07
+P99 ITL (ms):                            114.46
+Max ITL (ms):                            1224.45
+==================================================
+```
+
+- High Concurrency (Throughput-Optimized)
+
+```bash Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --model mistralai/Devstral-2-123B-Instruct-2512 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf \
+  --port 8888
+```
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  116.08
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    252523
+Request throughput (req/s):              4.31
+Input token throughput (tok/s):          2152.21
+Output token throughput (tok/s):         2176.60
+Peak output token throughput (tok/s):    3600.00
+Peak concurrent requests:                107
+Total token throughput (tok/s):          4328.81
+Concurrency:                             92.42
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   21456.71
+Median E2E Latency (ms):                 20126.82
+---------------Time to First Token----------------
+Mean TTFT (ms):                          291.60
+Median TTFT (ms):                        199.24
+P99 TTFT (ms):                           866.02
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          42.42
+Median TPOT (ms):                        45.18
+P99 TPOT (ms):                           53.32
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           41.97
+Median ITL (ms):                         27.59
+P95 ITL (ms):                            130.43
+P99 ITL (ms):                            137.87
+Max ITL (ms):                            616.73
+==================================================
+```
+
+
+
+#### 5.2 Understanding the Results
+
+**Key Metrics:**
+
+- **Request Throughput (req/s)**: Number of requests processed per second
+- **Output Token Throughput (tok/s)**: Total tokens generated per second
+- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
+- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
+- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
+
+**Why These Configurations Matter:**
+
+- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
+- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
+- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
+- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
+
+**Interpreting Results:**
+
+- Compare your results against baseline numbers for your hardware
+- Higher throughput at same latency = better performance
+- Lower TTFT = more responsive user experience
+- Lower TPOT = faster generation speed
+
+### 5.3 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.3.1 GSM8K Benchmark
+
+- Benchmark Command
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py \
+  --num-shots 8 \
+  --num-questions 1316 \
+  --parallel 1316 \
+  --port 8888
+```
+
+**Test Results:**
+
+```text Output
+Accuracy: 0.922
+Invalid: 0.000
+Latency: 35.800 s
+Output throughput: 4507.697 token/s
+```

docs_new/cookbook/autoregressive/Mistral/Ministral-3.mdx

@@ -0,0 +1,288 @@
+---
+title: Ministral-3
+metatags:
+    description: "Deploy Mistral 3 with SGLang - deployment configurations and usage patterns for Mistral's latest model."
+---
+
+import { Ministral3Deployment } from '/src/snippets/autoregressive/ministral-3-deployment.jsx';
+
+## 1. Model Introduction
+The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.
+
+The Ministral 3 14B Instruct model offers the following capabilities:
+
+Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
+Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
+System Prompt: Maintains strong adherence and support for system prompts.
+Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
+Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
+Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
+Large Context Window: Supports a 256k context window.
+
+For further details, please refer to the [official documentation](https://github.com/mistralai)
+
+## 2. SGLang Installation
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
+
+<Ministral3Deployment />
+
+### 3.2 Configuration Tips
+**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.
+
+**Pre-installation steps**: Adding the following steps after launching the docker
+```shell Command
+pip install mistral-common --upgrade
+pip install transformers==5.0.0.rc0
+```
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Launch the docker
+```shell Command
+docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
+```
+
+```shell Command
+docker run -d -it --ipc=host --network=host --privileged \
+  --cap-add=CAP_SYS_ADMIN \
+  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
+  --group-add video --cap-add=SYS_PTRACE \
+  --security-opt seccomp=unconfined \
+  -v /:/work \
+  -e SHELL=/bin/bash \
+  --name Ministral \
+ lmsysorg/sglang:v0.5.9-rocm720-mi30x \
+  /bin/bash
+```
+
+#### 4.2.2 Launch the server
+```shell Command
+sglang serve \
+  --model-path mistralai/Ministral-3-14B-Instruct-2512 \
+  --tp 1 \
+  --trust-remote-code
+```
+
+## 5. Benchmark
+
+This section uses **industry-standard configurations** for comparable benchmark results.
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: MI300X GPU (8x)
+- Model: mistralai/Ministral-3-14B-Instruct-2512
+- Tensor Parallelism: 1
+- SGLang Version: 0.5.7
+
+- Model Deployment Command:
+
+```bash Command
+sglang serve \
+  --model-path mistralai/Ministral-3-14B-Instruct-2512 \
+  --tp 1 \
+  --trust-remote-code
+```
+
+#####  Low Concurrency
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model mistralai/Ministral-3-14B-Instruct-2512 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  65.08
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4218
+Request throughput (req/s):              0.15
+Input token throughput (tok/s):          93.75
+Output token throughput (tok/s):         64.84
+Peak output token throughput (tok/s):    151.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          158.59
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6505.51
+Median E2E Latency (ms):                 3037.37
+---------------Time to First Token----------------
+Mean TTFT (ms):                          3709.33
+Median TTFT (ms):                        53.72
+P99 TTFT (ms):                           33320.77
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          6.63
+Median TPOT (ms):                        6.64
+P99 TPOT (ms):                           6.66
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.64
+Median ITL (ms):                         6.65
+P95 ITL (ms):                            6.75
+P99 ITL (ms):                            6.82
+Max ITL (ms):                            8.45
+==================================================
+```
+
+##### Medium Concurrency
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model mistralai/Ministral-3-14B-Instruct-2512 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  31.20
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40783
+Request throughput (req/s):              2.56
+Input token throughput (tok/s):          1271.38
+Output token throughput (tok/s):         1307.82
+Peak output token throughput (tok/s):    1760.00
+Peak concurrent requests:                22
+Total token throughput (tok/s):          2579.20
+Concurrency:                             13.72
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5351.07
+Median E2E Latency (ms):                 5626.45
+---------------Time to First Token----------------
+Mean TTFT (ms):                          280.87
+Median TTFT (ms):                        68.16
+P99 TTFT (ms):                           1194.79
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.47
+Median TPOT (ms):                        10.10
+P99 TPOT (ms):                           20.00
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           9.96
+Median ITL (ms):                         9.10
+P95 ITL (ms):                            9.87
+P99 ITL (ms):                            51.39
+Max ITL (ms):                            888.63
+==================================================
+```
+
+##### High Concurrency
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model mistralai/Ministral-3-14B-Instruct-2512 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  88.75
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    252547
+Request throughput (req/s):              5.63
+Input token throughput (tok/s):          2815.01
+Output token throughput (tok/s):         2846.91
+Peak output token throughput (tok/s):    4271.00
+Peak concurrent requests:                110
+Total token throughput (tok/s):          5661.93
+Concurrency:                             93.04
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   16514.45
+Median E2E Latency (ms):                 15834.45
+---------------Time to First Token----------------
+Mean TTFT (ms):                          148.57
+Median TTFT (ms):                        99.15
+P99 TTFT (ms):                           455.86
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          32.93
+Median TPOT (ms):                        34.73
+P99 TPOT (ms):                           38.05
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           32.45
+Median ITL (ms):                         27.30
+P95 ITL (ms):                            71.73
+P99 ITL (ms):                            73.45
+Max ITL (ms):                            328.10
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+Document model accuracy on standard benchmarks:
+
+#### 5.2.1 GSM8K Benchmark
+
+- Benchmark Command
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py \
+  --num-shots 8 \
+  --num-questions 1316 \
+  --parallel 1316
+```
+
+**Test Results:**
+
+```text Output
+Accuracy: 0.959
+Invalid: 0.000
+Latency: 29.185 s
+Output throughput: 4854.672 token/s
+```

docs_new/cookbook/autoregressive/Mistral/Mistral-Small-4.mdx

@@ -0,0 +1,393 @@
+---
+title: Mistral Small 4
+metatags:
+    description: "Deploy Mistral Small 4 with SGLang - unified hybrid model combining instruct, reasoning, and agentic capabilities with multimodal support."
+---
+
+import { MistralSmall4Deployment } from '/src/snippets/autoregressive/mistral-small-4-deployment.jsx';
+
+## 1. Model Introduction
+
+**Mistral Small 4** is a powerful hybrid model from Mistral AI that unifies the capabilities of three different model families — **Instruct**, **Reasoning** (formerly called Magistral), and **Agentic (formerly called Devstral)** — into a single, unified model.
+
+With its multimodal capabilities, efficient MoE architecture, and flexible mode switching, Mistral Small 4 is a versatile general-purpose model for virtually any task. In a latency-optimized setup, it achieves a 40% reduction in end-to-end completion time; in a throughput-optimized setup, it delivers 3× more requests per second compared to Mistral Small 3.
+
+**Key Features:**
+
+- **Hybrid Reasoning**: Switch between instant reply mode and deep reasoning/thinking mode — reasoning effort is configurable per request
+- **Vision**: Accepts both text and image inputs, providing insights based on visual content
+- **Function Calling**: Native tool calling and JSON output support with best-in-class agentic capabilities
+- **Multilingual**: Supports dozens of languages including English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more
+- **Context Window**: 256K context window
+- **Efficient MoE**: 119B total parameters, 128 experts, 4 active per token (6.5B activated parameters)
+- **Apache 2.0 License**: Open-source, usable and modifiable for commercial and non-commercial purposes
+- Reasoning effort supported are only **"none" and "high"**
+
+**Architecture:**
+
+- Same general architecture as Mistral 3
+- MoE: 128 experts, 4 active per token
+- 119B total parameters, 6.5B activated per token
+- Multimodal input: text + image
+
+**Models:**
+
+- **[mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)** (FP8)
+- **[mistralai/Mistral-Small-4-119B-2603-NVFP4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)**
+- **[mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603)** — same architecture, use the same launch commands as Mistral-Small-4-119B-2603
+- **[mistralai/Mistral-Small-4-119B-2603-eagle](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)** — EAGLE speculative decoding weights for faster inference
+
+---
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+<Info>
+Mistral Small 4 support landed in [sgl-project/sglang#20708](https://github.com/sgl-project/sglang/pull/20708) and has been merged into `main`. A model-specific Docker image is no longer required. Use the standard SGLang installation methods from the [official installation guide](../../../docs/get-started/installation).
+</Info>
+
+---
+
+## 3. Model Deployment
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Small 4.
+
+<MistralSmall4Deployment />
+
+### 3.2 Configuration Tips
+
+- **Tensor Parallelism**: Mistral Small 4 FP8 (~119 GB) requires tp=2 on Hopper (H100/H200), tp=1 on Blackwell (B200/B300). NVFP4 (~60 GB, Blackwell only) runs with tp=1.
+- **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
+- **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
+- **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
+- **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
+- **Speculative decoding (EAGLE)**: Enable with `--speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle` using the [EAGLE weights](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) for lower latency.
+
+---
+
+## 4. Model Invocation
+
+### 4.1 Thinking Mode
+
+Mistral Small 4 is a hybrid reasoning model. By default, it does not produce a default reasoning response. Use `--reasoning_effort high` to toggle reasoning on.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Small-4-119B-2603",
+    messages=[
+        {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
+    ],
+    extra_body={"reasoning_effort": "high"},
+)
+
+print("Reasoning:", response.choices[0].message.reasoning_content)
+print("Answer:", response.choices[0].message.content)
+```
+
+**Output:**
+
+```text Output
+Reasoning: First, I'll break down the problem into two parts: the multiplication and
+the division. According to the order of operations (PEMDAS/BODMAS), multiplication and
+division are performed from left to right before addition.
+
+17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391
+144 / 12 = 12
+
+Finally, add the results: 391 + 12 = 403
+
+Answer: The solution to the problem is as follows:
+
+1. First, perform the multiplication: 17 × 23.
+   - 17 × 20 = 340
+   - 17 × 3 = 51
+   - 340 + 51 = 391
+
+2. Then, perform the division: 144 / 12 = 12.
+
+3. Finally, add the results:
+   - 391 + 12 = 403
+
+**Answer:** \boxed{403}
+```
+
+### 4.2 Instruct Mode (Reasoning Off)
+
+To skip the reasoning trace and get a fast direct response, set `reasoning_effort` to `"none"`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Small-4-119B-2603",
+    messages=[
+        {"role": "user", "content": "Write a Python function to reverse a string."},
+    ],
+    extra_body={"reasoning_effort": "none"},
+)
+
+print(response.choices[0].message.content)
+```
+
+**Output:**
+
+````text Output
+# Python Function to Reverse a String
+
+Here are several ways to write a Python function to reverse a string:
+
+## Method 1: Using String Slicing (Most Pythonic)
+```python
+def reverse_string(s):
+    """Reverse a string using slicing."""
+    return s[::-1]
+```
+
+## Method 2: Using a Loop
+```python Example
+def reverse_string(s):
+    """Reverse a string using a loop."""
+    reversed_str = ""
+    for char in s:
+        reversed_str = char + reversed_str
+    return reversed_str
+```
+
+## Method 3: Using reversed() function
+```python Example
+def reverse_string(s):
+    """Reverse a string using reversed() function."""
+    return ''.join(reversed(s))
+```
+
+The first method using string slicing (`s[::-1]`) is generally the most efficient and
+recommended approach in Python.
+
+Example usage:
+```python Example
+original = "Hello, World!"
+reversed_str = reverse_string(original)
+print(reversed_str)  # Output: "!dlroW ,olleH"
+```
+````
+
+### 4.3 Streaming with Reasoning
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+stream = client.chat.completions.create(
+    model="mistralai/Mistral-Small-4-119B-2603",
+    messages=[
+        {"role": "user", "content": "Explain the difference between async and threading in Python."},
+    ],
+    extra_body={"reasoning_effort": "high"},
+    stream=True,
+)
+
+print("=== Reasoning ===")
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
+        print(delta.reasoning_content, end="", flush=True)
+    elif delta.content:
+        print("\n=== Response ===")
+        print(delta.content, end="", flush=True)
+print()
+```
+
+**Output:**
+
+```text Output
+=== Reasoning ===
+Okay, the user is asking about the difference between async and threading in Python.
+I need to break this down clearly, covering the key aspects of both, like their
+purposes, performance characteristics, and use cases...
+=== Response ===
+In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency
+models, each suited for specific use cases. Here's a breakdown of their key differences:
+
+### 1. Model of Concurrency
+- **Threading**: Based on preemptive multitasking using OS threads.
+- **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield...
+```
+
+### 4.4 Tool Calling
+
+Mistral Small 4 supports native function calling. Enable with `--tool-call-parser mistral`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a city",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {"type": "string", "description": "City name"},
+                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
+                },
+                "required": ["location"],
+            },
+        },
+    }
+]
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Small-4-119B-2603",
+    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
+    tools=tools,
+    tool_choice="auto",
+)
+
+tool_calls = response.choices[0].message.tool_calls
+for tc in tool_calls:
+    print(f"Tool: {tc.function.name}")
+    print(f"Args: {tc.function.arguments}")
+```
+
+**Output:**
+
+```text Output
+Tool: get_weather
+Args: {"location": "Paris"}
+```
+
+### 4.5 Vision (Image Input)
+
+Mistral Small 4 accepts image inputs alongside text:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+response = client.chat.completions.create(
+    model="mistralai/Mistral-Small-4-119B-2603",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": "Describe what you see in this image."},
+                {
+                    "type": "image_url",
+                    "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
+                },
+            ],
+        }
+    ],
+)
+
+print(response.choices[0].message.content)
+```
+
+**Output:**
+
+```text Output
+The image is a copyright symbol, represented by a stylized version of the lowercase
+letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and
+the circle is orange. The design is simple yet striking, using oval and elliptical
+shapes to create a distinct symbol which signifies copyright protection.
+```
+
+---
+
+## 5. Benchmarks
+
+### 5.1 Accuracy Benchmarks
+
+#### GSM8K
+
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --port 30000
+```
+
+**Results:**
+
+```text Output
+TODO
+```
+
+#### MMLU
+
+```bash Command
+python3 benchmark/mmlu/bench_sglang.py --port 30000
+```
+
+**Results:**
+
+```text Output
+TODO
+```
+
+### 5.2 Speed Benchmarks
+
+#### Latency (Low Concurrency)
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --random-input-len 1024 \
+  --random-output-len 512 \
+  --port 30000
+```
+
+**Results:**
+
+```text Output
+TODO
+```
+
+#### Throughput (High Concurrency)
+
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --num-prompts 1000 \
+  --max-concurrency 100 \
+  --random-input-len 1024 \
+  --random-output-len 512 \
+  --port 30000
+```
+
+**Results:**
+
+```text Output
+TODO
+```

docs_new/cookbook/autoregressive/Moonshotai/Kimi-K2.mdx

@@ -0,0 +1,520 @@
+---
+title: Kimi-K2
+metatags:
+    description: "Deploy Kimi-K2 MoE model with SGLang - 1T total parameters, 32B active, step-by-step reasoning and tool calling capabilities."
+---
+
+import { KimiK2Deployment } from '/src/snippets/autoregressive/kimi-k2-deployment.jsx';
+
+## 1. Model Introduction
+
+[Kimi-K2](https://moonshotai.github.io/Kimi-K2/) is a state-of-the-art MoE language model by Moonshot AI with 32B activated parameters and 1T total parameters.
+
+**Model Variants:**
+
+- **[Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)**: Post-trained model optimized for general-purpose chat and agentic tasks. Compatible with vLLM, SGLang, KTransformers, and TensorRT-LLM.
+- **[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)**: Advanced thinking model with step-by-step reasoning and tool calling. Native INT4 quantization with 256k context window. Ideal for complex reasoning and multi-step tool use.
+- **ROCm Support**: Compatible with AMD MI300X GPUs via SGLang (verified).
+
+For details, see [official documentation](https://github.com/MoonshotAI/Kimi-K2) and [technical report](https://www.arxiv.org/abs/2507.20534).
+
+## 2. SGLang Installation
+
+Refer to the [official SGLang installation guide](../../../docs/get-started/installation).
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and capabilities.
+
+<KimiK2Deployment />
+
+### 3.2 Configuration Tips
+
+- **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory.
+- **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism).
+- **Data Parallel (DP)**: Enable with `--dp 4 --enable-dp-attention` for production throughput.
+- **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+).
+- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content.
+- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
+- **AMD GPU**: Set `SGLANG_ROCM_FUSED_DECODE_MLA=0` before launching AMD GPU.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+See [Basic API Usage](../../../docs/get-started/quickstart).
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+Enable reasoning parser for Kimi-K2-Thinking:
+
+```shell Command
+python -m sglang.launch_server \
+  --model moonshotai/Kimi-K2-Thinking \
+  --reasoning-parser kimi_k2 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="moonshotai/Kimi-K2-Thinking",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.6,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+  The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step.
+
+Step 1: Understand what "percent" means.
+- "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15.
+
+Step 2: Convert the percentage to a decimal.
+- 15% = 15 / 100 = 0.15
+
+Step 3: Multiply the decimal by the number.
+- 0.15 * 240
+
+Step 4: Perform the multiplication.
+- 0.15 * 240 = (15/100) * 240
+- = 15 * 240 / 100
+- = 3600 / 100
+- = 36
+
+Alternatively, I can calculate it directly:
+- 0.15 * 240
+- 15 * 240 = 3600
+- 3600 / 100 = 36
+
+Or, break it down:
+- 10% of 240 = 24
+- 5% of 240 = half of 10% = 12
+- 15% of 240 = 10% + 5% = 24 + 12 = 36
+
+I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying.
+
+Let me structure the answer:
+1. Convert the percentage to a decimal.
+2. Multiply the decimal by the number.
+3. Show the calculation.
+4. State the final answer.
+
+This is simple and easy to follow.
+=============== Content =================
+ Here is the step-by-step solution:
+
+**Step 1: Convert the percentage to a decimal**
+15% means 15 per 100, which is 15 ÷ 100 = **0.15**
+
+**Step 2: Multiply the decimal by the number**
+0.15 × 240
+
+**Step 3: Calculate the result**
+0.15 × 240 = **36**
+
+**Answer:** 15% of 240 is **36**.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.2 Tool Calling
+
+Kimi-K2-Instruct and Kimi-K2-Thinking support tool calling capabilities. Enable the tool call parser during deployment:
+
+**Deployment Command:**
+
+```shell Command
+python -m sglang.launch_server \
+  --model moonshotai/Kimi-K2-Instruct \
+  --tool-call-parser kimi_k2 \
+  --tp 8 \
+  --trust-remote-code \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="moonshotai/Kimi-K2-Thinking",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+  The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you.
+=============== Content =================
+
+  🔧 Tool Call: get_weather
+   Arguments: {"location":"Beijing"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="moonshotai/Kimi-K2-Thinking",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (8x)
+- Model: Kimi-K2-Instruct
+- sglang version: 0.5.6.post1
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+    --model-path moonshotai/Kimi-K2-Instruct \
+    --tp 8 \
+    --dp 4 \
+    --enable-dp-attention \
+    --trust-remote-code \
+    --host 0.0.0.0 \
+    --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model moonshotai/Kimi-K2-Instruct\
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- **Test Results**:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  44.93
+Total input tokens:                      1951
+Total input text tokens:                 1951
+Total input vision tokens:               0
+Total generated tokens:                  2755
+Total generated tokens (retokenized):    2748
+Request throughput (req/s):              0.22
+Input token throughput (tok/s):          43.42
+Output token throughput (tok/s):         61.32
+Peak output token throughput (tok/s):    64.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          104.74
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   4489.56
+Median E2E Latency (ms):                 4994.53
+---------------Time to First Token----------------
+Mean TTFT (ms):                          141.22
+Median TTFT (ms):                        158.28
+P99 TTFT (ms):                           166.90
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          18.40
+Median TPOT (ms):                        15.63
+P99 TPOT (ms):                           39.88
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           15.78
+Median ITL (ms):                         15.76
+P95 ITL (ms):                            16.36
+P99 ITL (ms):                            16.59
+Max ITL (ms):                            19.94
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+    --model-path moonshotai/Kimi-K2-Instruct \
+    --tp 8 \
+    --dp 4 \
+    --ep 4 \
+    --enable-dp-attention \
+    --trust-remote-code \
+    --host 0.0.0.0 \
+    --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model moonshotai/Kimi-K2-Instruct\
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+- **Test Results**:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  174.11
+Total input tokens:                      296642
+Total input text tokens:                 296642
+Total input vision tokens:               0
+Total generated tokens:                  193831
+Total generated tokens (retokenized):    168687
+Request throughput (req/s):              5.74
+Input token throughput (tok/s):          1703.73
+Output token throughput (tok/s):         1113.25
+Peak output token throughput (tok/s):    2383.00
+Peak concurrent requests:                112
+Total token throughput (tok/s):          2816.97
+Concurrency:                             89.60
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   15601.09
+Median E2E Latency (ms):                 10780.52
+---------------Time to First Token----------------
+Mean TTFT (ms):                          457.42
+Median TTFT (ms):                        221.62
+P99 TTFT (ms):                           2475.32
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          97.23
+Median TPOT (ms):                        85.61
+P99 TPOT (ms):                           435.95
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           78.61
+Median ITL (ms):                         43.66
+P95 ITL (ms):                            169.53
+P99 ITL (ms):                            260.91
+Max ITL (ms):                            1703.21
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- Server Command
+
+```shell Command
+python3 -m sglang.launch_server \
+    --model-path moonshotai/Kimi-K2-Instruct \
+    --tp 8 \
+    --dp 4 \
+    --trust-remote-code  \
+    --host 0.0.0.0 \
+    --port 8000
+```
+
+- Benchmark Command
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
+```
+
+- **Result**:
+
+```text Output
+Accuracy: 0.960
+Invalid: 0.000
+Latency: 15.956 s
+Output throughput: 1231.699 token/s
+```

docs_new/cookbook/autoregressive/Moonshotai/Kimi-Linear.mdx

@@ -0,0 +1,297 @@
+---
+title: Kimi-Linear
+metatags:
+    description: "Deploy Kimi-Linear with SGLang - community contribution guide for Moonshot AI's Kimi-Linear model deployment."
+---
+
+import { KimiLinearDeployment } from '/src/snippets/autoregressive/kimi-linear-deployment.jsx';
+
+## AMD GPU Support
+
+## 1. Model Introduction
+Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory.
+
+This generation delivers comprehensive upgrades across the board:
+
+Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating.
+Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention.
+Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons.
+High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT).
+
+For more details, please refer to the [official Kimi Linear GitHub Repository]: https://github.com/MoonshotAI/Kimi-Linear
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
+
+<KimiLinearDeployment />
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Launch the docker
+```shell Command
+docker pull lmsysorg/sglang:v0.5.7-rocm700-mi30x
+```
+
+```shell Command
+docker run -d -it --ipc=host --network=host --privileged \
+  --cap-add=CAP_SYS_ADMIN \
+  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
+  --group-add video --cap-add=SYS_PTRACE \
+  --security-opt seccomp=unconfined \
+  -v /:/work \
+  -e SHELL=/bin/bash \
+  --name Kimi-linear \
+  lmsysorg/sglang:v0.5.7-rocm700-mi30x \
+  /bin/bash
+```
+
+#### 4.2.2 pre-installation steps inside the docker
+
+```shell Command
+pip install sentencepiece tiktoken
+```
+
+#### 4.2.3 Launch the server
+```shell Command
+export SGLANG_ROCM_FUSED_DECODE_MLA=0
+
+SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
+  --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --tokenizer-path  moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --tp 4 \
+  --trust-remote-code
+```
+
+## 5. Benchmark
+### 5.1 Speed Benchmark
+Test Environment:
+
+Hardware: AMD MI300X GPU
+
+Model: Kimi-Linear-48B-A3B-Instruct
+
+Tensor Parallelism: 4
+
+sglang version: 0.5.7
+
+- **Model Deployment**
+
+```bash Command
+SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
+  --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --tokenizer-path  moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --tp 4 \
+  --trust-remote-code
+```
+
+### 5.1.1 Low Concurrency (Latency-Optimized)
+
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  23.86
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4001
+Request throughput (req/s):              0.42
+Input token throughput (tok/s):          255.70
+Output token throughput (tok/s):         176.86
+Peak output token throughput (tok/s):    190.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          432.56
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   2383.93
+Median E2E Latency (ms):                 1911.63
+---------------Time to First Token----------------
+Mean TTFT (ms):                          141.33
+Median TTFT (ms):                        126.27
+P99 TTFT (ms):                           294.76
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          5.32
+Median TPOT (ms):                        5.33
+P99 TPOT (ms):                           5.36
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           5.33
+Median ITL (ms):                         5.32
+P95 ITL (ms):                            5.44
+P99 ITL (ms):                            5.58
+Max ITL (ms):                            11.46
+==================================================
+```
+
+### 5.1.2 Medium Concurrency (Balanced)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  31.38
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    39667
+Request throughput (req/s):              2.55
+Input token throughput (tok/s):          1264.13
+Output token throughput (tok/s):         1300.37
+Peak output token throughput (tok/s):    1801.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          2564.50
+Concurrency:                             14.13
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5543.18
+Median E2E Latency (ms):                 5755.31
+---------------Time to First Token----------------
+Mean TTFT (ms):                          175.25
+Median TTFT (ms):                        137.87
+P99 TTFT (ms):                           292.92
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.75
+Median TPOT (ms):                        10.87
+P99 TPOT (ms):                           16.74
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           10.54
+Median ITL (ms):                         7.95
+P95 ITL (ms):                            13.68
+P99 ITL (ms):                            116.80
+Max ITL (ms):                            299.89
+==================================================
+
+```
+
+### 5.1.3 High Concurrency (Throughput-Optimized)
+- Benchmark Command:
+```bash Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100 \
+  --request-rate inf
+```
+
+- Test Results:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  79.71
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    228448
+Request throughput (req/s):              6.27
+Input token throughput (tok/s):          3134.20
+Output token throughput (tok/s):         3169.72
+Peak output token throughput (tok/s):    6109.00
+Peak concurrent requests:                110
+Total token throughput (tok/s):          6303.92
+Concurrency:                             94.80
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   15113.92
+Median E2E Latency (ms):                 13851.52
+---------------Time to First Token----------------
+Mean TTFT (ms):                          564.46
+Median TTFT (ms):                        226.04
+P99 TTFT (ms):                           2683.14
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          29.63
+Median TPOT (ms):                        31.28
+P99 TPOT (ms):                           38.84
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           28.85
+Median ITL (ms):                         16.29
+P95 ITL (ms):                            123.42
+P99 ITL (ms):                            157.80
+Max ITL (ms):                            2481.11
+==================================================
+```
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- Server Command
+
+```shell Command
+SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
+  --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --tokenizer-path  moonshotai/Kimi-Linear-48B-A3B-Instruct \
+  --tp 4 \
+  --trust-remote-code
+```
+
+- Benchmark Command
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+
+- **Result**:
+
+```text Output
+Accuracy: 0.705
+Invalid: 0.000
+Latency: 11.855 s
+Output throughput: 3224.982 token/s
+```

docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx

@@ -0,0 +1,375 @@
+---
+title: Nemotron3-Nano
+metatags:
+    description: "Deploy NVIDIA Nemotron3-Nano 30B hybrid LLM with SGLang - MoE, Mamba2, and attention layers with BF16/FP8 precision options."
+---
+
+import { Nemotron3NanoDeployment } from '/src/snippets/autoregressive/nemotron3-nano-deployment.jsx';
+
+## 1. Model Introduction
+
+`NVIDIA Nemotron3-Nano` is a 30B-parameter hybrid LLM that mixes Mixture-of-Experts (MoE) feed-forward layers, Mamba2 sequence-modeling layers, and standard self-attention layers in a single stack rather than classic “attention + MLP” transformer blocks.
+
+The BF16 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`) is designed as a high-fidelity reference model. For optimized inference performance on modern NVIDIA GPUs, the FP8 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`) and the NVFP4 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4`) are supported.
+
+At a high level:
+
+- **Hybrid layer stack (Mamba2 + MoE + attention):** The network is composed of interleaved layers that are *either* Mamba2, *or* MoE feed-forward, *or* attention-only.
+- **Non-uniform layer ordering:** The order and mix of these specialized layers is not a simple, rigid pattern, enabling the model to trade off sequence modeling, routing capacity, and expressivity across depth.
+- **Deployment-friendly precision:** Use BF16 for accuracy-sensitive and evaluation workloads; use FP8 for latency- and throughput-critical serving on recent NVIDIA GPUs.
+
+## 2. SGLang Installation
+
+Refer to the [official SGLang installation guide](../../../docs/get-started/installation), or install nightly wheel through:
+```bash Command
+uv pip install sglang==0.5.6.post3.dev1278+gad1b4e472 --extra-index-url https://sgl-project.github.io/whl/nightly/
+```
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance tuning.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.
+
+<Nemotron3NanoDeployment />
+
+### 3.2 Configuration Tips
+
+- **Attention backend**:
+
+    **H200**: Use flash attention 3 backend by default.
+    **B200**: Use flashinfer backend by default.
+
+- **TP support**:
+
+    To set tp size, use `--tp <1|2|4|8>`.
+
+- **FP8 KV cache**:
+
+    To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage (OpenAI-Compatible API)
+
+SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Summarize what MoE models are in 5 bullets."},
+    ],
+    temperature=0.7,
+    max_tokens=256,
+)
+
+print(resp.choices[0].message.content)
+
+```
+
+Streaming chat completion
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+stream = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "What are the first 5 prime numbers?"}
+    ],
+    temperature=0.7,
+    max_tokens=1024,
+    stream=True,
+)
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if delta and delta.content:
+        print(delta.content, end="", flush=True)
+```
+
+### 4.2 Reasoning
+To enable reasoning, `--reasoning-parser nemotron_3` should be appended to the launching command. The model supports two modes - Reasoning ON (default) vs OFF. This can be toggled by setting enable_thinking to False, as shown below.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+# Reasoning on (default)
+print("Reasoning on")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs."}
+    ],
+    temperature=0.7,
+    max_tokens=512,
+)
+print(resp.choices[0].message.reasoning_content)
+
+# Reasoning off
+print("Reasoning off")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs."}
+    ],
+    temperature=0.6,
+    max_tokens=256,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+print(resp.choices[0].message.reasoning_content)
+
+```
+
+### 4.3 Tool calling
+To enable reasoning, `--tool-call-parser qwen3_coder` should be appended to the launching command. Call functions using the OpenAI Tools schema and inspect returned tool_calls.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY",
+)
+
+# Tool calling via OpenAI tools schema
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "calculate_tip",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "bill_total": {
+                        "type": "integer",
+                        "description": "The total amount of the bill"
+                    },
+                    "tip_percentage": {
+                        "type": "integer",
+                        "description": "The percentage of tip to be applied"
+                    }
+                },
+                "required": ["bill_total", "tip_percentage"]
+            }
+        }
+    }
+]
+
+completion = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
+    messages=[
+        {"role": "system", "content": ""},
+        {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
+    ],
+    tools=TOOLS,
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=512,
+    stream=False
+)
+
+print(completion.choices[0].message.reasoning_content)
+print(completion.choices[0].message.tool_calls)
+```
+
+---
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU
+
+**FP8 variant**
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
+  --trust-remote-code \
+  --max-running-requests 1024 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  183.18
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total input vision tokens:               0
+Total generated tokens:                  2116125
+Total generated tokens (retokenized):    1076256
+Request throughput (req/s):              22.36
+Input token throughput (tok/s):          11364.25
+Output token throughput (tok/s):         11552.04
+Peak output token throughput (tok/s):    24692.00
+Peak concurrent requests:                294
+Total token throughput (tok/s):          22916.30
+Concurrency:                             251.19
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   11233.74
+Median E2E Latency (ms):                 11142.97
+---------------Time to First Token----------------
+Mean TTFT (ms):                          172.99
+Median TTFT (ms):                        116.57
+P99 TTFT (ms):                           1193.68
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          21.74
+Median TPOT (ms):                        21.14
+P99 TPOT (ms):                           41.12
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           21.45
+Median ITL (ms):                         9.06
+P95 ITL (ms):                            62.59
+P99 ITL (ms):                            110.83
+Max ITL (ms):                            5368.19
+==================================================
+```
+
+**BF16 variant**
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
+  --trust-remote-code \
+  --max-running-requests 1024 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  360.22
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total input vision tokens:               0
+Total generated tokens:                  2087288
+Total generated tokens (retokenized):    1940652
+Request throughput (req/s):              11.37
+Input token throughput (tok/s):          5779.10
+Output token throughput (tok/s):         5794.55
+Peak output token throughput (tok/s):    9169.00
+Peak concurrent requests:                276
+Total token throughput (tok/s):          11573.65
+Concurrency:                             249.76
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   21965.10
+Median E2E Latency (ms):                 21706.35
+---------------Time to First Token----------------
+Mean TTFT (ms):                          211.54
+Median TTFT (ms):                        93.06
+P99 TTFT (ms):                           2637.66
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          43.27
+Median TPOT (ms):                        43.04
+P99 TPOT (ms):                           61.15
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           42.77
+Median ITL (ms):                         28.46
+P95 ITL (ms):                            71.85
+P99 ITL (ms):                            113.20
+Max ITL (ms):                            5237.28
+==================================================
+
+```
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+**Environment**
+- Hardware: NVIDIA B200 GPU
+- Model: BF16 checkpoint
+
+**Launch Model**
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
+  --trust-remote-code \
+  --reasoning-parser nemotron_3
+```
+
+**Run Benchmark with lm-eval**
+```bash Command
+pip install lm-eval[api]==0.4.9.2
+
+lm_eval --model local-completions --tasks gsm8k --model_args "model=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=4,max_retries=3,tokenized_requests=False,max_lengths=16384" --gen_kwargs '{"chat_template_kwargs":{"thinking":true}}' --batch_size 256
+```
+
+**Test Results:**
+```text Output
+|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
+|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
+|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.5603|±  |0.0137|
+|     |       |strict-match    |     5|exact_match|↑  |0.8453|±  |0.0100|
+```

docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Super.mdx

@@ -0,0 +1,571 @@
+---
+title: NVIDIA Nemotron3-Super
+metatags:
+    description: "Deploy NVIDIA Nemotron3-Super with SGLang - 120B hybrid MoE model (12B active) with 1M context window optimized for multi-agent systems and tool use."
+---
+
+import { Nemotron3SuperDeployment } from '/src/snippets/autoregressive/nemotron3-super-deployment.jsx';
+
+## 1. Model Introduction
+
+`NVIDIA Nemotron3-Super` is a leading open model in the Nemotron 3 family, built for running many collaborating agents together. It is optimized for agentic systems that chain planning, reasoning, and tool use workloads that generate far more tokens than single turn chat and require strong reasoning at every step.
+
+Nemotron 3 Super is a 120B parameter hybrid MoE model that activates only 12B parameters per forward pass, delivering strong accuracy for coding, tool calling, and instruction following at a fraction of the cost. It also supports a 1M token context window so agents can keep conversation history and plan state in view across long workflows.
+
+Architecture and key features:
+
+- **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
+- **Highest throughput efficiency in its size category:** Delivers up to 5x higher throughput compared to the previous Nemotron Super model (Llama Nemotron Super 1.5).
+- **Multi-Token Prediction (MTP):** By predicting several future tokens simultaneously in a single forward pass, MTP drastically accelerates the generation of long-form text.
+- **Thinking Budget support:** Supports Thinking Budget for optimal accuracy with minimum reasoning token generation.
+
+## 2. SGLang Installation
+
+SGLang from the main branch is required for Nemotron3-Super. You can install from source and with a nightly docker.
+
+```bash Command
+# Install from source
+uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
+
+# Or use Docker
+docker pull lmsysorg/sglang:nightly-dev-20260310-0fd9a57d
+```
+
+For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance tuning.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: select hardware, tensor parallelism, and common knobs to generate a launch command.
+
+<Nemotron3SuperDeployment />
+
+### 3.2 Configuration Tips
+
+- **Attention backend**:
+
+    **H200**: Use flash attention 3 backend by default.
+    **B200**: Use flashinfer backend by default.
+
+- **TP support**:
+
+    To set tp size, use `--tp <2|4|8>`.
+
+- **FP8 KV cache**:
+
+    To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.
+
+## 4. Model Invocation
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --host 0.0.0.0 \
+  --port 5000 \
+  --trust-remote-code \
+  --tp 4 \
+  --tool-call-parser qwen3_coder \
+  --reasoning-parser nemotron_3
+```
+
+### 4.1 Basic Usage (OpenAI-Compatible API)
+
+SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
+    ],
+    temperature=0.6,
+    max_tokens=1024,
+)
+print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
+print("\n")
+```
+
+Output:
+```text Output
+Reasoning: Okay, the user is asking for 3 bullet points about SGLang. Let me recall what I know about SGLang. It's a framework for serving large language models, right? Developed by the team at UC Berkeley and others.
+
+First, I should verify the key features. SGLang is known for its high-performance serving capabilities, especially with features like Radix Attention and chunked prefill. Those are important points to mention...(more tokens)
+
+Content: - SGLang introduces **Radix Attention**, an innovative attention mechanism that significantly reduces KV cache memory usage and improves computational efficiency during LLM serving by reusing intermediate states across tokens.
+- It features **chunked prefill** for handling long prompts efficiently, breaking input sequences into manageable chunks to minimize latency and memory pressure while maintaining high throughput.
+- Designed for **high-performance LLM serving**, SGLang achieves superior throughput and lower latency compared to traditional systems (like vLLM or TensorRT-LLM) through optimized kernel fusion, dynamic batching, and seamless integration with Hugging Face Transformers.
+```
+
+Streaming chat completion:
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+stream = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful AI assistant."},
+        {"role": "user", "content": "What are the first 5 prime numbers?"}
+    ],
+    temperature=0.7,
+    max_tokens=1024,
+    stream=True,
+)
+for chunk in stream:
+    delta = chunk.choices[0].delta
+    if delta and delta.content:
+        print(delta.content, end="", flush=True)
+```
+
+Output:
+```text Output
+The first 5 prime numbers are:
+**2, 3, 5, 7, 11**.
+
+### Explanation:
+- A **prime number** is a natural number greater than 1 that has no positive divisors other than 1 and itself.
+- **2** is the smallest and only even prime number.
+- **3** is prime (divisible only by 1 and 3).
+- **4** is not prime (divisible by 2).
+- **5** is prime.
+- **6** is not prime (divisible by 2 and 3).
+- **7** is prime.
+- **8, 9, 10** are not prime.
+- **11** is prime (the fifth in the sequence).
+
+Note: **1 is not considered a prime number** by definition, as it has only one positive divisor.
+This list is universally accepted in mathematics. Let me know if you'd like to explore more primes or related concepts! 😊
+```
+
+### 4.2 Reasoning
+
+The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+# Reasoning on (default)
+print("Reasoning on")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs. Please make thinking process short."}
+    ],
+    temperature=1,
+    max_tokens=1024,
+)
+print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
+print("\n")
+# Reasoning off
+print("Reasoning off")
+resp = client.chat.completions.create(
+    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Give me 3 facts about SGLang."}
+    ],
+    temperature=0,
+    max_tokens=256,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+print(f"Content: \n{resp.choices[0].message.reasoning_content[:200]}...")
+```
+
+Output:
+```text Output
+Reasoning on
+Reasoning:
+We need to output a haiku about GPUs, with short thinking process. Probably we just need to produce the haiku. No extra commentary needed. Provide a haiku: 5-7-5 syllable lines about GPUs.
+
+Let's deci...
+Content:
+Silicon hearts beat
+Paint vivid worlds with bright light
+GPU dreams rise...
+
+Reasoning off
+Content:
+Certainly! Here are three accurate and informative facts about **SGLang**:
+
+1. **SGLang is a high-performance serving system for large language models (LLMs)**
+   Developed by researchers at UC Berk...
+```
+
+### 4.3 Tool Calling
+
+Call functions using the OpenAI Tools schema and inspect returned `tool_calls`.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:5000/v1",
+    api_key="EMPTY",
+)
+
+# Tool calling via OpenAI tools schema
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "calculate_tip",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "bill_total": {
+                        "type": "integer",
+                        "description": "The total amount of the bill"
+                    },
+                    "tip_percentage": {
+                        "type": "integer",
+                        "description": "The percentage of tip to be applied"
+                    }
+                },
+                "required": ["bill_total", "tip_percentage"]
+            }
+        }
+    }
+]
+
+completion = client.chat.completions.create(
+    model="nemotron",
+    messages=[
+        {"role": "system", "content": ""},
+        {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
+    ],
+    tools=TOOLS,
+    temperature=0.6,
+    top_p=0.95,
+    max_tokens=512,
+    stream=False
+)
+
+print(completion.choices[0].message.reasoning_content)
+print(completion.choices[0].message.tool_calls)
+```
+
+Output:
+```text Output
+The user wants to calculate a 15% tip on a $50 bill. I have a function called calculate_tip that takes bill_total and tip_percentage as parameters. The bill_total is $50, and tip_percentage is 15. I need to call the function with these values. Let me do that.
+
+[ChatCompletionMessageFunctionToolCall(id='call_ced9a83a3baa448e9d587aaf', function=Function(arguments='{"bill_total": 50, "tip_percentage": 15}', name='calculate_tip'), type='function', index=0)]
+```
+
+### 4.4 Controlling Reasoning Budget
+
+The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.
+
+If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.
+
+```python Example
+from typing import Any, Dict, List
+import openai
+from transformers import AutoTokenizer
+
+class ThinkingBudgetClient:
+    def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
+        self.base_url = base_url
+        self.api_key = api_key
+        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
+        self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)
+
+    def chat_completion(
+        self,
+        model: str,
+        messages: List[Dict[str, Any]],
+        reasoning_budget: int = 512,
+        max_tokens: int = 1024,
+        **kwargs,
+    ) -> Dict[str, Any]:
+        assert (
+            max_tokens > reasoning_budget
+        ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"
+
+        # 1. first call chat completion to get reasoning content
+        response = self.client.chat.completions.create(
+            model=model,
+            messages=messages,
+            max_tokens=reasoning_budget,
+            **kwargs
+        )
+
+        reasoning_content = response.choices[0].message.reasoning_content or ""
+
+        if "</think>" not in reasoning_content:
+            # reasoning content is too long, closed with a period (.)
+            reasoning_content = f"{reasoning_content}.\n</think>\n\n"
+
+        reasoning_tokens_used = len(
+            self.tokenizer.encode(reasoning_content, add_special_tokens=False)
+        )
+        remaining_tokens = max_tokens - reasoning_tokens_used
+
+        assert (
+            remaining_tokens > 0
+        ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."
+
+        # 2. append reasoning content to messages and call completion
+        messages.append({"role": "assistant", "content": reasoning_content})
+        prompt = self.tokenizer.apply_chat_template(
+            messages,
+            tokenize=False,
+            continue_final_message=True,
+        )
+
+        response = self.client.completions.create(
+            model=model,
+            prompt=prompt,
+            max_tokens=remaining_tokens,
+            **kwargs
+        )
+
+        response_data = {
+            "reasoning_content": reasoning_content.strip().strip("</think>").strip(),
+            "content": response.choices[0].text,
+            "finish_reason": response.choices[0].finish_reason,
+        }
+        return response_data
+```
+
+Usage example with `reasoning_budget=128`:
+
+```python Example
+SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"
+
+# Client
+client = ThinkingBudgetClient(
+    base_url="http://127.0.0.1:5000/v1",
+    api_key="null",
+    tokenizer_name_or_path=SERVED_MODEL_NAME
+)
+
+resp = client.chat_completion(
+    model=SERVED_MODEL_NAME,
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Write a haiku about GPUs."}
+    ],
+    temperature=1,
+    max_tokens=512,
+    reasoning_budget=128
+)
+print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
+```
+
+Output:
+```text Output
+Reasoning: Okay, the user wants a haiku about GPUs. Let me recall what a haiku is: a traditional Japanese poem with three lines, 5-7-5 syllable structure. So I need to make sure the syllable count is exact.
+
+First, I should think about what makes GPUs interesting. They're used for graphics rendering, parallel processing, AI, gaming, etc. Maybe focus on their speed, power, or how they handle many tasks at once.
+
+Let me brainstorm some words and phrases related to GPUs: silicon, cores, transistors, parallel, rendering, pixels, frames per second, CUDA, tensor.
+Content:
+
+Silicon minds awaken,
+Thousands of cores hum in unison—
+Lightning paints the void.
+```
+
+---
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+- Hardware: H200 (4x)
+- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+- Model Deployment Command:
+
+```shell Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --trust-remote-code \
+  --tp 4 \
+  --max-running-requests 1024 \
+  --host 0.0.0.0 \
+  --port 5000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 5000 \
+  --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --dataset-name random \
+  --random-input-len 1024 \
+  --random-output-len 1024 \
+  --num-prompts 4096 \
+  --max-concurrency 256
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 256
+Successful requests:                     4096
+Benchmark duration (s):                  623.49
+Total input tokens:                      2081726
+Total input text tokens:                 2081726
+Total generated tokens:                  2087288
+Total generated tokens (retokenized):    2044666
+Request throughput (req/s):              6.57
+Input token throughput (tok/s):          3338.85
+Output token throughput (tok/s):         3347.77
+Peak output token throughput (tok/s):    6349.00
+Peak concurrent requests:                270
+Total token throughput (tok/s):          6686.62
+Concurrency:                             250.35
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   38108.46
+Median E2E Latency (ms):                 37186.80
+P90 E2E Latency (ms):                    69325.24
+P99 E2E Latency (ms):                    77776.90
+---------------Time to First Token----------------
+Mean TTFT (ms):                          436.49
+Median TTFT (ms):                        114.90
+P99 TTFT (ms):                           6938.11
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          75.02
+Median TPOT (ms):                        76.02
+P99 TPOT (ms):                           92.27
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           74.07
+Median ITL (ms):                         38.45
+P95 ITL (ms):                            230.42
+P99 ITL (ms):                            242.70
+Max ITL (ms):                            7181.72
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+**Environment**
+- Hardware: H200 (4x)
+- Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
+- Tensor Parallelism: 4
+- SGLang Version: main branch
+
+**Launch Model**
+```bash Command
+python3 -m sglang.launch_server \
+  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
+  --trust-remote-code \
+  --tp 4 \
+  --reasoning-parser nemotron_3
+```
+
+**Run Benchmark**
+```bash Command
+python3 benchmark/gsm8k/bench_sglang.py --port 5000
+```
+
+**Test Results:**
+```text Output
+Accuracy: 0.950
+Invalid: 0.000
+Latency: 21.442 s
+Output throughput: 996.815 token/s
+```
+
+#### 5.2.2 MMLU Benchmark
+
+**Run Benchmark**
+```bash Command
+python3 benchmark/mmlu/bench_sglang.py --port 5000
+```
+
+**Test Results:**
+```text Output
+subject: abstract_algebra, #q:100, acc: 0.730
+subject: anatomy, #q:135, acc: 0.830
+subject: astronomy, #q:152, acc: 0.934
+subject: business_ethics, #q:100, acc: 0.830
+subject: clinical_knowledge, #q:265, acc: 0.879
+subject: college_biology, #q:144, acc: 0.931
+subject: college_chemistry, #q:100, acc: 0.620
+subject: college_computer_science, #q:100, acc: 0.840
+subject: college_mathematics, #q:100, acc: 0.820
+subject: college_medicine, #q:173, acc: 0.821
+subject: college_physics, #q:102, acc: 0.794
+subject: computer_security, #q:100, acc: 0.880
+subject: conceptual_physics, #q:235, acc: 0.919
+subject: econometrics, #q:114, acc: 0.746
+subject: electrical_engineering, #q:145, acc: 0.828
+subject: elementary_mathematics, #q:378, acc: 0.926
+subject: formal_logic, #q:126, acc: 0.857
+subject: global_facts, #q:100, acc: 0.570
+subject: high_school_biology, #q:310, acc: 0.952
+subject: high_school_chemistry, #q:203, acc: 0.828
+subject: high_school_computer_science, #q:100, acc: 0.940
+subject: high_school_european_history, #q:165, acc: 0.861
+subject: high_school_geography, #q:198, acc: 0.939
+subject: high_school_government_and_politics, #q:193, acc: 0.990
+subject: high_school_macroeconomics, #q:390, acc: 0.928
+subject: high_school_mathematics, #q:270, acc: 0.700
+subject: high_school_microeconomics, #q:238, acc: 0.966
+subject: high_school_physics, #q:151, acc: 0.834
+subject: high_school_psychology, #q:545, acc: 0.960
+subject: high_school_statistics, #q:216, acc: 0.852
+subject: high_school_us_history, #q:204, acc: 0.926
+subject: high_school_world_history, #q:237, acc: 0.937
+subject: human_aging, #q:223, acc: 0.879
+subject: human_sexuality, #q:131, acc: 0.939
+subject: international_law, #q:121, acc: 0.934
+subject: jurisprudence, #q:108, acc: 0.898
+subject: logical_fallacies, #q:163, acc: 0.914
+subject: machine_learning, #q:112, acc: 0.821
+subject: management, #q:103, acc: 0.903
+subject: marketing, #q:234, acc: 0.944
+subject: medical_genetics, #q:100, acc: 0.980
+subject: miscellaneous, #q:783, acc: 0.945
+subject: moral_disputes, #q:346, acc: 0.861
+subject: moral_scenarios, #q:895, acc: 0.542
+subject: nutrition, #q:306, acc: 0.902
+subject: philosophy, #q:311, acc: 0.884
+subject: prehistory, #q:324, acc: 0.920
+subject: professional_accounting, #q:282, acc: 0.805
+subject: professional_law, #q:1534, acc: 0.681
+subject: professional_medicine, #q:272, acc: 0.923
+subject: professional_psychology, #q:612, acc: 0.889
+subject: public_relations, #q:110, acc: 0.800
+subject: security_studies, #q:245, acc: 0.837
+subject: sociology, #q:201, acc: 0.960
+subject: us_foreign_policy, #q:100, acc: 0.920
+subject: virology, #q:166, acc: 0.590
+subject: world_religions, #q:171, acc: 0.906
+Total latency: 150.267
+Average accuracy: 0.841
+```

docs_new/cookbook/autoregressive/OpenAI/GPT-OSS.mdx

@@ -0,0 +1,572 @@
+---
+title: GPT-OSS
+metatags:
+    description: "Deploy GPT-OSS (20B/120B) with SGLang - configurable reasoning, full chain-of-thought, MXFP4 quantization for single GPU deployment."
+---
+
+## 1.Model Introduction
+
+[GPT-OSS](https://huggingface.co/openai/gpt-oss-20b) is an advanced large language model developed by OpenAI designed for power reasoning, agentic tasks, and versatile developer use cases. It has versions with two model sizes.
+
+- **gpt-oss-120b** — for production, general purpose, high reasoning use cases that fit into a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) (117B parameters with 5.1B active parameters)
+- **gpt-oss-20b** — for lower latency, and local or specialized use cases (21B parameters with 3.6B active parameters)
+
+GPT-OSS introduces several groundbreaking innovations:
+
+- **Configurable reasoning effort**: Easily adjust the reasoning effort (low, medium, high) based on your specific use case and latency needs.
+- **Full chain-of-thought**: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs. It’s not intended to be shown to end users.
+- **Fine-tunable**: Fully customize models to your specific use case through parameter fine-tuning.
+- **Agentic capabilities**: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs.
+- **MXFP4 quantization**: The models were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) and the gpt-oss-20b model run within 16GB of memory. All evals were performed with the same MXFP4 quantization.
+
+## 2.SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3.Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+The GPT-OSS series comes in two sizes. Recommended starting configurations vary depending on hardware.
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
+
+import { GPTOSSDeployment } from "/src/snippets/autoregressive/gpt-oss-deployment.jsx";
+
+<GPTOSSDeployment />
+
+### 3.2 Configuration Tips
+
+For more detailed configuration tips, please refer to [GPS-OSS Usage](../../../docs/basic_usage/gpt_oss).
+
+## 4.Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+GPT-OSS supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model openai/gpt-oss-120b \
+  --reasoning-parser gpt-oss \
+  --tp 8
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="openai/gpt-oss-120b",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user asks: "Solve this problem step by step: What is 15% of 240?" So we need to provide step-by-step solution. Compute 15% of 240: 0.15 * 240 = 36. Provide steps: convert percent to decimal, multiply, maybe use fraction. Provide answer.
+=============== Content =================
+**Step‑by‑step solution**
+
+1. **Understand what “percent” means**
+   “15 %” means 15 out of every 100 parts, i.e. the fraction \(\displaystyle \frac{15}{100}\).
+
+2. **Convert the percent to a decimal (or fraction)**
+   \[
+   \frac{15}{100}=0.15
+   \]
+
+3. **Set up the multiplication**
+   To find 15 % of 240 we multiply 240 by the decimal 0.15:
+   \[
+   240 \times 0.15
+   \]
+
+4. **Do the multiplication**
+   One convenient way is to break it into two easier parts:
+   \[
+   240 \times 0.15 = 240 \times \left(\frac{15}{100}\right)
+                = \frac{240 \times 15}{100}
+   \]
+
+   - First compute \(240 \times 15\):
+     \[
+     240 \times 15 = 240 \times (10 + 5) = 2400 + 1200 = 3600
+     \]
+
+   - Then divide by 100:
+     \[
+     \frac{3600}{100} = 36
+     \]
+
+5. **Write the result**
+   \[
+   15\% \text{ of } 240 = 36
+   \]
+
+---
+
+**Answer:** \(36\)
+```
+
+#### 4.2.2 Tool Calling
+
+GPT-OSS supports tool calling capabilities. Enable the tool call parser:
+
+**Python Example (without Thinking Process):**
+
+Start sglang server:
+
+```shell Command
+python -m sglang.launch_server \
+  --model openai/gpt-oss-120b \
+  --tool-call-parser gpt-oss \
+  --tp 8
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="openai/gpt-oss-120b",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"🔧 Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Python Example (with Thinking Process):**
+
+Start sglang server:
+
+```shell Command
+python -m sglang.launch_server \
+  --model openai/gpt-oss-120b \
+  --reasoning-parser gpt-oss \
+  --tool-call-parser gpt-oss \
+  --tp 8
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="openai/gpt-oss-120b",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"🔧 Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+User asks: "What's the weather in Beijing?" We need to get current weather. Use function get_weather with location "Beijing". No unit specified; default? Probably use default (maybe Celsius). We can specify unit as "celsius". We'll call function.
+=============== Content =================
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="openai/gpt-oss-120b",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The current weather in Beijing is 22 °C and sunny. Let me know if you’d like a forecast for the next few days or any other details!"
+```
+
+## 5.Benchmark
+
+### 5.1 Speed Benchmark
+
+- Hardware: NVIDIA B200 GPU (8x)
+- Tensor Parallelism: 8
+- Model: openai/gpt-oss-120b
+- sglang version: 0.5.6
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Server Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model openai/gpt-oss-120b \
+  --tp 8
+```
+
+- Test Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --num-prompt 100 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     100
+Benchmark duration (s):                  52.35
+Total input tokens:                      33178
+Total input text tokens:                 33178
+Total input vision tokens:               0
+Total generated tokens:                  21251
+Total generated tokens (retokenized):    20868
+Request throughput (req/s):              1.91
+Input token throughput (tok/s):          633.76
+Output token throughput (tok/s):         405.93
+Peak output token throughput (tok/s):    433.00
+Peak concurrent requests:                8
+Total token throughput (tok/s):          1039.69
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   523.30
+Median E2E Latency (ms):                 389.91
+---------------Time to First Token----------------
+Mean TTFT (ms):                          33.71
+Median TTFT (ms):                        31.79
+P99 TTFT (ms):                           108.98
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          2.31
+Median TPOT (ms):                        2.31
+P99 TPOT (ms):                           2.39
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           2.31
+Median ITL (ms):                         2.31
+P95 ITL (ms):                            2.35
+P99 ITL (ms):                            2.38
+Max ITL (ms):                            3.54
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Server Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model openai/gpt-oss-120b \
+  --tp 8
+```
+
+- Test Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --num-prompt 1000 \
+  --max-concurrency 100
+```
+
+**Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  24.76
+Total input tokens:                      297156
+Total input text tokens:                 297156
+Total input vision tokens:               0
+Total generated tokens:                  192432
+Total generated tokens (retokenized):    187145
+Request throughput (req/s):              40.39
+Input token throughput (tok/s):          12003.57
+Output token throughput (tok/s):         7773.26
+Peak output token throughput (tok/s):    13780.00
+Peak concurrent requests:                156
+Total token throughput (tok/s):          19776.83
+Concurrency:                             89.23
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   2208.97
+Median E2E Latency (ms):                 1591.11
+---------------Time to First Token----------------
+Mean TTFT (ms):                          102.94
+Median TTFT (ms):                        31.53
+P99 TTFT (ms):                           674.32
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          14.31
+Median TPOT (ms):                        11.00
+P99 TPOT (ms):                           91.28
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.00
+Median ITL (ms):                         5.75
+P95 ITL (ms):                            25.35
+P99 ITL (ms):                            43.18
+Max ITL (ms):                            621.42
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
+```
+
+- **Results**:
+
+  - GPT-OSS-120b
+
+    ```text Output
+    Accuracy: 0.880
+    Invalid: 0.005
+    Latency: 5.262 s
+    Output throughput: 12143.675 token/s
+    ```
+
+  - GPT-OSS-20b
+
+    ```text Output
+    Accuracy: 0.535
+    Invalid: 0.165
+    Latency: 4.157 s
+    Output throughput: 19589.165 token/s
+    ```

docs_new/cookbook/autoregressive/Qwen/Qwen2.5-VL.mdx

@@ -0,0 +1,392 @@
+---
+title: Qwen2.5-VL
+metatags:
+    description: "Deploy Qwen2.5-VL vision-language models with SGLang on AMD MI300X - available in 3B to 72B sizes with enhanced visual understanding."
+---
+
+import { Qwen25VLDeployment } from '/src/snippets/autoregressive/qwen25-vl-deployment.jsx';
+
+## 1. Model Introduction
+
+**[Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl)** is a vision-language model series from the Qwen team, offering significant improvements over its predecessor in understanding, reasoning, and multi-modal processing.
+
+**Key Features:**
+
+- **Understand things visually**: Proficient in recognizing common objects such as flowers, birds, fish, and insects, and it is highly capable of analyzing texts, charts, icons, graphics, and layouts within images.
+- **More Agentic**: Play as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use.
+- **Understanding long videos and capturing events**: Supports comprehending videos of over 1 hour, and this time it has a new ability of capturing event by pinpointing the relevant video segments.
+- **Capable of visual localization in different formats**: Accurately localize objects in an image by generating bounding boxes or points, and it can provide stable JSON outputs for coordinates and attributes.
+- **Generating structured outputs**: Supports structured outputs of the contents, benefiting usages in finance, commerce, etc for data like scans of invoices, forms, tables, etc.
+- **Dynamic Resolution and Frame Rate Training for Video Understanding**: Extend dynamic resolution to the temporal dimension by adopting dynamic FPS sampling, enabling the model to comprehend videos at various sampling rates. Accordingly, we update mRoPE in the time dimension with IDs and absolute time alignment, enabling the model to learn temporal sequence and speed, and ultimately acquire the ability to pinpoint specific moments.
+- **Multiple Sizes**: Available in 3B, 7B, 32B, and 72B variants to suit different deployment needs.
+- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
+
+For more details, please refer to the [official Qwen2.5-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for AMD MI300X, MI325X and MI355X hardware platforms and different use cases.
+
+### 3.1 Basic Configuration
+
+The Qwen2.5-VL series offers models in various sizes. The following configurations have been verified on AMD MI300X, MI325X and MI355X GPUs.
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model size.
+
+<Qwen25VLDeployment />
+
+### 3.2 Configuration Tips
+
+* **Memory Management**: For the 72B model on MI300X/MI325X/MI355X, we have verified successful deployment with `--context-length 128000`. Smaller context lengths can be used to reduce memory usage if needed.
+* **Multi-GPU Deployment**: Use Tensor Parallelism (`--tp`) to scale across multiple GPUs. For example, use `--tp 8` for the 72B model and `--tp 2` for the 32B model on MI300X/MI325X/MI355X.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Multi-Modal Inputs
+
+Qwen2.5-VL supports image inputs. Here's a basic example with single image input:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Read all the text in the image."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="Qwen/Qwen2.5-VL-7B-Instruct",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 2.31s
+Generated text: Auntie Anne's
+
+CINNAMON SUGAR
+1 x 17,000
+SUB TOTAL
+17,000
+
+GRAND TOTAL
+17,000
+
+CASH IDR
+20,000
+
+CHANGE DUE
+3,000
+```
+
+**Multi-Image Input Example:**
+
+Qwen2.5-VL can process multiple images in a single request for comparison or analysis:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
+                }
+            },
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Compare these two images and describe the differences in 100 words or less."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="Qwen/Qwen2.5-VL-7B-Instruct",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 13.79s
+Generated text: The first image shows a single red taxi driving on a street with a few other taxis in the background. The second image shows a large number of taxis parked in a lot, with some appearing to be in various states of repair. The first image has a single taxi with a visible license plate, while the second image has multiple taxis with different license plates. The first image has a clear view of the street and surrounding area, while the second image is taken from an elevated perspective, showing a wider view of the parking lot and the surrounding area.
+```
+
+**Note:**
+
+- You can also provide local file paths using `file://` protocol.
+- For larger images, you may need more memory, adjust `--mem-fraction-static` accordingly.
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X GPU (8x)
+- Model: Qwen2.5-VL-72B-Instruct
+- Tensor Parallelism: 8
+- SGLang Version: 0.5.6
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen2.5-VL-72B-Instruct \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen2.5-VL-72B-Instruct \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen2.5-VL-72B-Instruct \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  37.99
+Total input tokens:                      24781
+Total input text tokens:                 821
+Total input vision tokens:               23960
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    2365
+Request throughput (req/s):              0.26
+Input token throughput (tok/s):          652.26
+Output token throughput (tok/s):         111.07
+Peak output token throughput (tok/s):    128.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          763.34
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3797.61
+Median E2E Latency (ms):                 3140.90
+P90 E2E Latency (ms):                    6545.54
+P99 E2E Latency (ms):                    7939.56
+---------------Time to First Token----------------
+Mean TTFT (ms):                          504.45
+Median TTFT (ms):                        510.93
+P99 TTFT (ms):                           521.78
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          7.82
+Median TPOT (ms):                        7.82
+P99 TPOT (ms):                           7.84
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           10.07
+Median ITL (ms):                         7.90
+P95 ITL (ms):                            15.79
+P99 ITL (ms):                            15.93
+Max ITL (ms):                            23.60
+==================================================
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen2.5-VL-72B-Instruct \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  454.68
+Total input tokens:                      2481865
+Total input text tokens:                 85865
+Total input vision tokens:               2396000
+Total generated tokens:                  510855
+Total generated tokens (retokenized):    296466
+Request throughput (req/s):              2.20
+Input token throughput (tok/s):          5458.50
+Output token throughput (tok/s):         1123.55
+Peak output token throughput (tok/s):    5004.00
+Peak concurrent requests:                106
+Total token throughput (tok/s):          6582.05
+Concurrency:                             98.63
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   44844.92
+Median E2E Latency (ms):                 42866.15
+P90 E2E Latency (ms):                    82798.20
+P99 E2E Latency (ms):                    106306.30
+---------------Time to First Token----------------
+Mean TTFT (ms):                          4507.79
+Median TTFT (ms):                        1180.83
+P99 TTFT (ms):                           39975.22
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          80.26
+Median TPOT (ms):                        82.38
+P99 TPOT (ms):                           152.89
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           100.66
+Median ITL (ms):                         13.26
+P95 ITL (ms):                            428.45
+P99 ITL (ms):                            1393.35
+Max ITL (ms):                            31943.26
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 MMMU Benchmark
+
+You can evaluate the model's accuracy using the MMMU dataset:
+
+- Benchmark Command:
+
+```shell Command
+python3 benchmark/mmmu/bench_sglang.py \
+    --port 30000 \
+    --concurrency 64
+```
+```text Output
+Benchmark time: 97.75084622902796
+answers saved to: ./answer_sglang.json
+Evaluating...
+answers saved to: ./answer_sglang.json
+{'Accounting': {'acc': 0.633, 'num': 30},
+ 'Agriculture': {'acc': 0.5, 'num': 30},
+ 'Architecture_and_Engineering': {'acc': 0.367, 'num': 30},
+ 'Art': {'acc': 0.767, 'num': 30},
+ 'Art_Theory': {'acc': 0.9, 'num': 30},
+ 'Basic_Medical_Science': {'acc': 0.7, 'num': 30},
+ 'Biology': {'acc': 0.467, 'num': 30},
+ 'Chemistry': {'acc': 0.433, 'num': 30},
+ 'Clinical_Medicine': {'acc': 0.733, 'num': 30},
+ 'Computer_Science': {'acc': 0.567, 'num': 30},
+ 'Design': {'acc': 0.833, 'num': 30},
+ 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30},
+ 'Economics': {'acc': 0.767, 'num': 30},
+ 'Electronics': {'acc': 0.433, 'num': 30},
+ 'Energy_and_Power': {'acc': 0.467, 'num': 30},
+ 'Finance': {'acc': 0.533, 'num': 30},
+ 'Geography': {'acc': 0.633, 'num': 30},
+ 'History': {'acc': 0.7, 'num': 30},
+ 'Literature': {'acc': 0.867, 'num': 30},
+ 'Manage': {'acc': 0.633, 'num': 30},
+ 'Marketing': {'acc': 0.733, 'num': 30},
+ 'Materials': {'acc': 0.333, 'num': 30},
+ 'Math': {'acc': 0.533, 'num': 30},
+ 'Mechanical_Engineering': {'acc': 0.433, 'num': 30},
+ 'Music': {'acc': 0.367, 'num': 30},
+ 'Overall': {'acc': 0.62, 'num': 900},
+ 'Overall-Art and Design': {'acc': 0.717, 'num': 120},
+ 'Overall-Business': {'acc': 0.66, 'num': 150},
+ 'Overall-Health and Medicine': {'acc': 0.693, 'num': 150},
+ 'Overall-Humanities and Social Science': {'acc': 0.775, 'num': 120},
+ 'Overall-Science': {'acc': 0.553, 'num': 150},
+ 'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210},
+ 'Pharmacy': {'acc': 0.833, 'num': 30},
+ 'Physics': {'acc': 0.7, 'num': 30},
+ 'Psychology': {'acc': 0.767, 'num': 30},
+ 'Public_Health': {'acc': 0.733, 'num': 30},
+ 'Sociology': {'acc': 0.767, 'num': 30}}
+eval out saved to ./val_sglang.json
+Overall accuracy: 0.62
+```

docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx

@@ -0,0 +1,902 @@
+---
+title: Qwen3-Coder-Next
+metatags:
+    description: "Deploy Qwen3-Coder-Next code-focused models with SGLang on AMD MI300X - available in 3B to 80B sizes with enhanced code understanding."
+---
+
+import { Qwen3CoderNextDeployment } from '/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx';
+
+## 1. Model Introduction
+
+[Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) is a cost-efficient code-focused language model from the Qwen team (Alibaba). With 80B total parameters but only 3B activated parameters, it achieves performance comparable to models with 10–20x more active parameters through its innovative hybrid architecture.
+
+**Key Features:**
+
+- **Hybrid Architecture**: Uses a 48-layer hybrid layout combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared), enabling exceptional efficiency.
+- **Tool Calling Support**: Advanced agentic capabilities with native support for function calling and tool use via the `qwen3_coder` parser.
+- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
+- **Cost-Efficient Inference**: Only 3B parameters activated per token, making it ideal for local development and cost-effective deployment at scale.
+- **IDE Integration**: Compatible with Claude Code, Qwen Code, Cline, and other IDE platforms.
+
+For more details, please refer to the [Qwen3-Coder-Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+**Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later.
+
+## 3. Model Deployment
+
+This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options.
+
+<Qwen3CoderNextDeployment />
+
+### 3.2 Configuration Tips
+
+- **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`.
+- **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag.
+- **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed.
+- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
+  - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
+  - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
+
+## 4. Model Invocation
+
+**Deployment Command:**
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Coder-Next \
+  --tp 2 \
+  --tool-call-parser qwen3_coder \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Code Generation Example
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Coder-Next",
+    messages=[
+        {"role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include type hints."}
+    ],
+    max_tokens=2048
+)
+
+print(response.choices[0].message.content)
+```
+
+**Example Output:**
+
+````text Output
+Here's a Python function implementing binary search on a sorted list, with comprehensive type hints:
+
+```python
+from typing import Sequence, TypeVar, Optional
+
+T = TypeVar('T')
+
+def binary_search(sorted_list: Sequence[T], target: T) -> Optional[int]:
+    """
+    Perform binary search on a sorted list to find the index of a target element.
+
+    Args:
+        sorted_list: A sequence (e.g., list, tuple) sorted in ascending order.
+        target: The element to search for in the list.
+
+    Returns:
+        The index of the target element if found, or None if not found.
+
+    Time Complexity: O(log n)
+    Space Complexity: O(1)
+
+    Note:
+        The function assumes the list is sorted in ascending order.
+        If the list contains duplicate elements, it returns the index of one of them.
+    """
+    left = 0
+    right = len(sorted_list) - 1
+
+    while left <= right:
+        mid = (left + right) // 2
+        mid_val = sorted_list[mid]
+
+        if mid_val == target:
+            return mid
+        elif mid_val < target:
+            left = mid + 1
+        else:
+            right = mid - 1
+
+    return None
+```
+
+### Example usage:
+
+```python
+# Example 1: Finding an existing element
+numbers = [1, 3, 5, 7, 9, 11]
+print(binary_search(numbers, 7))  # Output: 3
+
+# Example 2: Element not in the list
+print(binary_search(numbers, 4))  # Output: None
+
+# Example 3: Empty list
+print(binary_search([], 5))  # Output: None
+
+# Example 4: Single element
+print(binary_search([1], 1))  # Output: 0
+print(binary_search([1], 2))  # Output: None
+```
+
+### Key features:
+- Uses `TypeVar` to support generic types (as long as comparison operations are defined)
+- Returns `Optional[int]` to indicate either the index or no match found
+- Uses `Sequence[T]` to accept any sequence type (list, tuple, etc.)
+- Includes comprehensive docstring with time/space complexity
+- Implements standard iterative binary search for O(1) space complexity
+````
+
+#### 4.2.2 Streaming Example
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Coder-Next",
+    messages=[
+        {"role": "user", "content": "Explain the difference between a stack and a queue in 3 sentences."}
+    ],
+    max_tokens=512,
+    stream=True
+)
+
+for chunk in response:
+    if chunk.choices and chunk.choices[0].delta.content:
+        print(chunk.choices[0].delta.content, end="", flush=True)
+print()
+```
+
+**Example Output:**
+
+```text Output
+A **stack** follows the **Last In, First Out (LIFO)** principle, meaning the last element added is the first one removed—operations like `push` (add) and `pop` (remove) occur at the same end, called the *top*. In contrast, a **queue** follows the **First In, First Out (FIFO)** principle, where elements are added at the *back* (enqueue) and removed from the *front* (dequeue), preserving the order of insertion. This structural difference makes stacks ideal for tasks like function call management and expression evaluation, while queues suit scheduling, buffering, and breadth-first traversal.
+```
+
+#### 4.2.3 Tool Calling Example
+
+Qwen3-Coder-Next supports tool calling capabilities. Make sure `--tool-call-parser qwen3_coder` is included in the deployment command above.
+
+**Python Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "execute_code",
+            "description": "Execute Python code and return the result",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "code": {
+                        "type": "string",
+                        "description": "The Python code to execute"
+                    }
+                },
+                "required": ["code"]
+            }
+        }
+    }
+]
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Coder-Next",
+    messages=[
+        {"role": "user", "content": "Calculate the factorial of 10 using Python"}
+    ],
+    tools=tools
+)
+
+# Check if the model wants to call a tool
+if response.choices[0].message.tool_calls:
+    tool_call = response.choices[0].message.tool_calls[0]
+    print(f"Tool: {tool_call.function.name}")
+    print(f"Arguments: {tool_call.function.arguments}")
+else:
+    print(response.choices[0].message.content)
+```
+
+**Example Output:**
+
+```text Output
+Tool: execute_code
+Arguments: {"code": "import math\nmath.factorial(10)"}
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (2x)
+- Model: Qwen/Qwen3-Coder-Next
+- Tensor Parallelism: 2
+- sglang version: 0.5.8+
+
+#### 5.1.1 Standard Scenario Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Coder-Next \
+  --tp 2 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+##### 5.1.1.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  27.86
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4218
+Request throughput (req/s):              0.36
+Input token throughput (tok/s):          219.00
+Output token throughput (tok/s):         151.48
+Peak output token throughput (tok/s):    166.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          370.48
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   2784.14
+Median E2E Latency (ms):                 2258.08
+P90 E2E Latency (ms):                    5044.43
+P99 E2E Latency (ms):                    6130.52
+---------------Time to First Token----------------
+Mean TTFT (ms):                          161.68
+Median TTFT (ms):                        168.09
+P99 TTFT (ms):                           183.26
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          6.19
+Median TPOT (ms):                        6.23
+P99 TPOT (ms):                           6.32
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.23
+Median ITL (ms):                         6.23
+P95 ITL (ms):                            6.51
+P99 ITL (ms):                            6.64
+Max ITL (ms):                            13.45
+==================================================
+```
+
+##### 5.1.1.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  39.06
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40789
+Request throughput (req/s):              2.05
+Input token throughput (tok/s):          1015.62
+Output token throughput (tok/s):         1044.73
+Peak output token throughput (tok/s):    1664.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          2060.34
+Concurrency:                             14.16
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6910.97
+Median E2E Latency (ms):                 7248.27
+P90 E2E Latency (ms):                    11612.63
+P99 E2E Latency (ms):                    13933.91
+---------------Time to First Token----------------
+Mean TTFT (ms):                          183.48
+Median TTFT (ms):                        156.50
+P99 TTFT (ms):                           311.46
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          13.61
+Median TPOT (ms):                        13.59
+P99 TPOT (ms):                           21.11
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           13.22
+Median ITL (ms):                         9.76
+P95 ITL (ms):                            10.43
+P99 ITL (ms):                            158.04
+Max ITL (ms):                            394.39
+==================================================
+```
+
+##### 5.1.1.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100
+```
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  102.81
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    252536
+Request throughput (req/s):              4.86
+Input token throughput (tok/s):          2429.99
+Output token throughput (tok/s):         2457.53
+Peak output token throughput (tok/s):    5299.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          4887.52
+Concurrency:                             94.28
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   19385.20
+Median E2E Latency (ms):                 17584.09
+P90 E2E Latency (ms):                    36762.15
+P99 E2E Latency (ms):                    42518.35
+---------------Time to First Token----------------
+Mean TTFT (ms):                          270.62
+Median TTFT (ms):                        159.65
+P99 TTFT (ms):                           938.90
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          38.57
+Median TPOT (ms):                        41.78
+P99 TPOT (ms):                           53.28
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           37.90
+Median ITL (ms):                         18.26
+P95 ITL (ms):                            167.82
+P99 ITL (ms):                            311.45
+Max ITL (ms):                            993.20
+==================================================
+```
+
+#### 5.1.2 Reasoning Scenario Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Coder-Next \
+  --tp 2 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+##### 5.1.2.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Result:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  285.02
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  44462
+Total generated tokens (retokenized):    44432
+Request throughput (req/s):              0.04
+Input token throughput (tok/s):          21.41
+Output token throughput (tok/s):         156.00
+Peak output token throughput (tok/s):    173.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          177.40
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   28499.54
+Median E2E Latency (ms):                 30424.65
+P90 E2E Latency (ms):                    49132.26
+P99 E2E Latency (ms):                    51075.28
+---------------Time to First Token----------------
+Mean TTFT (ms):                          95.51
+Median TTFT (ms):                        93.86
+P99 TTFT (ms):                           112.56
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          6.24
+Median TPOT (ms):                        6.30
+P99 TPOT (ms):                           6.60
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.39
+Median ITL (ms):                         6.34
+P95 ITL (ms):                            7.16
+P99 ITL (ms):                            7.42
+Max ITL (ms):                            12.48
+==================================================
+```
+
+##### 5.1.2.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Result:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  237.77
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total generated tokens:                  318306
+Total generated tokens (retokenized):    315646
+Request throughput (req/s):              0.34
+Input token throughput (tok/s):          166.83
+Output token throughput (tok/s):         1338.72
+Peak output token throughput (tok/s):    1727.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          1505.55
+Concurrency:                             13.88
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   41266.21
+Median E2E Latency (ms):                 41010.10
+P90 E2E Latency (ms):                    77574.22
+P99 E2E Latency (ms):                    82688.04
+---------------Time to First Token----------------
+Mean TTFT (ms):                          140.73
+Median TTFT (ms):                        84.52
+P99 TTFT (ms):                           365.86
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.32
+Median TPOT (ms):                        10.38
+P99 TPOT (ms):                           10.87
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           10.34
+Median ITL (ms):                         10.19
+P95 ITL (ms):                            10.75
+P99 ITL (ms):                            11.18
+Max ITL (ms):                            206.79
+==================================================
+```
+
+##### 5.1.2.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64
+```
+
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  384.82
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total generated tokens:                  1301025
+Total generated tokens (retokenized):    1299908
+Request throughput (req/s):              0.83
+Input token throughput (tok/s):          413.02
+Output token throughput (tok/s):         3380.83
+Peak output token throughput (tok/s):    4317.00
+Peak concurrent requests:                69
+Total token throughput (tok/s):          3793.85
+Concurrency:                             56.42
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   67847.54
+Median E2E Latency (ms):                 70724.38
+P90 E2E Latency (ms):                    120888.83
+P99 E2E Latency (ms):                    133234.48
+---------------Time to First Token----------------
+Mean TTFT (ms):                          212.24
+Median TTFT (ms):                        115.96
+P99 TTFT (ms):                           652.93
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          16.76
+Median TPOT (ms):                        16.99
+P99 TPOT (ms):                           18.18
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           16.64
+Median ITL (ms):                         15.83
+P95 ITL (ms):                            31.64
+P99 ITL (ms):                            90.85
+Max ITL (ms):                            576.60
+==================================================
+```
+
+#### 5.1.3 Summarization Scenario Benchmark
+
+##### 5.1.3.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  29.42
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4220
+Request throughput (req/s):              0.34
+Input token throughput (tok/s):          1425.35
+Output token throughput (tok/s):         143.42
+Peak output token throughput (tok/s):    169.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          1568.77
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   2941.19
+Median E2E Latency (ms):                 2411.84
+P90 E2E Latency (ms):                    5661.26
+P99 E2E Latency (ms):                    6497.45
+---------------Time to First Token----------------
+Mean TTFT (ms):                          139.46
+Median TTFT (ms):                        160.33
+P99 TTFT (ms):                           184.30
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          6.56
+Median TPOT (ms):                        6.65
+P99 TPOT (ms):                           7.29
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.65
+Median ITL (ms):                         6.68
+P95 ITL (ms):                            7.39
+P99 ITL (ms):                            7.51
+Max ITL (ms):                            16.34
+==================================================
+```
+
+##### 5.1.3.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  41.62
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total generated tokens:                  41669
+Total generated tokens (retokenized):    41664
+Request throughput (req/s):              1.92
+Input token throughput (tok/s):          7208.67
+Output token throughput (tok/s):         1001.19
+Peak output token throughput (tok/s):    1536.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          8209.86
+Concurrency:                             14.27
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   7421.29
+Median E2E Latency (ms):                 7985.77
+P90 E2E Latency (ms):                    12122.09
+P99 E2E Latency (ms):                    14595.05
+---------------Time to First Token----------------
+Mean TTFT (ms):                          248.49
+Median TTFT (ms):                        179.25
+P99 TTFT (ms):                           915.90
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          14.13
+Median TPOT (ms):                        14.28
+P99 TPOT (ms):                           24.02
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           13.80
+Median ITL (ms):                         10.46
+P95 ITL (ms):                            11.00
+P99 ITL (ms):                            173.14
+Max ITL (ms):                            823.32
+==================================================
+```
+
+##### 5.1.3.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model Qwen/Qwen3-Coder-Next \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64
+```
+
+- Result:
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  85.74
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total generated tokens:                  170000
+Total generated tokens (retokenized):    169983
+Request throughput (req/s):              3.73
+Input token throughput (tok/s):          14858.12
+Output token throughput (tok/s):         1982.80
+Peak output token throughput (tok/s):    3734.00
+Peak concurrent requests:                70
+Total token throughput (tok/s):          16840.92
+Concurrency:                             59.75
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   16008.12
+Median E2E Latency (ms):                 15460.65
+P90 E2E Latency (ms):                    27705.81
+P99 E2E Latency (ms):                    32874.74
+---------------Time to First Token----------------
+Mean TTFT (ms):                          476.99
+Median TTFT (ms):                        177.50
+P99 TTFT (ms):                           3014.39
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          29.81
+Median TPOT (ms):                        31.19
+P99 TPOT (ms):                           45.53
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           29.29
+Median ITL (ms):                         15.75
+P95 ITL (ms):                            173.94
+P99 ITL (ms):                            202.00
+Max ITL (ms):                            2783.23
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python benchmark/gsm8k/bench_sglang.py --port 30000
+```
+
+- **Test Results:**
+
+```text Output
+Accuracy: 0.965
+Invalid: 0.000
+Latency: 26.407 s
+Output throughput: 929.132 token/s
+```
+
+#### 5.2.2 MMLU Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+cd benchmark/mmlu
+bash download_data.sh
+python3 bench_sglang.py --port 30000
+```
+
+- **Test Results:**
+
+```text Output
+subject: abstract_algebra, #q:100, acc: 0.780
+subject: anatomy, #q:135, acc: 0.807
+subject: astronomy, #q:152, acc: 0.921
+subject: business_ethics, #q:100, acc: 0.820
+subject: clinical_knowledge, #q:265, acc: 0.860
+subject: college_biology, #q:144, acc: 0.944
+subject: college_chemistry, #q:100, acc: 0.590
+subject: college_computer_science, #q:100, acc: 0.820
+subject: college_mathematics, #q:100, acc: 0.800
+subject: college_medicine, #q:173, acc: 0.803
+subject: college_physics, #q:102, acc: 0.775
+subject: computer_security, #q:100, acc: 0.880
+subject: conceptual_physics, #q:235, acc: 0.936
+subject: econometrics, #q:114, acc: 0.807
+subject: electrical_engineering, #q:145, acc: 0.834
+subject: elementary_mathematics, #q:378, acc: 0.854
+subject: formal_logic, #q:126, acc: 0.802
+subject: global_facts, #q:100, acc: 0.610
+subject: high_school_biology, #q:310, acc: 0.971
+subject: high_school_chemistry, #q:203, acc: 0.803
+subject: high_school_computer_science, #q:100, acc: 0.920
+subject: high_school_european_history, #q:165, acc: 0.891
+subject: high_school_geography, #q:198, acc: 0.929
+subject: high_school_government_and_politics, #q:193, acc: 0.969
+subject: high_school_macroeconomics, #q:390, acc: 0.903
+subject: high_school_mathematics, #q:270, acc: 0.689
+subject: high_school_microeconomics, #q:238, acc: 0.962
+subject: high_school_physics, #q:151, acc: 0.854
+subject: high_school_psychology, #q:545, acc: 0.947
+subject: high_school_statistics, #q:216, acc: 0.815
+subject: high_school_us_history, #q:204, acc: 0.907
+subject: high_school_world_history, #q:237, acc: 0.937
+subject: human_aging, #q:223, acc: 0.821
+subject: human_sexuality, #q:131, acc: 0.840
+subject: international_law, #q:121, acc: 0.934
+subject: jurisprudence, #q:108, acc: 0.870
+subject: logical_fallacies, #q:163, acc: 0.847
+subject: machine_learning, #q:112, acc: 0.812
+subject: management, #q:103, acc: 0.922
+subject: marketing, #q:234, acc: 0.923
+subject: medical_genetics, #q:100, acc: 0.970
+subject: miscellaneous, #q:783, acc: 0.941
+subject: moral_disputes, #q:346, acc: 0.850
+subject: moral_scenarios, #q:895, acc: 0.726
+subject: nutrition, #q:306, acc: 0.915
+subject: philosophy, #q:311, acc: 0.859
+subject: prehistory, #q:324, acc: 0.889
+subject: professional_accounting, #q:282, acc: 0.723
+subject: professional_law, #q:1534, acc: 0.648
+subject: professional_medicine, #q:272, acc: 0.923
+subject: professional_psychology, #q:612, acc: 0.845
+subject: public_relations, #q:110, acc: 0.782
+subject: security_studies, #q:245, acc: 0.796
+subject: sociology, #q:201, acc: 0.925
+subject: us_foreign_policy, #q:100, acc: 0.950
+subject: virology, #q:166, acc: 0.572
+subject: world_religions, #q:171, acc: 0.883
+Total latency: 208.985
+Average accuracy: 0.834
+```

docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder.mdx

@@ -0,0 +1,520 @@
+---
+title: Qwen3-Coder
+metatags:
+    description: "Deploy Qwen3-Coder(480B, 30B) MoE coding model with SGLang on AMD MI300X (MI325X, MI355X)"
+---
+
+import { Qwen3CoderDeployment } from '/src/snippets/autoregressive/qwen3-coder-deployment.jsx';
+
+## 1. Model Introduction
+
+[Qwen3-Coder](https://huggingface.co/collections/Qwen/qwen3-coder) is the latest code-focused large language model series from the Qwen team. Built on the foundation of Qwen3, Qwen3-Coder delivers exceptional performance in code generation, understanding, and reasoning tasks.
+
+**Key Features:**
+
+- **State-of-the-art Coding Performance**: Achieves top-tier results on HumanEval, MBPP, LiveCodeBench, and other major coding benchmarks.
+- **Tool Calling Support**: Native support for function calling and tool use, enabling seamless integration with external APIs and services.
+- **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
+- **Multilingual Code Support**: Proficient in Python, JavaScript, TypeScript, Java, C++, Go, Rust, and many other programming languages.
+- **MoE Architecture**: Efficient Mixture-of-Experts design for optimal performance-to-cost ratio.
+- **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
+- **NVIDIA GPU Support**: Compatible with NVIDIA GB200 and B200 GPUs via SGLang (verified).
+
+For more details, please refer to the [official Qwen3-Coder GitHub Repository](https://github.com/QwenLM/Qwen3-Coder).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X and NVIDIA B200, GB200 hardware platforms.
+
+### 3.1 Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and quantization method.
+
+<Qwen3CoderDeployment />
+
+### 3.2 Configuration Tips
+
+**AMD (MI300X/MI325X/MI355X):**
+* **Memory Management**: We have verified successful deployment on MI300X/MI325X/MI355X with `--context-length 8192`. Larger context lengths may be supported but require additional memory.
+* **Expert Parallelism**: For 480B-A35B with FP8 quantization, `--ep 2` is required to satisfy the dimension alignment requirement.
+* **Page Size**: `--page-size 32` is recommended for MoE models to optimize memory usage.
+* **Environment Variable**: If you encounter aiter-related issues, try setting `SGLANG_USE_AITER=0`.
+
+**NVIDIA (B200/GB200):**
+* **MOE Runner Backend**: FP8 uses `--moe-runner-backend triton`, NVFP4 uses `--moe-runner-backend flashinfer_cutlass`.
+* **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`).
+* **DP Attention**: NVFP4 configuration supports `--enable-dp-attention` for improved throughput.
+
+**General:**
+* **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Code Generation Example
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints."
+    }
+]
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
+    messages=messages,
+    max_tokens=2048,
+    temperature=0.7
+)
+
+print(response.choices[0].message.content)
+```
+
+**Example Output:**
+
+````text Output
+```python
+from typing import List, Optional, TypeVar
+
+T = TypeVar('T')
+
+def binary_search(arr: List[T], target: T) -> Optional[int]:
+    """
+    Perform binary search on a sorted list to find the index of a target element.
+
+    This function implements the binary search algorithm, which efficiently finds
+    a target value in a sorted array by repeatedly dividing the search interval
+    in half.
+
+    Args:
+        arr (List[T]): A sorted list of elements to search through.
+        target (T): The element to search for in the list.
+
+    Returns:
+        Optional[int]: The index of the target element if found, None otherwise.
+
+    Time Complexity:
+        O(log n) where n is the number of elements in the array.
+
+    Space Complexity:
+        O(1) - iterative implementation uses constant extra space.
+
+    Examples:
+        >>> binary_search([1, 2, 3, 4, 5], 3)
+        2
+        >>> binary_search([1, 2, 3, 4, 5], 6)
+        None
+        >>> binary_search(['a', 'b', 'c', 'd'], 'b')
+        1
+        >>> binary_search([], 1)
+        None
+    """
+    if not arr:
+        return None
+
+    left: int = 0
+    right: int = len(arr) - 1
+
+    while left <= right:
+        mid: int = (left + right) // 2
+
+        if arr[mid] == target:
+            return mid
+        elif arr[mid] < target:
+            left = mid + 1
+        else:
+            right = mid - 1
+
+    return None
+
+# Alternative recursive implementation
+def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]:
+    """
+    Perform binary search recursively on a sorted list to find the index of a target element.
+
+    Args:
+        arr (List[T]): A sorted list of elements to search through.
+        target (T): The element to search for in the list.
+        left (int): Left boundary of the search range (inclusive).
+        right (Optional[int]): Right boundary of the search range (inclusive).
+
+    Returns:
+        Optional[int]: The index of the target element if found, None otherwise.
+
+    Time Complexity:
+        O(log n) where n is the number of elements in the array.
+
+    Space Complexity:
+        O(log n) due to recursive call stack.
+
+    Examples:
+        >>> binary_search_recursive([1, 2, 3, 4, 5], 3)
+        2
+        >>> binary_search_recursive([1, 2, 3, 4, 5], 6)
+        None
+    """
+    if not arr:
+        return None
+
+    if right is None:
+        right = len(arr) - 1
+
+    if left > right:
+        return None
+
+    mid: int = (left + right) // 2
+
+    if arr[mid] == target:
+        return mid
+    elif arr[mid] < target:
+        return binary_search_recursive(arr, target, mid + 1, right)
+    else:
+        return binary_search_recursive(arr, target, left, mid - 1)
+```
+
+This implementation provides:
+
+1. **Main function** (`binary_search`): An iterative implementation that's more memory-efficient
+2. **Alternative function** (`binary_search_recursive`): A recursive implementation for educational purposes
+3. **Type hints**: Using generics (`TypeVar`) to work with any comparable type
+4. **Comprehensive docstring**: Including description, parameters, return value, complexity analysis, and examples
+5. **Edge case handling**: Empty lists, elements not found, etc.
+6. **Clear variable names**: Self-documenting code
+7. **Examples**: Doctest-style examples in the docstring
+
+The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or `None` if not found.
+````
+
+#### 4.2.2 Tool Calling Example
+
+Qwen3-Coder supports tool calling capabilities. Enable the tool call parser during deployment. The following example uses 30B-A3B model:
+
+```shell Command
+SGLANG_USE_AITER=0 python -m sglang.launch_server \
+  --model Qwen/Qwen3-Coder-30B-A3B-Instruct \
+  --tp 1 \
+  --context-length 8192 \
+  --page-size 32 \
+  --tool-call-parser qwen3_coder
+```
+
+**Python Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "execute_code",
+            "description": "Execute Python code and return the result",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "code": {
+                        "type": "string",
+                        "description": "The Python code to execute"
+                    }
+                },
+                "required": ["code"]
+            }
+        }
+    }
+]
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Coder-30B-A3B-Instruct",
+    messages=[
+        {"role": "user", "content": "Calculate the factorial of 10 using Python"}
+    ],
+    tools=tools,
+    temperature=0.7
+)
+
+# Check if the model wants to call a tool
+if response.choices[0].message.tool_calls:
+    tool_call = response.choices[0].message.tool_calls[0]
+    print(f"Tool: {tool_call.function.name}")
+    print(f"Arguments: {tool_call.function.arguments}")
+else:
+    # Model may return tool call in content format
+    print(response.choices[0].message.content)
+```
+
+**Example Output:**
+
+```text Output
+Tool: execute_code
+Arguments: {"code": "def factorial(n):\n    if n == 0 or n == 1:\n        return 1\n    else:\n        return n * factorial(n-1)\n\nresult = factorial(10)\nresult"}
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: AMD MI300X GPU (8x)
+- Model: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
+- Tensor Parallelism: 8
+- Expert Parallelism: 2
+- sglang version: 0.5.7
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation with random dataset.
+
+#### 5.1.1 Standard Scenario Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+SGLANG_USE_AITER=0 python -m sglang.launch_server \
+  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
+  --tp 8 \
+  --ep 2 \
+  --context-length 8192 \
+  --page-size 32 \
+  --trust-remote-code
+```
+
+##### 5.1.1.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  73.79
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4104
+Request throughput (req/s):              0.14
+Input token throughput (tok/s):          82.68
+Output token throughput (tok/s):         57.19
+Peak output token throughput (tok/s):    59.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          139.86
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   7376.26
+Median E2E Latency (ms):                 5851.51
+P90 E2E Latency (ms):                    13351.89
+P99 E2E Latency (ms):                    16908.32
+---------------Time to First Token----------------
+Mean TTFT (ms):                          191.93
+Median TTFT (ms):                        126.06
+P99 TTFT (ms):                           662.15
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          17.06
+Median TPOT (ms):                        17.07
+P99 TPOT (ms):                           17.08
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           17.06
+Median ITL (ms):                         17.06
+P95 ITL (ms):                            17.14
+P99 ITL (ms):                            17.19
+Max ITL (ms):                            18.53
+==================================================
+```
+
+##### 5.1.1.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  87.04
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40364
+Request throughput (req/s):              0.92
+Input token throughput (tok/s):          455.77
+Output token throughput (tok/s):         468.83
+Peak output token throughput (tok/s):    608.00
+Peak concurrent requests:                20
+Total token throughput (tok/s):          924.59
+Concurrency:                             13.76
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   14966.88
+Median E2E Latency (ms):                 15871.93
+P90 E2E Latency (ms):                    24983.41
+P99 E2E Latency (ms):                    29504.85
+---------------Time to First Token----------------
+Mean TTFT (ms):                          388.94
+Median TTFT (ms):                        157.49
+P99 TTFT (ms):                           1318.63
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          29.41
+Median TPOT (ms):                        29.22
+P99 TPOT (ms):                           43.48
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           28.64
+Median ITL (ms):                         26.42
+P95 ITL (ms):                            27.51
+P99 ITL (ms):                            131.63
+Max ITL (ms):                            995.11
+==================================================
+```
+
+##### 5.1.1.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  177.82
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total generated tokens:                  170134
+Total generated tokens (retokenized):    168387
+Request throughput (req/s):              1.80
+Input token throughput (tok/s):          893.84
+Output token throughput (tok/s):         956.80
+Peak output token throughput (tok/s):    1728.00
+Peak concurrent requests:                70
+Total token throughput (tok/s):          1850.64
+Concurrency:                             58.88
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   32716.53
+Median E2E Latency (ms):                 30896.37
+P90 E2E Latency (ms):                    65605.24
+P99 E2E Latency (ms):                    80970.63
+---------------Time to First Token----------------
+Mean TTFT (ms):                          372.97
+Median TTFT (ms):                        181.67
+P99 TTFT (ms):                           529.01
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          62.98
+Median TPOT (ms):                        50.44
+P99 TPOT (ms):                           204.24
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           60.95
+Median ITL (ms):                         37.87
+P95 ITL (ms):                            143.98
+P99 ITL (ms):                            148.02
+Max ITL (ms):                            36863.32
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+
+##### AMD (MI300X/MI325X/MI355X)
+
+- **Results**:
+
+  - Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
+    ```
+    Accuracy: 0.965
+    Invalid: 0.000
+    Latency: 23.084 s
+    Output throughput: 1148.425 token/s
+    ```
+
+##### NVIDIA (B200/GB200)
+
+For deployment commands, see [Section 3.1](#31-configuration).
+
+  - Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 (tp=8, ep=2)
+    ```
+    Accuracy: 0.950
+    Invalid: 0.000
+    Latency: 12.914 s
+    Output throughput: 2065.515 token/s
+    ```
+
+  - nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4, tp=8, ep=1)
+    ```
+    Accuracy: 0.970
+    Invalid: 0.000
+    Latency: 71.280 s
+    Output throughput: 390.080 token/s
+    ```

docs_new/cookbook/autoregressive/Qwen/Qwen3-Next.mdx

@@ -0,0 +1,774 @@
+---
+title: Qwen3-Next
+metatags:
+    description: "Deploy Qwen3-Next with SGLang - hybrid attention architecture supporting 262K context, 80B MoE with 3B active parameters, and multi-token prediction."
+---
+
+import { Qwen3NextDeployment } from '/src/snippets/autoregressive/qwen3-next-deployment.jsx';
+
+## 1. Model Introduction
+
+[Qwen3-Next](https://huggingface.co/collections/Qwen/qwen3-next) is an advanced large language model architecture developed by Alibaba's Qwen team, designed to enhance efficiency and performance in handling extensive contexts and large-scale parameters. It features advanced capabilities in reasoning, function calling, and multilingual understanding.
+
+Qwen3-Next introduces several groundbreaking innovations:
+
+- **Hybrid Attention Mechanism**: Replaces standard attention with a combination of **Gated DeltaNet** (linear attention) and **Full Attention**, enabling efficient processing of context lengths up to 262,144 tokens. This hybrid approach makes it ideal for analyzing lengthy documents such as entire books or contracts.
+
+- **Highly Sparse Mixture-of-Experts (MoE)**: Features an 80-billion parameter architecture where only 3 billion parameters are active during inference. This design reduces computational costs by up to 90% while maintaining high performance, drastically reducing FLOPs per token without compromising model capacity.
+
+- **Multi-Token Prediction (MTP)**: Enables generation of multiple tokens per inference step, significantly reducing latency and enhancing user experience in real-time applications. This innovation boosts both pretraining performance and inference speed.
+
+- **Multilingual Support**: Natively supports 119 languages, facilitating seamless cross-lingual tasks and making it versatile for global applications.
+
+- **Enterprise-Ready Deployment**: Released under the Apache 2.0 license, offering flexible deployment options including on-premises, virtual private cloud (VPC), and private cloud environments, ensuring security and compliance for enterprise use.
+
+- **Advanced Reasoning & Stability**: Demonstrates clear improvement in reasoning performance with support for tool use during inference. Includes stability optimizations such as **zero-centered** and **weight-decayed layernorm** for robust pre-training and post-training.
+
+For more details, please refer to the [official Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+The Qwen3-Next series comes in only one size but offers different thinking modes. Recommended starting configurations vary depending on hardware.
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
+
+<Qwen3NextDeployment />
+
+### 3.2 Configuration Tips
+
+- `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.
+
+- `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`.
+
+- `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`.
+
+- **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
+  - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
+  - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+1. **Streaming with Thinking Process:**
+
+   Qwen3-Next-80B-A3B-Thinking only supports thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections.
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Next-80B-A3B-Thinking \
+  --reasoning-parser qwen3 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36.
+
+=============== Content =================
+
+To find **15% of 240**, follow these steps:
+
+---
+
+### **Step 1: Understand what "percent" means**
+- "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form.
+
+---
+
+### **Step 2: Multiply the number by the decimal**
+- To find 15% of 240, multiply:
+  $$
+  240 \times 0.15
+  $$
+
+---
+
+### **Step 3: Break it down for clarity (optional but helpful)**
+- **10% of 240** = $ 240 \times 0.1 = 24 $
+- **5% of 240** = $ 240 \times 0.05 = 12 $
+- Add them together:
+  $$
+  24 + 12 = 36
+  $$
+
+---
+
+### **Step 4: Confirm with direct multiplication**
+- $ 240 \times 0.15 = 36 $
+
+---
+
+### ✅ Final Answer:
+$$
+\boxed{36}
+$$
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+2. **Turn off Thinking:**
+
+   Qwen3-Next-80B-A3B-Instruct only supports instruct (non-thinking) mode.
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Turn off thinking process
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Next-80B-A3B-Instruct",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True,
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+To find **15% of 240**, follow these steps:
+
+---
+
+### **Step 1: Understand what percentage means**
+"Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction:
+
+$$
+\frac{15}{100}
+$$
+
+---
+
+### **Step 2: Multiply the fraction by the number**
+To find 15% of 240, multiply:
+
+$$
+\frac{15}{100} \times 240
+$$
+
+---
+
+### **Step 3: Simplify the multiplication**
+You can simplify this in a couple of ways.
+
+#### **Option A: Multiply first, then divide**
+$$
+15 \times 240 = 3600
+$$
+Then divide by 100:
+$$
+\frac{3600}{100} = 36
+$$
+
+#### **Option B: Simplify the fraction first**
+$$
+\frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)}
+$$
+Now multiply:
+$$
+\frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36
+$$
+
+---
+
+### **Step 4: Final Answer**
+$$
+\boxed{36}
+$$
+
+So, **15% of 240 is 36**.
+```
+
+#### 4.2.2 Tool Calling
+
+Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen/Qwen3-Next-80B-A3B-Thinking both support tool calling capabilities. Enable the tool call parser:
+
+**Python Example (without Thinking Process):**
+
+Start sglang server:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
+  --tool-call-parser qwen \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Next-80B-A3B-Instruct",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"🔧 Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+<tool_call>
+{"name": "get_weather", "arguments": {"location": "Beijing"}}
+</tool_call>
+```
+
+**Python Example (with Thinking Process):**
+
+Start sglang server:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Next-80B-A3B-Thinking \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"🔧 Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}.
+
+<tool_call>
+{"name": "get_weather", "arguments": {"location": "Beijing"}}
+</tool_call>
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+#### 4.2.3 Processing Ultra-Long Texts
+
+Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method.
+
+**Qwen3-Next-80B-A3B-Instruct**
+
+```shell Command
+SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct    --tp 8   --host 0.0.0.0   --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
+
+```
+
+**Qwen3-Next-80B-A3B-Thinking**
+
+```shell Command
+SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking   --reasoning-parser qwen3     --tp 8   --host 0.0.0.0   --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000
+
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (8x)
+- Tensor Parallelism: 8
+- Model: Qwen/Qwen3-Next-80B-A3B-Instruct
+- sglang version: 0.5.6
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Server Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
+  --tp 8
+```
+
+- Test Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --num-prompt 100 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     100
+Benchmark duration (s):                  146.52
+Total input tokens:                      33839
+Total input text tokens:                 33839
+Total input vision tokens:               0
+Total generated tokens:                  21640
+Total generated tokens (retokenized):    21619
+Request throughput (req/s):              0.68
+Input token throughput (tok/s):          230.95
+Output token throughput (tok/s):         147.70
+Peak output token throughput (tok/s):    164.00
+Peak concurrent requests:                6
+Total token throughput (tok/s):          378.65
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   1464.81
+Median E2E Latency (ms):                 1077.48
+---------------Time to First Token----------------
+Mean TTFT (ms):                          127.88
+Median TTFT (ms):                        132.88
+P99 TTFT (ms):                           212.85
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          6.19
+Median TPOT (ms):                        6.17
+P99 TPOT (ms):                           6.64
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.21
+Median ITL (ms):                         6.16
+P95 ITL (ms):                            6.51
+P99 ITL (ms):                            6.71
+Max ITL (ms):                            10.07
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Server Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
+  --tp 8 \
+```
+
+- Test Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --num-prompt 1000 \
+  --max-concurrency 100
+```
+
+**Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  100.32
+Total input tokens:                      302118
+Total input text tokens:                 302118
+Total input vision tokens:               0
+Total generated tokens:                  195775
+Total generated tokens (retokenized):    195016
+Request throughput (req/s):              9.97
+Input token throughput (tok/s):          3011.69
+Output token throughput (tok/s):         1951.60
+Peak output token throughput (tok/s):    5909.00
+Peak concurrent requests:                120
+Total token throughput (tok/s):          4963.29
+Concurrency:                             93.05
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   9333.98
+Median E2E Latency (ms):                 6054.12
+---------------Time to First Token----------------
+Mean TTFT (ms):                          161.77
+Median TTFT (ms):                        137.94
+P99 TTFT (ms):                           503.29
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          50.87
+Median TPOT (ms):                        50.28
+P99 TPOT (ms):                           122.87
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           47.11
+Median ITL (ms):                         13.84
+P95 ITL (ms):                            195.33
+P99 ITL (ms):                            289.56
+Max ITL (ms):                            486.38
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
+```
+
+- **Results**:
+
+  - Qwen3-Next-80B-A3B-Instruct
+
+    ```
+    Accuracy: 0.960
+    Invalid: 0.000
+    Latency: 12.673 s
+    Output throughput: 2538.255 token/s
+    ```
+
+  - Qwen3-Next-80B-A3B-Thinking
+    ```
+    Accuracy: 0.935
+    Invalid: 0.000
+    Latency: 9.912 s
+    Output throughput: 3288.737 token/s
+    ```
+
+### 5.2.2 MMLU Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+cd sglang
+bash benchmark/mmlu/download_data.sh
+python3 benchmark/mmlu/bench_sglang.py --nsub 10
+```
+
+- **Results**:
+
+  - Qwen3-Next-80B-A3B-Instruct
+
+    ```
+    subject: abstract_algebra, #q:100, acc: 0.800
+    subject: anatomy, #q:135, acc: 0.807
+    subject: astronomy, #q:152, acc: 0.947
+    subject: business_ethics, #q:100, acc: 0.810
+    subject: clinical_knowledge, #q:265, acc: 0.894
+    subject: college_biology, #q:144, acc: 0.972
+    subject: college_chemistry, #q:100, acc: 0.680
+    subject: college_computer_science, #q:100, acc: 0.860
+    subject: college_mathematics, #q:100, acc: 0.780
+    subject: college_medicine, #q:173, acc: 0.861
+    Total latency: 10.098
+    Average accuracy: 0.856
+    ```
+
+  - Qwen3-Next-80B-A3B-Thinking
+    ```
+    subject: abstract_algebra, #q:100, acc: 0.780
+    subject: anatomy, #q:135, acc: 0.815
+    subject: astronomy, #q:152, acc: 0.941
+    subject: business_ethics, #q:100, acc: 0.870
+    subject: clinical_knowledge, #q:265, acc: 0.894
+    subject: college_biology, #q:144, acc: 0.965
+    subject: college_chemistry, #q:100, acc: 0.670
+    subject: college_computer_science, #q:100, acc: 0.840
+    subject: college_mathematics, #q:100, acc: 0.770
+    subject: college_medicine, #q:173, acc: 0.861
+    Total latency: 10.236
+    Average accuracy: 0.855
+    ```

docs_new/cookbook/autoregressive/Qwen/Qwen3-VL.mdx

@@ -0,0 +1,777 @@
+---
+title: Qwen3-VL
+metatags:
+    description: "Deploy Qwen3-VL vision-language models with SGLang - open model for text, 262K context, enhanced visual reasoning and agent capabilities."
+---
+
+
+## 1. Model Introduction
+
+[Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
+
+This generation delivers comprehensive upgrades across the board:
+
+- **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942)
+- **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities.
+- **Extended context length**: Supports up to 262K tokens for processing long documents and videos.
+- **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics.
+- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
+- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
+
+For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs. The recommended launch configurations vary by hardware and model size.
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
+
+import { Qwen3VLDeployment } from "/src/snippets/autoregressive/qwen3-vl-deployment.jsx";
+
+<Qwen3VLDeployment />
+
+### 3.2 Configuration Tips
+
+* **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility.
+* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
+* **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
+* **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
+* **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Multi-Modal Inputs
+
+Qwen3-VL supports both image and video inputs. Here's a basic example with image input:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Read all the text in the image."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 3.37s
+Generated text: Auntie Anne's
+
+CINNAMON SUGAR
+1 x 17,000                    17,000
+
+SUB TOTAL                    17,000
+
+GRAND TOTAL                  17,000
+
+CASH IDR                     20,000
+
+CHANGE DUE                  3,000
+```
+
+**Multi-Image Input Example:**
+
+Qwen3-VL can process multiple images in a single request for comparison or analysis:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
+                }
+            },
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 10.18s
+Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering.
+
+The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy.
+
+In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout.
+
+Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance.
+
+In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose.
+```
+
+**Video Input Example:**
+
+Qwen3-VL supports video understanding by processing video URLs:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:8000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "video_url",
+                "video_url": {
+                    "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Describe what happens in this video."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
+    messages=messages,
+    max_tokens=2048
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Note:**
+
+- For video processing, ensure you have sufficient context length configured (up to 262K tokens)
+- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
+- You can also provide local file paths using `file://` protocol
+
+**Example Output:**
+
+```text Output
+Response costs: 3.89s
+Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
+```
+
+#### 4.2.2 Reasoning Parser
+
+Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-VL-235B-A22B-Thinking \
+  --reasoning-parser qwen3 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+To solve this problem, I need to calculate 15% of 240.
+Step 1: Convert 15% to decimal: 15% = 0.15
+Step 2: Multiply 240 by 0.15
+Step 3: 240 × 0.15 = 36
+=============== Content =================
+
+The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.3 Tool Calling
+
+Qwen3-VL supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-VL-235B-A22B-Thinking \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
+I should call the function with location="Beijing".
+=============== Content =================
+
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The weather in Beijing is currently 22°C and sunny."
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (8x)
+- Model: Qwen3-VL-235B-A22B-Instruct
+- Tensor Parallelism: 8
+- sglang version: 0.5.6
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  45.97
+Total input tokens:                      18348
+Total input text tokens:                 708
+Total input vision tokens:               17640
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    3423
+Request throughput (req/s):              0.22
+Input token throughput (tok/s):          399.17
+Output token throughput (tok/s):         91.81
+Peak output token throughput (tok/s):    96.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          490.98
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   4594.52
+Median E2E Latency (ms):                 3725.04
+---------------Time to First Token----------------
+Mean TTFT (ms):                          193.35
+Median TTFT (ms):                        196.32
+P99 TTFT (ms):                           222.75
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.44
+Median TPOT (ms):                        10.44
+P99 TPOT (ms):                           10.47
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           11.78
+Median ITL (ms):                         10.48
+P95 ITL (ms):                            21.01
+P99 ITL (ms):                            31.40
+Max ITL (ms):                            31.92
+==================================================
+```
+
+**Optimized Results (with CUDA IPC Transport):**
+
+For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features.
+
+- Model Deployment Command:
+
+```shell Command
+SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
+  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 100 \
+  --max-concurrency 1
+```
+
+- **Test Results:**
+
+  With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     100
+Benchmark duration (s):                  566.84
+Total input tokens:                      183667
+Total input text tokens:                 7267
+Total input vision tokens:               176400
+Total generated tokens:                  52444
+Total generated tokens (retokenized):    28702
+Request throughput (req/s):              0.18
+Input token throughput (tok/s):          324.02
+Output token throughput (tok/s):         92.52
+Peak output token throughput (tok/s):    96.00
+Peak concurrent requests:                3
+Total token throughput (tok/s):          416.54
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   5667.50
+Median E2E Latency (ms):                 5830.00
+---------------Time to First Token----------------
+Mean TTFT (ms):                          191.16
+Median TTFT (ms):                        182.58
+P99 TTFT (ms):                           244.58
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.46
+Median TPOT (ms):                        10.46
+P99 TPOT (ms):                           10.48
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           13.91
+Median ITL (ms):                         10.56
+P95 ITL (ms):                            21.35
+P99 ITL (ms):                            31.55
+Max ITL (ms):                            42.36
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --host 127.0.0.1 \
+  --port 8000 \
+  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+- **Test Results:**
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  584.65
+Total input tokens:                      1839015
+Total input text tokens:                 75015
+Total input vision tokens:               1764000
+Total generated tokens:                  510855
+Total generated tokens (retokenized):    284284
+Request throughput (req/s):              1.71
+Input token throughput (tok/s):          3145.50
+Output token throughput (tok/s):         873.78
+Peak output token throughput (tok/s):    2855.00
+Peak concurrent requests:                107
+Total token throughput (tok/s):          4019.29
+Concurrency:                             98.35
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   57502.05
+Median E2E Latency (ms):                 54301.08
+---------------Time to First Token----------------
+Mean TTFT (ms):                          5802.23
+Median TTFT (ms):                        1444.75
+P99 TTFT (ms):                           46675.92
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          100.22
+Median TPOT (ms):                        105.43
+P99 TPOT (ms):                           144.37
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           134.20
+Median ITL (ms):                         25.57
+P95 ITL (ms):                            558.14
+P99 ITL (ms):                            1449.01
+Max ITL (ms):                            33453.23
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 MMMU Benchmark
+
+You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`:
+
+- Benchmark Command:
+
+```shell Command
+uv pip install lmms_eval
+
+python3 -m lmms_eval \
+  --model openai_compatible \
+  --model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:8000/v1/" \
+  --tasks mmmu_val \
+  --batch_size 128 \
+  --log_samples \
+  --log_samples_suffix "openai_compatible" \
+  --output_path ./logs \
+  --gen_kwargs "max_new_tokens=4096"
+```
+
+- **Test Results:**
+
+```text Output
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "12%"}} />
+    <col style={{width: "11%"}} />
+    <col style={{width: "11%"}} />
+    <col style={{width: "11%"}} />
+    <col style={{width: "11%"}} />
+    <col style={{width: "11%"}} />
+    <col style={{width: "11%"}} />
+    <col style={{width: "11%"}} />
+    <col style={{width: "11%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Stderr</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_val</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_acc</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6567</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>±</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
+    </tr>
+  </tbody>
+</table>
+```

docs_new/cookbook/autoregressive/Qwen/Qwen3.6.mdx

@@ -0,0 +1,471 @@
+---
+title: Qwen3.6
+metatags:
+    description: "Deploy Qwen3.6 with SGLang - open-weight 35B MoE multimodal model with 3B active parameters, thinking preservation, tool calling, MTP, and long-context support."
+tag: NEW
+---
+
+import { Qwen36Deployment } from '/src/snippets/autoregressive/qwen36-deployment.jsx';
+
+## 1. Model Introduction
+
+[Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) is the first open-weight variant of the Qwen3.6 series developed by Alibaba. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, delivering substantial upgrades in agentic coding and thinking preservation.
+
+Qwen3.6 features a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (35B total parameters, 3B activated), supporting multimodal inputs (text, image, video) and natively handles context lengths of up to 262,144 tokens, extensible to over 1M tokens.
+
+**Key Features:**
+
+- **Agentic Coding**: Handles frontend workflows and repository-level reasoning with greater fluency and precision
+- **Thinking Preservation**: New option to retain reasoning context from historical messages, streamlining iterative development
+- **Efficient Hybrid Architecture**: Gated Delta Networks + sparse MoE (35B total / 3B active) for high-throughput inference
+- **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
+- **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser
+- **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency
+- **Multimodal**: Unified vision-language model supporting text, image, and video inputs
+
+**Available Models:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr>
+      <th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Model</th>
+      <th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Weights</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3.6-35B-A3B (BF16)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3.6-35B-A3B (FP8)</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)</td>
+    </tr>
+  </tbody>
+</table>
+
+**License:** Apache 2.0
+
+## 2. SGLang Installation
+
+SGLang `>=0.5.10` is required for Qwen3.6. You can install from source or use a Docker image:
+
+```bash Command
+# Install from PyPI
+uv pip install "sglang[all]"
+
+# Or install from source
+uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
+
+# Or use Docker (NVIDIA GPUs)
+docker pull lmsysorg/sglang:latest
+```
+
+For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/installation).
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities.
+
+
+<Qwen36Deployment />
+
+### 3.2 Configuration Tips
+
+- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
+- **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
+  - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
+  - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput.
+- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
+- Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
+- **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
+- **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200.
+- For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
+- Hardware requirements:
+    - **BF16**: ~35B parameters require ~70GB of GPU memory for weights. TP=1 fits on all supported hardware.
+    - **FP8**: The FP8 quantized model requires ~35GB for weights. TP=1 fits on all supported hardware.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <thead>
+    <tr>
+      <th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Hardware</th>
+      <th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>Memory</th>
+      <th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>BF16 TP</th>
+      <th style={{padding: "9px 12px", textAlign: "left", borderBottom: "1px solid rgba(148,163,184,0.3)"}}>FP8 TP</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>80GB</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.05)"}}>H200</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>141GB</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>183GB</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1</td>
+    </tr>
+  </tbody>
+</table>
+
+
+## 4. Model Invocation
+
+Deploy Qwen3.6-35B-A3B with the following command (H200, all features enabled):
+
+```shell Command
+SGLANG_ENABLE_SPEC_V2=1 sglang serve \
+  --model-path Qwen/Qwen3.6-35B-A3B-FP8 \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen3_coder \
+  --speculative-algorithm EAGLE \
+  --speculative-num-steps 3 \
+  --speculative-eagle-topk 1 \
+  --speculative-num-draft-tokens 4 \
+  --mem-fraction-static 0.8 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Vision Input
+
+Qwen3.6 supports image and video inputs as a unified vision-language model.
+
+**Image Input Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3.6-35B-A3B-FP8",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
+                    }
+                },
+                {
+                    "type": "text",
+                    "text": "Describe this image in detail."
+                }
+            ]
+        }
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+thinking_started = False
+has_thinking = False
+has_answer = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        if delta.content:
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Video Input Example:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3.6-35B-A3B-FP8",
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "video_url",
+                    "video_url": {
+                        "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4"
+                    }
+                },
+                {
+                    "type": "text",
+                    "text": "Describe what happens in this video."
+                }
+            ]
+        }
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+thinking_started = False
+has_thinking = False
+has_answer = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        if delta.content:
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+### 4.3 Advanced Usage
+
+#### 4.3.1 Reasoning Parser
+
+Qwen3.6 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
+
+To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
+
+- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
+- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
+
+**Example 1: Thinking Mode (Default)**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3.6-35B-A3B-FP8",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    max_tokens=2048,
+    stream=True
+)
+
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        if delta.content:
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Example 2: Instruct Mode (Thinking Off)**
+
+To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3.6-35B-A3B-FP8",
+    messages=[
+        {"role": "user", "content": "What is 15% of 240?"}
+    ],
+    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
+    max_tokens=2048,
+    stream=True
+)
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+#### 4.3.2 Thinking Preservation
+
+Qwen3.6 has been trained to preserve and leverage thinking traces from historical messages. Enable this for agent scenarios where maintaining full reasoning context improves decision consistency:
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3.6-35B-A3B-FP8",
+    messages=[
+        {"role": "user", "content": "Help me plan a web app architecture."}
+    ],
+    extra_body={"chat_template_kwargs": {"preserve_thinking": True}},
+    max_tokens=2048,
+    stream=True
+)
+
+thinking_started = False
+has_thinking = False
+has_answer = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        if delta.content:
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+#### 4.3.3 Tool Calling
+
+Qwen3.6 supports tool calling capabilities. Enable the tool call parser during deployment.
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+response = client.chat.completions.create(
+    model="Qwen/Qwen3.6-35B-A3B-FP8",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    stream=True
+)
+
+thinking_started = False
+has_thinking = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                if tool_call.function:
+                    print(f"Tool Call: {tool_call.function.name}")
+                    print(f"   Arguments: {tool_call.function.arguments}")
+
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+print()
+```

docs_new/cookbook/autoregressive/Qwen/Qwen3.mdx

@@ -0,0 +1,884 @@
+---
+title: Qwen3
+metatags:
+    description: "Deploy Qwen3 series models with SGLang - featuring advanced reasoning, 256K context, and flexible Dense/MoE architectures for edge to cloud."
+---
+
+
+## 1. Model Introduction
+
+[Qwen3 series](https://github.com/QwenLM/Qwen3) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.
+
+This generation delivers comprehensive upgrades across the board:
+
+- **Stronger general intelligence**: Significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage.
+- **Broader multilingual knowledge**: Substantial gains in long-tail knowledge coverage across multiple languages.
+- **More helpful & aligned responses**: Markedly better alignment with user preferences in subjective and open-ended tasks, enabling higher-quality, more useful text generation.
+- **Extended context length**: Enhanced capabilities in understanding and reasoning over 256K-token long contexts.
+- **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
+- **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.
+
+For more details, please refer to the [official Qwen3 GitHub Repository](https://github.com/QwenLM/Qwen3).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs. The recommended launch configurations vary by hardware and model size.
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
+
+import { Qwen3Deployment } from "/src/snippets/autoregressive/qwen3-deployment.jsx";
+
+<Qwen3Deployment />
+
+### 3.2 Configuration Tips
+
+- **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
+- **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
+- **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused_moe_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
+- **Speculative Decoding**: Using Speculative Decoding for latency-sensitive scenarios.
+  - `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
+  - `--speculative-num-steps 3`: Number of speculative verification rounds
+  - `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
+  - `--speculative-num-draft-tokens 4`: Number of draft tokens per step
+  - `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan`](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan).
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+Qwen3-235B-A22B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-235B-A22B-Thinking-2507 \
+  --reasoning-parser qwen3 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-235B-A22B-Thinking-2507",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+
+Okay, so I need to figure out what 15% of 240 is. Hmm, percentages can sometimes trip me up, but I think I remember some basics. Let me start by recalling that "percent" means "per hundred," so 15% is the same as 15 per 100, or 15/100. So, maybe I can convert 15% into a decimal first? Yeah, I think that's a common method.
+...
+So conclusion: The answer is 36.
+
+=============== Content =================
+
+
+To determine what 15% of 240 is, we can follow a systematic approach that involves converting the percentage to a decimal and then performing multiplication. Here's a step-by-step breakdown of the solution:
+
+....
+
+### Final Answer:
+
+$$
+\boxed{36}
+$$
+
+Thus, 15% of 240 is **36**.
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.3 Tool Calling
+
+Qwen3 supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-235B-A22B-Thinking-2507 \
+  --reasoning-parser qwen3 \
+  --tool-call-parser qwen25 \
+  --tp 8 \
+  --host 0.0.0.0 \
+  --port 8000
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="Qwen/Qwen3-235B-A22B-Thinking-2507",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"🔧 Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+
+Okay, the user is asking for the weather in Beijing. Let me check the tools available. There's a function called get_weather that takes location and unit parameters. The location is required, so I need to specify Beijing as the location. The unit is optional and can be either celsius or fahrenheit. Since the user didn't specify the unit, maybe I should default to a common one. In China, they usually use celsius, so I'll set unit to celsius. I'll call the get_weather function with location: Beijing and unit: celsius. That should get the current weather for them.
+
+
+
+=============== Content =================
+
+🔧 Tool Call: get_weather
+   Arguments: {"location": "Beijing", "unit": "celsius"}
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="Qwen/Qwen3-235B-A22B-Thinking-2507",
+    messages=messages,
+    temperature=0.7
+)
+
+print(final_response.choices[0].message.content)
+# Output: "The current weather in Beijing is **22°C** and **sunny**. A perfect day to enjoy outdoor activities! 🌞"
+```
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (8x)
+- Model: Qwen3-235B-A22B-Instruct-2507
+- Tensor Parallelism: 8
+- sglang version: 0.5.6
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
+
+#### 5.1.1 Standard Scenario Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --tp 8
+```
+
+##### 5.1.1.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  43.56
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  4210
+Total generated tokens (retokenized):    4206
+Request throughput (req/s):              0.23
+Input token throughput (tok/s):          140.07
+Output token throughput (tok/s):         96.65
+Peak output token throughput (tok/s):    100.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          236.72
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   4353.63
+Median E2E Latency (ms):                 3475.79
+---------------Time to First Token----------------
+Mean TTFT (ms):                          99.03
+Median TTFT (ms):                        92.18
+P99 TTFT (ms):                           166.05
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.12
+Median TPOT (ms):                        10.12
+P99 TPOT (ms):                           10.15
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           10.13
+Median ITL (ms):                         10.12
+P95 ITL (ms):                            10.49
+P99 ITL (ms):                            10.70
+Max ITL (ms):                            13.45
+==================================================
+```
+
+##### 5.1.1.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  48.95
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  40725
+Total generated tokens (retokenized):    40716
+Request throughput (req/s):              1.63
+Input token throughput (tok/s):          810.44
+Output token throughput (tok/s):         832.04
+Peak output token throughput (tok/s):    1151.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          1642.48
+Concurrency:                             13.61
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   8326.72
+Median E2E Latency (ms):                 8827.86
+---------------Time to First Token----------------
+Mean TTFT (ms):                          215.70
+Median TTFT (ms):                        88.82
+P99 TTFT (ms):                           727.08
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          16.36
+Median TPOT (ms):                        16.12
+P99 TPOT (ms):                           24.09
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           15.96
+Median ITL (ms):                         14.52
+P95 ITL (ms):                            16.04
+P99 ITL (ms):                            67.69
+Max ITL (ms):                            457.52
+==================================================
+```
+
+##### 5.1.1.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  92.07
+Total input tokens:                      249831
+Total input text tokens:                 249831
+Total input vision tokens:               0
+Total generated tokens:                  252162
+Total generated tokens (retokenized):    251124
+Request throughput (req/s):              5.43
+Input token throughput (tok/s):          2713.46
+Output token throughput (tok/s):         2738.78
+Peak output token throughput (tok/s):    4400.00
+Peak concurrent requests:                110
+Total token throughput (tok/s):          5452.24
+Concurrency:                             90.50
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   16665.09
+Median E2E Latency (ms):                 16060.10
+---------------Time to First Token----------------
+Mean TTFT (ms):                          260.55
+Median TTFT (ms):                        122.68
+P99 TTFT (ms):                           863.11
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          32.94
+Median TPOT (ms):                        34.04
+P99 TPOT (ms):                           41.19
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           32.59
+Median ITL (ms):                         23.54
+P95 ITL (ms):                            69.79
+P99 ITL (ms):                            119.09
+Max ITL (ms):                            577.70
+==================================================
+```
+
+#### 5.1.2 Reasoning Scenario Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --tp 8
+```
+
+##### 5.1.2.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  457.45
+Total input tokens:                      6101
+Total input text tokens:                 6101
+Total input vision tokens:               0
+Total generated tokens:                  44452
+Total generated tokens (retokenized):    44059
+Request throughput (req/s):              0.02
+Input token throughput (tok/s):          13.34
+Output token throughput (tok/s):         97.17
+Peak output token throughput (tok/s):    100.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          110.51
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   45742.42
+Median E2E Latency (ms):                 49266.87
+---------------Time to First Token----------------
+Mean TTFT (ms):                          110.60
+Median TTFT (ms):                        109.36
+P99 TTFT (ms):                           167.43
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.23
+Median TPOT (ms):                        10.24
+P99 TPOT (ms):                           10.32
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           10.27
+Median ITL (ms):                         10.26
+P95 ITL (ms):                            10.71
+P99 ITL (ms):                            10.97
+Max ITL (ms):                            15.79
+==================================================
+```
+
+##### 5.1.2.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  340.17
+Total input tokens:                      39668
+Total input text tokens:                 39668
+Total input vision tokens:               0
+Total generated tokens:                  318226
+Total generated tokens (retokenized):    318104
+Request throughput (req/s):              0.24
+Input token throughput (tok/s):          116.61
+Output token throughput (tok/s):         935.49
+Peak output token throughput (tok/s):    1120.00
+Peak concurrent requests:                19
+Total token throughput (tok/s):          1052.10
+Concurrency:                             13.85
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   58885.30
+Median E2E Latency (ms):                 59238.70
+---------------Time to First Token----------------
+Mean TTFT (ms):                          169.71
+Median TTFT (ms):                        101.61
+P99 TTFT (ms):                           455.71
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          14.82
+Median TPOT (ms):                        14.91
+P99 TPOT (ms):                           15.20
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           14.76
+Median ITL (ms):                         14.63
+P95 ITL (ms):                            15.46
+P99 ITL (ms):                            16.62
+Max ITL (ms):                            104.94
+==================================================
+```
+
+##### 5.1.2.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 8000 \
+  --num-prompts 320 \
+  --max-concurrency 64
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  544.83
+Total input tokens:                      158939
+Total input text tokens:                 158939
+Total input vision tokens:               0
+Total generated tokens:                  1300705
+Total generated tokens (retokenized):    1293015
+Request throughput (req/s):              0.59
+Input token throughput (tok/s):          291.72
+Output token throughput (tok/s):         2387.34
+Peak output token throughput (tok/s):    3008.00
+Peak concurrent requests:                68
+Total token throughput (tok/s):          2679.06
+Concurrency:                             56.35
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   95937.70
+Median E2E Latency (ms):                 99362.32
+---------------Time to First Token----------------
+Mean TTFT (ms):                          265.03
+Median TTFT (ms):                        129.11
+P99 TTFT (ms):                           823.85
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          23.66
+Median TPOT (ms):                        24.07
+P99 TPOT (ms):                           24.97
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           23.54
+Median ITL (ms):                         23.07
+P95 ITL (ms):                            25.92
+P99 ITL (ms):                            63.87
+Max ITL (ms):                            408.30
+==================================================
+```
+
+#### 5.1.3 Summarization Scenario Benchmark
+
+##### 5.1.3.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  44.82
+Total input tokens:                      41941
+Total input text tokens:                 41941
+Total input vision tokens:               0
+Total generated tokens:                  4210
+Total generated tokens (retokenized):    4210
+Request throughput (req/s):              0.22
+Input token throughput (tok/s):          935.86
+Output token throughput (tok/s):         93.94
+Peak output token throughput (tok/s):    99.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          1029.80
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   4479.60
+Median E2E Latency (ms):                 3622.99
+---------------Time to First Token----------------
+Mean TTFT (ms):                          139.90
+Median TTFT (ms):                        114.85
+P99 TTFT (ms):                           225.17
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          10.31
+Median TPOT (ms):                        10.33
+P99 TPOT (ms):                           10.51
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           10.33
+Median ITL (ms):                         10.33
+P95 ITL (ms):                            10.73
+P99 ITL (ms):                            10.93
+Max ITL (ms):                            14.48
+==================================================
+```
+
+##### 5.1.3.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  50.68
+Total input tokens:                      300020
+Total input text tokens:                 300020
+Total input vision tokens:               0
+Total generated tokens:                  41589
+Total generated tokens (retokenized):    41578
+Request throughput (req/s):              1.58
+Input token throughput (tok/s):          5920.41
+Output token throughput (tok/s):         820.69
+Peak output token throughput (tok/s):    1200.00
+Peak concurrent requests:                20
+Total token throughput (tok/s):          6741.10
+Concurrency:                             13.90
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   8805.54
+Median E2E Latency (ms):                 9368.79
+---------------Time to First Token----------------
+Mean TTFT (ms):                          284.29
+Median TTFT (ms):                        168.48
+P99 TTFT (ms):                           1027.21
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          16.81
+Median TPOT (ms):                        16.66
+P99 TPOT (ms):                           27.18
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           16.42
+Median ITL (ms):                         13.68
+P95 ITL (ms):                            17.23
+P99 ITL (ms):                            90.75
+Max ITL (ms):                            574.64
+==================================================
+```
+
+##### 5.1.3.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
+  --dataset-name random \
+  --random-input-len 8000 \
+  --random-output-len 1000 \
+  --num-prompts 320 \
+  --max-concurrency 64
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 64
+Successful requests:                     320
+Benchmark duration (s):                  94.77
+Total input tokens:                      1273893
+Total input text tokens:                 1273893
+Total input vision tokens:               0
+Total generated tokens:                  169680
+Total generated tokens (retokenized):    169640
+Request throughput (req/s):              3.38
+Input token throughput (tok/s):          13441.86
+Output token throughput (tok/s):         1790.43
+Peak output token throughput (tok/s):    2687.00
+Peak concurrent requests:                70
+Total token throughput (tok/s):          15232.28
+Concurrency:                             58.63
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   17364.14
+Median E2E Latency (ms):                 17495.95
+---------------Time to First Token----------------
+Mean TTFT (ms):                          238.22
+Median TTFT (ms):                        203.27
+P99 TTFT (ms):                           510.48
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          32.50
+Median TPOT (ms):                        34.27
+P99 TPOT (ms):                           40.59
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           32.36
+Median ITL (ms):                         22.50
+P95 ITL (ms):                            97.81
+P99 ITL (ms):                            151.55
+Max ITL (ms):                            352.79
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+
+- **Results**:
+
+  - Qwen/Qwen3-235B-A22B-Instruct-2507
+    ```text Output
+    Accuracy: 0.945
+    Invalid: 0.000
+    Latency: 11.980 s
+    Output throughput: 2358.105 token/s
+    ```

docs_new/cookbook/autoregressive/StepFun/Step3-VL-10B.mdx

@@ -0,0 +1,695 @@
+---
+title: Step3-VL-10B
+metatags:
+    description: "Deploy Step3-VL-10B multimodal model with SGLang - compact 10B dense model with frontier-level vision understanding, complex reasoning, and tool calling capabilities."
+tag: NEW
+---
+
+import { Step3VL10BDeployment } from '/src/snippets/autoregressive/step-3vl-10b-deployment.jsx';
+
+## 1. Model Introduction
+
+[Step3-VL-10B](https://huggingface.co/stepfun-ai/Step3-VL-10B) is a lightweight open-source multimodal model developed by StepFun, designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence. Despite its compact 10B parameter footprint, Step3-VL-10B excels in visual perception, complex reasoning, and human-centric alignment.
+
+Key highlights of Step3-VL-10B include:
+
+- **STEM Reasoning**: Achieves 94.43% on AIME 2025 and 75.95% on MathVision (with PaCoRe), demonstrating exceptional complex reasoning capabilities that outperform models 10×–20× larger.
+- **Visual Perception**: Records 92.05% on MMBench and 80.11% on MMMU, establishing strong general visual understanding and multimodal reasoning.
+- **GUI & OCR**: Delivers state-of-the-art performance on ScreenSpot-V2 (92.61%), ScreenSpot-Pro (51.55%), and OCRBench (86.75%), optimized for agentic and document understanding tasks.
+- **Spatial Understanding**: Demonstrates emergent spatial awareness with 66.79% on BLINK and 57.21% on All-Angles-Bench, establishing strong potential for embodied intelligence applications.
+
+For more details, please refer to the [Step3-VL-10B model card on Hugging Face](https://huggingface.co/stepfun-ai/Step3-VL-10B).
+
+## 2. SGLang Installation
+
+SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
+
+Please refer to the [official SGLang installation guide](../../../docs/get-started/installation) for installation instructions.
+
+## 3. Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+Step3-VL-10B is a compact 10B dense model that can run on a single GPU. Recommended starting configurations vary depending on hardware.
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and quantization method. SGLang supports serving Step3-VL-10B on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs.
+
+<Step3VL10BDeployment />
+
+### 3.2 Configuration Tips
+
+- **Single GPU Deployment**: Step3-VL-10B fits comfortably on a single GPU with BF16 precision, no tensor parallelism required.
+- **Memory Management**: Set lower `--context-length` to conserve memory if needed. A value of `32768` is sufficient for most scenarios.
+- **FP8 Quantization**: Use FP8 quantization to further reduce memory usage while maintaining quality.
+
+## 4. Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Multi-Modal Inputs
+
+Step3-VL-10B supports image inputs. Here's a basic example with image input:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Read all the text in the image."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="stepfun-ai/Step3-VL-10B",
+    messages=messages,
+    max_tokens=2048,
+    extra_body={"top_k": -1}
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example output:**
+
+```text Output
+Response costs: 5.89s
+Generated text: Auntie Anne's
+
+CINNAMON SUGAR
+1 × 17,000               17,000
+
+SUB TOTAL                    17,000
+
+GRAND TOTAL                 17,000
+
+CASH IDR                    20,000
+
+CHANGE DUE                 3,000
+```
+
+**Multi-Image Input Example:**
+
+Step3-VL-10B can process multiple images in a single request for comparison or analysis:
+
+```python Example
+import time
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="EMPTY",
+    base_url="http://localhost:30000/v1",
+    timeout=3600
+)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
+                }
+            },
+            {
+                "type": "image_url",
+                "image_url": {
+                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
+                }
+            },
+            {
+                "type": "text",
+                "text": "Compare these two images and describe the differences in 100 words or less."
+            }
+        ]
+    }
+]
+
+start = time.time()
+response = client.chat.completions.create(
+    model="stepfun-ai/Step3-VL-10B",
+    messages=messages,
+    max_tokens=2048,
+    extra_body={"top_k": -1}
+)
+print(f"Response costs: {time.time() - start:.2f}s")
+print(f"Generated text: {response.choices[0].message.content}")
+```
+
+**Example Output:**
+
+```text Output
+Response costs: 3.24s
+Generated text: First image: Single red Hong Kong taxi close - up, clear license plate (RX 5004), “4 SEATS” sticker, urban street with shops behind. Second image: Aerial view of many taxis (red, green) on a highway with a viaduct, some hoods open, dense arrangement. Differences: Scale (single vs many), perspective (close - up vs aerial), context (street shops vs highway), and taxi conditions (normal vs some open hoods).
+```
+
+#### 4.2.2 Reasoning Parser
+
+Step3-VL-10B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+python -m sglang.launch_server \
+  --model stepfun-ai/Step3-VL-10B \
+  --reasoning-parser deepseek-r1 \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --trust-remote-code
+```
+
+**Streaming with Thinking Process:**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="stepfun-ai/Step3-VL-10B",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True,
+    extra_body={"top_k": -1}
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Example Output:**
+```text Output
+=============== Thinking =================
+Okay, let's see. The problem is asking for 15% of 240. Hmm, I need to remember how to calculate percentages. So, percentage means "per hundred," right? So, 15% is the same as 15 per 100, or 15/100. To find a percentage of a number, I think you convert the percentage to a decimal and then multiply it by the number. Let me check that.
+
+First, 15% as a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Yeah, that's right. So 15% is 0.15 in decimal form. Then, to find 15% of 24
+0, I need to multiply 0.15 by 240. Let me do that calculation.
+
+Let me write it out: 0.15 * 240. Let's compute that. Maybe break it down. 0.1 is 10%, and 0.05 is 5%, so 10% of 240 is 24, and 5% of 240 is 12. Then 10% + 5% is 15%, so 24 + 12 is 36. Oh,
+ that's a good way to check. So 15% is 10% plus 5%, which adds up to 36. Let me verify with the decimal method. 0.15 * 240. Let's multiply 240 by 0.1 first: 24. Then 240 by 0.05: 12. Adding them gives 36. Yep, that matches. Alternatively, 240 * 15 = 3600, and then divide by 100 (since it's per hundred), so 3600 / 100 = 36. That's another way. So 15% of 240 is 36. Let me make sure I didn't make a mistake. Let's check with another method. 10% of 240 is 24, 20% would be 48, so 15% is halfway between 10% and 20%, which is (24 + 48)/2 = 36. Yep, that works too
+. So all methods point to 36. I think that's the answer.
+
+=============== Content =================
+
+To solve the problem "What is 15% of 240?" step by step:
+
+---
+
+### **Step 1: Understand the concept of percentage**
+A percentage represents a portion of a whole. Specifically, "percent" means "per hundred." So, 15% means **15 out of 100**, or **15/100**.
+
+---
+
+### **Step 2: Convert the percentage to a decimal**
+To use percentages in calculations, convert them to decimals by dividing by 100:
+$$
+15\% = \frac{15}{100} = 0.15
+$$
+
+---
+
+### **Step 3: Multiply the decimal by the given number**
+Now, multiply 0.15 (the decimal form of 15%) by 240:
+$$
+0.15 \times 240 = 36
+$$
+
+---
+
+### **Alternative Verification Methods**
+
+#### **Method A: Break into parts**
+- 10% of 240 = $ 0.10 \times 240 = 24 $
+- 5% of 240 = $ 0.05 \times 240 = 12 $
+- Add them: $ 24 + 12 = 36 $
+
+#### **Method B: Use direct multiplication**
+- $ 15\% \text{ of } 240 = \frac{15}{100} \times 240 = \frac{3600}{100} = 36 $
+
+#### **Method C: Estimate using known percentages**
+- 20% of 240 = $ 0.20 \times 240 = 48 $
+- 10% of 240 = $ 0.10 \times 240 = 24 $
+- 15% is halfway between 10% and 20%: $ \frac{24 + 48}{2} = 36 $
+
+---
+
+### **Final Answer**
+$$
+\boxed{36}
+$$
+```
+
+**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
+
+#### 4.2.3 Tool Calling
+
+Step3-VL-10B supports tool calling capabilities. Enable the tool call parser:
+
+```shell Command
+python -m sglang.launch_server \
+  --model stepfun-ai/Step3-VL-10B \
+  --reasoning-parser deepseek-r1 \
+  --tool-call-parser hermes \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --trust-remote-code
+```
+
+**Python Example (with Thinking Process):**
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Define available tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city name"
+                    },
+                    "unit": {
+                        "type": "string",
+                        "enum": ["celsius", "fahrenheit"],
+                        "description": "Temperature unit"
+                    }
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# Make request with streaming to see thinking process
+response = client.chat.completions.create(
+    model="stepfun-ai/Step3-VL-10B",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=0.7,
+    stream=True,
+    extra_body={"top_k": -1}
+)
+
+# Process streaming response
+thinking_started = False
+has_thinking = False
+tool_calls_accumulator = {}
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Accumulate tool calls
+        if hasattr(delta, 'tool_calls') and delta.tool_calls:
+            # Close thinking section if needed
+            if has_thinking and thinking_started:
+                print("\n=============== Content =================\n", flush=True)
+                thinking_started = False
+
+            for tool_call in delta.tool_calls:
+                index = tool_call.index
+                if index not in tool_calls_accumulator:
+                    tool_calls_accumulator[index] = {
+                        'name': None,
+                        'arguments': ''
+                    }
+
+                if tool_call.function:
+                    if tool_call.function.name:
+                        tool_calls_accumulator[index]['name'] = tool_call.function.name
+                    if tool_call.function.arguments:
+                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
+
+        # Print content
+        if delta.content:
+            print(delta.content, end="", flush=True)
+
+# Print accumulated tool calls
+for index, tool_call in sorted(tool_calls_accumulator.items()):
+    print(f"Tool Call: {tool_call['name']}")
+    print(f"   Arguments: {tool_call['arguments']}")
+
+print()
+```
+
+**Example Output:**
+```text Output
+=============== Thinking =================
+The user is asking about the weather in Beijing. I have a function called "get_weather" that can provide weather information for a location. Let me check the parameters:
+
+- location: required (string) - "Beijing"
+- unit: optional (string, enum: ["celsius", "fahrenheit"]) - not specified by the user, so I won't include it
+
+I should call the function with location="Beijing".
+
+<tool_calls>
+
+=============== Content =================
+
+</tool_calls>Tool Call: get_weather
+   Arguments: {"location": "Beijing"}
+```
+
+**Handling Tool Call Results:**
+
+```python Example
+# After getting the tool call, execute the function
+def get_weather(location, unit="celsius"):
+    # Your actual weather API call here
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# Send tool result back to the model
+messages = [
+    {"role": "user", "content": "What's the weather in Beijing?"},
+    {
+        "role": "assistant",
+        "content": None,
+        "tool_calls": [{
+            "id": "call_123",
+            "type": "function",
+            "function": {
+                "name": "get_weather",
+                "arguments": '{"location": "Beijing", "unit": "celsius"}'
+            }
+        }]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "call_123",
+        "content": get_weather("Beijing", "celsius")
+    }
+]
+
+final_response = client.chat.completions.create(
+    model="stepfun-ai/Step3-VL-10B",
+    messages=messages,
+    temperature=0.7,
+    extra_body={"top_k": -1}
+)
+
+print(final_response.choices[0].message.content)
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA B200 GPU (1x)
+- Model: stepfun-ai/Step3-VL-10B
+- Tensor Parallelism: 1
+- sglang version: 0.5.8+
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images.
+
+#### 5.1.1 Latency-Sensitive Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model stepfun-ai/Step3-VL-10B \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --trust-remote-code
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model stepfun-ai/Step3-VL-10B \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Result:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  30.85
+Total input tokens:                      14120
+Total input text tokens:                 720
+Total input vision tokens:               13400
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4217
+Request throughput (req/s):              0.32
+Input token throughput (tok/s):          457.71
+Output token throughput (tok/s):         136.79
+Peak output token throughput (tok/s):    240.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          594.50
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3083.40
+Median E2E Latency (ms):                 2747.00
+P90 E2E Latency (ms):                    4574.50
+P99 E2E Latency (ms):                    5462.49
+---------------Time to First Token----------------
+Mean TTFT (ms):                          1327.69
+Median TTFT (ms):                        1341.01
+P99 TTFT (ms):                           1486.11
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          4.16
+Median TPOT (ms):                        4.17
+P99 TPOT (ms):                           4.18
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           4.17
+Median ITL (ms):                         4.18
+P95 ITL (ms):                            4.30
+P99 ITL (ms):                            4.38
+Max ITL (ms):                            8.24
+==================================================
+```
+
+#### 5.1.2 Throughput-Sensitive Benchmark
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang-oai-chat \
+  --host 127.0.0.1 \
+  --port 30000 \
+  --model stepfun-ai/Step3-VL-10B \
+  --dataset-name image \
+  --image-count 2 \
+  --image-resolution 720p \
+  --random-input-len 128 \
+  --random-output-len 1024 \
+  --num-prompts 1000 \
+  --max-concurrency 100
+```
+
+- Result:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     1000
+Benchmark duration (s):                  976.52
+Total input tokens:                      1416949
+Total input text tokens:                 76949
+Total input vision tokens:               1340000
+Total generated tokens:                  510855
+Total generated tokens (retokenized):    510526
+Request throughput (req/s):              1.02
+Input token throughput (tok/s):          1451.02
+Output token throughput (tok/s):         523.14
+Peak output token throughput (tok/s):    20429.00
+Peak concurrent requests:                103
+Total token throughput (tok/s):          1974.16
+Concurrency:                             99.81
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   97463.22
+Median E2E Latency (ms):                 91872.75
+P90 E2E Latency (ms):                    118553.42
+P99 E2E Latency (ms):                    198445.56
+---------------Time to First Token----------------
+Mean TTFT (ms):                          94379.07
+Median TTFT (ms):                        87163.09
+P99 TTFT (ms):                           194871.41
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          5.89
+Median TPOT (ms):                        5.72
+P99 TPOT (ms):                           23.58
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           6.05
+Median ITL (ms):                         0.13
+P95 ITL (ms):                            0.56
+P99 ITL (ms):                            3.99
+Max ITL (ms):                            97551.06
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 MMMU Benchmark
+
+You can evaluate the model's accuracy using the MMMU dataset:
+
+- Model Deployment Command:
+
+```shell Command
+python -m sglang.launch_server \
+  --model stepfun-ai/Step3-VL-10B \
+  --host 0.0.0.0 \
+  --port 30000 \
+  --trust-remote-code
+```
+
+- Benchmark Command:
+
+```shell Command
+python3 benchmark/mmmu/bench_sglang.py \
+    --port 30000 \
+    --concurrency 64
+```
+
+- Result:
+
+```text Output
+Benchmark time: 934.6179109360091
+answers saved to: ./answer_sglang.json
+Evaluating...
+answers saved to: ./answer_sglang.json
+{'Accounting': {'acc': 0.667, 'num': 30},
+ 'Agriculture': {'acc': 0.367, 'num': 30},
+ 'Architecture_and_Engineering': {'acc': 0.4, 'num': 30},
+ 'Art': {'acc': 0.467, 'num': 30},
+ 'Art_Theory': {'acc': 0.5, 'num': 30},
+ 'Basic_Medical_Science': {'acc': 0.367, 'num': 30},
+ 'Biology': {'acc': 0.3, 'num': 30},
+ 'Chemistry': {'acc': 0.467, 'num': 30},
+ 'Clinical_Medicine': {'acc': 0.567, 'num': 30},
+ 'Computer_Science': {'acc': 0.467, 'num': 30},
+ 'Design': {'acc': 0.567, 'num': 30},
+ 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.3, 'num': 30},
+ 'Economics': {'acc': 0.6, 'num': 30},
+ 'Electronics': {'acc': 0.567, 'num': 30},
+ 'Energy_and_Power': {'acc': 0.633, 'num': 30},
+ 'Finance': {'acc': 0.733, 'num': 30},
+ 'Geography': {'acc': 0.333, 'num': 30},
+ 'History': {'acc': 0.533, 'num': 30},
+ 'Literature': {'acc': 0.533, 'num': 30},
+ 'Manage': {'acc': 0.6, 'num': 30},
+ 'Marketing': {'acc': 0.767, 'num': 30},
+ 'Materials': {'acc': 0.6, 'num': 30},
+ 'Math': {'acc': 0.7, 'num': 30},
+ 'Mechanical_Engineering': {'acc': 0.333, 'num': 30},
+ 'Music': {'acc': 0.4, 'num': 30},
+ 'Overall': {'acc': 0.523, 'num': 900},
+ 'Overall-Art and Design': {'acc': 0.483, 'num': 120},
+ 'Overall-Business': {'acc': 0.673, 'num': 150},
+ 'Overall-Health and Medicine': {'acc': 0.513, 'num': 150},
+ 'Overall-Humanities and Social Science': {'acc': 0.492, 'num': 120},
+ 'Overall-Science': {'acc': 0.5, 'num': 150},
+ 'Overall-Tech and Engineering': {'acc': 0.481, 'num': 210},
+ 'Pharmacy': {'acc': 0.6, 'num': 30},
+ 'Physics': {'acc': 0.7, 'num': 30},
+ 'Psychology': {'acc': 0.467, 'num': 30},
+ 'Public_Health': {'acc': 0.733, 'num': 30},
+ 'Sociology': {'acc': 0.433, 'num': 30}}
+eval out saved to ./val_sglang.json
+Overall accuracy: 0.523
+```

docs_new/cookbook/autoregressive/StepFun/Step3.5.mdx

@@ -0,0 +1,529 @@
+---
+title: Step-3.5
+metatags:
+    description: "Deploy Step-3.5 reasoning engine with SGLang. "
+tag: NEW
+---
+
+import { Step35Deployment } from '/src/snippets/autoregressive/step-35-deployment.jsx';
+
+## 1. Model Introduction
+
+[Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is StepFun's production-grade reasoning engine built to decouple elite intelligence from heavy compute, and cuts attention cost for low-latency, cost-effective long-context inference—purpose-built for autonomous agents in real-world workflows. The model is available in multiple quantization formats optimized for different hardware platforms.
+
+This generation delivers comprehensive upgrades across the board:
+- **Hybrid Attention Architecture**:  Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio and an aggressive 128-token window. This hybrid approach ensures consistent performance across massive datasets or long codebases while significantly reducing the computational overhead typical of standard long-context models.
+- **Sparse Mixture-of-Experts**: Only 11B active parameters out of 196B parameters.
+- **Multi-Layer Multi-Token Prediction (MTP)**: Equipped with a  3-way Multi-Token Prediction (MTP-3). This allows for complex, multi-step reasoning chains with immediate responsiveness.
+
+## 2.SGLang Installation
+
+Step-3.5-Flash is currently available in SGLang via Docker image install.
+
+### Docker (NVIDIA)
+```bash Command
+# Pull the docker image
+docker pull lmsysorg/sglang:dev-pr-18084
+
+# Launch the container
+docker run -it --gpus all \
+  --shm-size=32g \
+  --ipc=host \
+  --network=host \
+  lmsysorg/sglang:dev-pr-18084 bash
+```
+
+### Docker (AMD ROCm)
+```bash Command
+# For MI300X/MI325X
+docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
+
+# For MI350X/MI355X
+docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
+
+docker run -it \
+  --device=/dev/kfd --device=/dev/dri \
+  --shm-size=32g \
+  --ipc=host \
+  --network=host \
+  --group-add video --cap-add=SYS_PTRACE \
+  --security-opt seccomp=unconfined \
+  lmsysorg/sglang:v0.5.9-rocm700-mi30x bash  # or mi35x for MI350X/MI355X
+```
+
+## 3.Model Deployment
+
+This section provides deployment configurations optimized for different hardware platforms and use cases.
+
+### 3.1 Basic Configuration
+
+The Step-3.5-Flash series comes in only one sizes. Recommended starting configurations vary depending on hardware.
+
+**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.
+
+<Step35Deployment />
+
+### 3.2 Configuration Tips
+
+- **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4×, TP=4), MI300X/MI325X/MI350X/MI355X (4×, TP=4 EP=4).
+- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X and `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X.
+- **AMD Expert Parallelism Required**: On AMD GPUs, always use `--ep 4` with `--tp 4`. Both BF16 and FP8 models require expert parallelism. Without EP, the MoE intermediate dimension is split across GPUs (N=320), which triggers an AITER CK GEMM incompatibility. With EP=4, each GPU handles 72 full experts (N=1280), which works correctly with cuda graph enabled.
+- **AITER JIT Compilation**: First inference on AMD may take 30-40 seconds for AITER kernel JIT compilation. Subsequent requests use cached kernels.
+
+## 4.Model Invocation
+
+### 4.1 Basic Usage
+
+For basic API usage and request examples, please refer to:
+
+- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
+
+### 4.2 Advanced Usage
+
+#### 4.2.1 Reasoning Parser
+
+Step-3.5-Flash only supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
+
+```shell Command
+sglang serve \
+  --model-path stepfun-ai/Step-3.5-Flash \
+  --tp 4 \
+  --ep 4 \
+  --reasoning-parser step3p5
+```
+
+```python Example
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# Enable streaming to see the thinking process in real-time
+response = client.chat.completions.create(
+    model="stepfun-ai/Step-3.5-Flash",
+    messages=[
+        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
+    ],
+    temperature=0.7,
+    max_tokens=2048,
+    stream=True
+)
+
+# Process the stream
+has_thinking = False
+has_answer = False
+thinking_started = False
+
+for chunk in response:
+    if chunk.choices and len(chunk.choices) > 0:
+        delta = chunk.choices[0].delta
+
+        # Print thinking process
+        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
+            if not thinking_started:
+                print("=============== Thinking =================", flush=True)
+                thinking_started = True
+            has_thinking = True
+            print(delta.reasoning_content, end="", flush=True)
+
+        # Print answer content
+        if delta.content:
+            # Close thinking section and add content header
+            if has_thinking and not has_answer:
+                print("\n=============== Content =================", flush=True)
+                has_answer = True
+            print(delta.content, end="", flush=True)
+
+print()
+```
+
+**Output Example:**
+
+```text Output
+=============== Thinking =================
+We are asked: "What is 15% of 240?" We need to solve step by step.
+
+Step 1: Understand that "15% of 240" means we need to calculate 15 percent of 240. In mathematical terms, it is (15/100) * 240.
+
+Step 2: Simplify the calculation. We can compute 15% of 240 by first finding 10% of 240 and then 5% of 240, and adding them. Alternatively, we can multiply directly.
+
+Method 1:
+10% of 240 = 240 * 0.10 = 24.
+5% is half of 10%, so 5% of 240 = 24 / 2 = 12.
+Then 15% = 10% + 5% = 24 + 12 = 36.
+
+Method 2: Direct multiplication: 15% = 15/100 = 0.15, so 0.15 * 240 = 36.
+
+We can also compute fractionally: (15/100)*240 = (15*240)/100. 15*240 = 3600, divided by 100 gives 36.
+
+Thus, the answer is 36.
+
+We'll present the solution step by step.
+
+=============== Content =================
+
+To find 15% of 240, follow these steps:
+
+1. **Convert the percentage to a decimal**:
+   \( 15\% = \frac{15}{100} = 0.15 \)
+
+2. **Multiply by the number**:
+   \( 0.15 \times 240 = 36 \)
+
+Alternatively, break it down:
+- \( 10\% \text{ of } 240 = 240 \times 0.10 = 24 \)
+- \( 5\% \text{ of } 240 = \frac{24}{2} = 12 \) (since 5% is half of 10%)
+- \( 15\% = 10\% + 5\% = 24 + 12 = 36 \)
+
+**Answer:** 36
+```
+
+#### 4.2.2 Tool Calling
+
+Step-3.5 supports tool calling capabilities. Enable the tool call parser:
+
+**Python Example:**
+
+Start sglang server:
+
+```shell Command
+sglang serve \
+  --model-path stepfun-ai/Step-3.5-Flash \
+  --tp 4 \
+  --ep 4 \
+  --reasoning-parser step3p5 \
+  --tool-call-parser step3p5
+```
+
+```python Example
+from openai import OpenAI
+import json
+
+client = OpenAI(
+    base_url="http://localhost:30000/v1",
+    api_key="EMPTY"
+)
+
+# 1. define tools
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get the current weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {"type": "string", "description": "The city name"},
+                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"}
+                },
+                "required": ["location"]
+            }
+        }
+    }
+]
+
+# 2. tool run
+def get_weather(location, unit="celsius"):
+    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
+
+# 3. send first request
+print("--- Sending first request ---")
+response = client.chat.completions.create(
+    model="stepfun-ai/Step-3.5-Flash",
+    messages=[
+        {"role": "user", "content": "What's the weather in Beijing?"}
+    ],
+    tools=tools,
+    temperature=1.0,
+    stream=False
+)
+
+message = response.choices[0].message
+
+# 4. Handle Reasoning Content
+reasoning = getattr(message, 'reasoning_content', None)
+if reasoning:
+    print("=============== Thinking =================")
+    print(reasoning)
+    print("==========================================")
+
+# 5. Handle Tool Calls
+if message.tool_calls:
+    print("\n🔧 Tool Calls detected:")
+    history_messages = [
+        {"role": "user", "content": "What's the weather in Beijing?"},
+        message
+    ]
+
+    for tool_call in message.tool_calls:
+        print(f"   Tool: {tool_call.function.name}")
+        print(f"   Args: {tool_call.function.arguments}")
+
+        args = json.loads(tool_call.function.arguments)
+        tool_result = get_weather(args.get("location"), args.get("unit", "celsius"))
+
+        history_messages.append({
+            "role": "tool",
+            "tool_call_id": tool_call.id,
+            "content": tool_result
+        })
+
+    print("\n--- Sending tool results ---")
+    final_response = client.chat.completions.create(
+        model="stepfun-ai/Step-3.5-Flash",
+        messages=history_messages,
+        temperature=1.0,
+        stream=False
+    )
+
+    print("=============== Final Content =================")
+    print(final_response.choices[0].message.content)
+
+else:
+    if message.content:
+        print("=============== Content =================")
+        print(message.content)
+```
+
+**Output Example:**
+
+```text Output
+--- Sending first request ---
+=============== Thinking =================
+The user is asking for the weather in Beijing. I should use the get_weather function with location="Beijing". The unit parameter is optional and the user didn't specify a preference, so I'll leave it out (the default should be fine).
+
+==========================================
+
+🔧 Tool Calls detected:
+   Tool: get_weather
+   Args: {"location": "Beijing"}
+
+--- Sending tool results ---
+=============== Final Content =================
+The weather in Beijing is 22°C and sunny.
+```
+
+**Note:**
+
+- The reasoning parser shows how the model decides to use a tool
+- Tool calls are clearly marked with the function name and arguments
+- You can then execute the function and send the result back to continue the conversation
+
+## 5. Benchmark
+
+### 5.1 Speed Benchmark
+
+**Test Environment:**
+
+- Hardware: NVIDIA H200 GPU (4x)
+- Model: Step-3.5-Flash
+- Tensor Parallelism: 4
+- Expert Parallelism: 4
+- sglang version: 0.5.8
+
+We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
+
+#### 5.1.1 Standard Scenario Benchmark
+
+- Model Deployment Command:
+
+```shell Command
+sglang serve \
+  --model-path stepfun-ai/Step-3.5-Flash \
+  --tp 4 \
+  --ep 4
+```
+
+##### 5.1.1.1 Low Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model stepfun-ai/Step-3.5-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 10 \
+  --max-concurrency 1
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 1
+Successful requests:                     10
+Benchmark duration (s):                  35.30
+Total input tokens:                      6091
+Total input text tokens:                 6091
+Total generated tokens:                  4220
+Total generated tokens (retokenized):    4212
+Request throughput (req/s):              0.28
+Input token throughput (tok/s):          172.57
+Output token throughput (tok/s):         119.56
+Peak output token throughput (tok/s):    124.00
+Peak concurrent requests:                2
+Total token throughput (tok/s):          292.14
+Concurrency:                             1.00
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   3527.94
+Median E2E Latency (ms):                 2884.72
+P90 E2E Latency (ms):                    6350.38
+P99 E2E Latency (ms):                    7858.53
+---------------Time to First Token----------------
+Mean TTFT (ms):                          107.53
+Median TTFT (ms):                        80.93
+P99 TTFT (ms):                           269.52
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          8.12
+Median TPOT (ms):                        8.13
+P99 TPOT (ms):                           8.14
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           8.12
+Median ITL (ms):                         8.11
+P95 ITL (ms):                            8.61
+P99 ITL (ms):                            8.91
+Max ITL (ms):                            20.77
+==================================================
+```
+
+##### 5.1.1.2 Medium Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model stepfun-ai/Step-3.5-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 80 \
+  --max-concurrency 16
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 16
+Successful requests:                     80
+Benchmark duration (s):                  54.06
+Total input tokens:                      39588
+Total input text tokens:                 39588
+Total generated tokens:                  40805
+Total generated tokens (retokenized):    40479
+Request throughput (req/s):              1.48
+Input token throughput (tok/s):          732.33
+Output token throughput (tok/s):         754.84
+Peak output token throughput (tok/s):    928.00
+Peak concurrent requests:                21
+Total token throughput (tok/s):          1487.17
+Concurrency:                             14.06
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   9501.23
+Median E2E Latency (ms):                 10010.71
+P90 E2E Latency (ms):                    15655.09
+P99 E2E Latency (ms):                    18803.63
+---------------Time to First Token----------------
+Mean TTFT (ms):                          198.34
+Median TTFT (ms):                        89.50
+P99 TTFT (ms):                           984.66
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          18.97
+Median TPOT (ms):                        18.80
+P99 TPOT (ms):                           35.67
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           18.27
+Median ITL (ms):                         17.48
+P95 ITL (ms):                            18.44
+P99 ITL (ms):                            62.47
+Max ITL (ms):                            460.85
+==================================================
+```
+
+##### 5.1.1.3 High Concurrency
+
+- Benchmark Command:
+
+```shell Command
+python3 -m sglang.bench_serving \
+  --backend sglang \
+  --model stepfun-ai/Step-3.5-Flash \
+  --dataset-name random \
+  --random-input-len 1000 \
+  --random-output-len 1000 \
+  --num-prompts 500 \
+  --max-concurrency 100
+```
+
+- Test Results:
+
+```text Output
+============ Serving Benchmark Result ============
+Backend:                                 sglang
+Traffic request rate:                    inf
+Max request concurrency:                 100
+Successful requests:                     500
+Benchmark duration (s):                  125.88
+Total input tokens:                      249331
+Total input text tokens:                 249331
+Total generated tokens:                  252662
+Total generated tokens (retokenized):    251323
+Request throughput (req/s):              3.97
+Input token throughput (tok/s):          1980.77
+Output token throughput (tok/s):         2007.23
+Peak output token throughput (tok/s):    2500.00
+Peak concurrent requests:                109
+Total token throughput (tok/s):          3987.99
+Concurrency:                             92.25
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   23223.31
+Median E2E Latency (ms):                 22631.90
+P90 E2E Latency (ms):                    42269.38
+P99 E2E Latency (ms):                    47637.53
+---------------Time to First Token----------------
+Mean TTFT (ms):                          372.13
+Median TTFT (ms):                        127.26
+P99 TTFT (ms):                           1880.42
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          46.06
+Median TPOT (ms):                        47.61
+P99 TPOT (ms):                           51.34
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           45.31
+Median ITL (ms):                         39.86
+P95 ITL (ms):                            72.49
+P99 ITL (ms):                            117.05
+Max ITL (ms):                            1359.81
+==================================================
+```
+
+### 5.2 Accuracy Benchmark
+
+#### 5.2.1 GSM8K Benchmark
+
+- **Benchmark Command:**
+
+```shell Command
+python3 -m sglang.test.few_shot_gsm8k --num-questions 200
+```
+
+- **Results**:
+
+  - Step-3.5-Flash
+    ```
+    Accuracy: 0.885
+    Invalid: 0.005
+    Latency: 9.986 s
+    Output throughput: 1972.911 token/s
+    ```

docs_new/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash.mdx

@@ -0,0 +1,106 @@
+---
+title: MiMo-V2-Flash
+metatags:
+    description: "Deploy MiMo-V2-Flash 309B MoE model with SGLang - hybrid attention, multi-token prediction, and 256K context for efficient inference."
+---
+
+## Introduction
+
+XiaomiMiMo/MiMo-V2-Flash, with 309B total parameters and 15B activated parameters, is a new inference-centric model designed to maximize decoding efficiency created by XiaomiMiMo Team explicitly co-designed for real-world serving workloads, enabling flexible tradeoffs between throughput and latency on different hardware.
+
+This model creates a new balance between long-context modeling capability and inference efficiency. Key features include:
+- **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and an aggressive 128-token window. This reduces KV-cache storage by nearly 6x while maintaining long-context performance via learnable attention sink bias.
+- **Multi-Token Prediction (MTP)**: Equipped with a lightweight MTP module (0.33B params/block) using dense FFNs. This triples output speed during inference and will be good to accelerates rollout in RL training.
+- **Efficient Pre-Training**: Trained on 27T tokens using FP8 mixed precision and native 32k seq length. The context window supports up to 256k length.
+- **Agentic Capabilities**: Post-training utilizes Multi-Teacher On-Policy Distillation (MOPD) and large-scale agentic RL, achieving superior performance on SWE-Bench and complex reasoning tasks.
+
+
+## Installation
+
+MiMo-V2-Flash is currently available in SGLang via Docker image and pip install.
+
+### Docker
+
+```bash Command
+# Pull the docker image
+docker pull lmsysorg/sglang:dev-pr-15207
+
+# Launch the container
+docker run -it --gpus all \
+  --shm-size=32g \
+  --ipc=host \
+  --network=host \
+  lmsysorg/sglang:dev-pr-15207 bash
+```
+
+### Pip Installation
+
+```bash Command
+# On a machine with SGLang dependencies installed or inside a SGLang nightly container
+# Start an SGLang nightly container
+docker run -it --gpus all \
+  --shm-size=32g \
+  --ipc=host \
+  --network=host \
+  lmsysorg/sglang:nightly-dev-20251215-4449c170 bash
+
+# If you already have SGLang installed, uninstall the current SGLang version
+pip uninstall sglang -y
+
+# Install the PyPI Package
+pip install sglang==0.5.6.post2.dev8005+pr.15207.g39d5bd57a \
+  --extra-index-url https://sgl-project.github.io/whl/pr/
+```
+
+## Model Deployment
+
+Use the configuration selector below to automatically generate the appropriate deployment command.
+
+import { MiMoV2FlashDeployment } from "/src/snippets/autoregressive/mimo-v2-flash-deployment.jsx";
+
+<MiMoV2FlashDeployment />
+
+MI355X (ROCm) is validated in the selector above with `--tp-size 4`, Triton attention, and `--disable-custom-all-reduce`. `--tp-size 8` hit a QKV sharding error during validation. EAGLE speculative decoding is still WIP on MI355X.
+
+## Testing the deployment
+
+Once the server is running, test it with a chat completion request in another terminal:
+
+```bash Command
+curl http://localhost:30000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "XiaomiMiMo/MiMo-V2-Flash",
+    "messages": [
+      {"role": "user", "content": "Hello! What can you help me with?"}
+    ],
+    "temperature": 0.7,
+    "max_tokens": 100
+  }'
+```
+
+**Expected response:**
+
+```json Config
+{
+  "id": "...",
+  "object": "chat.completion",
+  "model": "XiaomiMiMo/MiMo-V2-Flash",
+  "choices": [{
+    "message": {
+      "role": "assistant",
+      "content": "Hello! I can help you with..."
+    }
+  }]
+}
+```
+
+## Troubleshooting
+
+**DeepGEMM Timeout Error**
+
+Occasionally DeepGEMM timeout errors occur during first launch. Simply rerun the server command in the same container - the compiled kernels are cached and subsequent launches will be fast.
+
+**ROCm MI355X Attention Backend**
+
+If you see an error such as `AiterAttnBackend.forward_decode() got an unexpected keyword argument 'sinks'` on MI355X, use the `MI355X` + `Performance Optimizations` command from the selector above, which switches to Triton attention and keeps `--disable-custom-all-reduce`.

docs_new/cookbook/autoregressive/intro.mdx

@@ -0,0 +1,118 @@
+---
+title: Overview
+mode: wide
+description: Practical guides for deploying and using large language models and vision language models with SGLang.
+metatags:
+    description: "Explore SGLang autoregressive model cookbooks for LLM and VLM deployment, invocation, optimization, and benchmarking examples."
+---
+
+<CardGroup cols={3}>
+  <Card
+    title="Qwen"
+    mode="card"
+    href="/cookbook/autoregressive/Qwen/Qwen3.6"
+    img="/cards/logos/qwen.png"
+  />
+  <Card
+    title="DeepSeek"
+    mode="card"
+    href="/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2"
+    img="/cards/logos/deepseek.png"
+  />
+  <Card
+    title="Llama"
+    mode="card"
+    href="/cookbook/autoregressive/Llama/Llama3.3-70B"
+    img="/cards/logos/llama.png"
+  />
+  <Card
+    title="GLM"
+    mode="card"
+    href="/cookbook/autoregressive/GLM/GLM-4.5"
+    img="/cards/logos/glm.png"
+  />
+  <Card
+    title="Google"
+    mode="card"
+    href="/cookbook/autoregressive/Google/Gemma4"
+    img="/cards/logos/google.png"
+  />
+  <Card
+    title="OpenAI"
+    mode="card"
+    href="/cookbook/autoregressive/OpenAI/GPT-OSS"
+    img="/cards/logos/openai.png"
+  />
+  <Card
+    title="Moonshotai"
+    mode="card"
+    href="/cookbook/autoregressive/Moonshotai/Kimi-K2.6"
+    img="/cards/logos/moonshotai.png"
+  />
+  <Card
+    title="MiniMax"
+    mode="card"
+    href="/cookbook/autoregressive/MiniMax/MiniMax-M2.5"
+    img="/cards/logos/minimax.png"
+  />
+  <Card
+    title="NVIDIA"
+    mode="card"
+    href="/cookbook/autoregressive/NVIDIA/Nemotron3-Nano"
+    img="/cards/logos/nvidia.png"
+  />
+  <Card
+    title="Ernie"
+    mode="card"
+    href="/cookbook/autoregressive/Ernie/Ernie4.5"
+    img="/cards/logos/ernie.png"
+  />
+  <Card
+    title="StepFun"
+    mode="card"
+    href="/cookbook/autoregressive/StepFun/Step3.5"
+    img="/cards/logos/stepfun.png"
+  />
+  <Card
+    title="InclusionAI"
+    mode="card"
+    href="/cookbook/autoregressive/InclusionAI/Ling-2.5-1T"
+    img="/cards/logos/inclusionai.png"
+  />
+  <Card
+    title="InternLM"
+    mode="card"
+    href="/cookbook/autoregressive/InternLM/Intern-S1"
+    img="/cards/logos/internlm.png"
+  />
+  <Card
+    title="InternVL"
+    mode="card"
+    href="/cookbook/autoregressive/InternVL/InternVL3.5"
+    img="/cards/logos/internvl.png"
+  />
+  <Card
+    title="Jina AI"
+    mode="card"
+    href="/cookbook/autoregressive/Jina/Jina-reranker-m0"
+    img="/cards/logos/jina.png"
+  />
+  <Card
+    title="Mistral"
+    mode="card"
+    href="/cookbook/autoregressive/Mistral/Ministral-3"
+    img="/cards/logos/mistral.png"
+  />
+  <Card
+    title="Xiaomi"
+    mode="card"
+    href="/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash"
+    img="/cards/logos/xiaomi.png"
+  />
+  <Card
+    title="FlashLabs"
+    mode="card"
+    href="/cookbook/autoregressive/FlashLabs/Chroma1.0"
+    img="/cards/logos/flashlabs.png"
+  />
+</CardGroup>

docs_new/cookbook/base/benchmarks/autoregressive_model_benchmark.mdx

@@ -0,0 +1,287 @@
+---
+title: Autoregressive Model Benchmark Documentation
+metatags:
+    description: "Benchmark LLM and VLM serving throughput and latency with sglang.bench_serving - supports SGLang, vLLM, and multiple datasets."
+---
+
+`sglang.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of Large Language Models (LLMs) and Vision Language Models(VLMs). It supports various backends (`SGLang`, `vLLM`, etc.) and offers flexible configurations for request rates, dataset types, and profiling.
+
+## 1. Quick Start
+
+### Basic Usage (Random Data)
+
+Run a benchmark using randomly generated prompts with a local SGLang server.
+
+```bash Command
+python -m sglang.bench_serving --backend sglang --port 30000 --dataset-name random --num-prompts 100
+```
+
+### Real-World Data (ShareGPT)
+
+Run a benchmark using the ShareGPT dataset with a specific request rate.
+
+```shell Command
+python -m sglang.bench_serving \
+  --backend sglang \
+  --dataset-name sharegpt \
+  --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
+  --num-prompts 1000 \
+  --request-rate 10
+```
+
+## 2. Parameter Reference
+
+### 2.1 Backend & Server Configuration
+
+These parameters define the target server and the inference engine being used.
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--backend`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**Required.** Specifies the backend engine. Options: `sglang`, `sglang-native`, `sglang-oai`, `sglang-oai-chat`, `vllm`, `vllm-chat`, `lmdeploy`, `lmdeploy-chat`, `trt`, `gserver`, `truss`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--base-url`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The API base URL (if not using specific host/port flags).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--host`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Server hostname. Default: `0.0.0.0`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--port`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Server port. If not set, it defaults to the specific backend's standard port.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--model`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Model name or path. If unset, it queries `/v1/models` for configuration.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--served-model-name`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The model name used in the API request body. Defaults to the value of `--model`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tokenizer`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Path or name of the tokenizer. Defaults to the model configuration.</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.2 Dataset Configuration
+
+Controls the source of the prompts used for benchmarking.
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dataset-name`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The type of dataset. Options: `sharegpt`, `custom`, `random`, `random-ids`, `generated-shared-prefix`, `mmmu`, `image`, `mooncake`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dataset-path`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>File path to the dataset (e.g., local JSON file for ShareGPT).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--num-prompts`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Total number of prompts to process. Default: `1000`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--seed`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Random seed for reproducibility.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tokenize-prompt`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Uses integer IDs instead of strings for inputs. Useful for precise length control.</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.3 Input/Output Length Control
+
+Parameters to control the shape of requests (context length and generation length).
+
+#### For Random/Image Datasets:
+
+- `--random-input-len`: Number of input tokens per request.
+- `--random-output-len`: Number of output tokens per request.
+- `--random-range-ratio`: Range ratio for sampling input/output lengths.
+
+#### For ShareGPT Dataset:
+
+- `--sharegpt-output-len`: Overrides the output length defined in the dataset for each request.
+- `--sharegpt-context-len`: Max context length. Requests exceeding this are dropped.
+
+#### General Request Modifiers:
+
+- `--extra-request-body`: Appends a JSON object to the request payload (e.g., \{"key": "value"\}). Useful for passing sampling parameters.
+- `--prompt-suffix`: A string suffix appended to all user prompts.
+- `--disable-ignore-eos`: If set, the model will stop generation upon hitting the EOS token (benchmarks usually ignore EOS to force max generation length).
+- `--apply-chat-template`: Applies the model's chat template to the input.
+
+### 2.4 Traffic & Concurrency
+
+Controls how fast requests are sent to the server.
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--request-rate`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Requests per second (RPS). If `inf` (default), all requests are sent immediately (burst). Otherwise, arrival times follow a Poisson process.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--max-concurrency`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The maximum number of active requests allowed at once. Even if `request-rate` is high, the client will hold back requests if this limit is reached.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--warmup-requests`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of requests to run before the actual measurement begins to warm up the server.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--flush-cache`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flushes the server cache before starting the benchmark.</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.5 Output & Logging
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--output-file`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Path to save the results in JSONL format.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--output-details`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Includes detailed metrics in the output.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--print-requests`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Prints requests to stdout as they are sent (useful for debugging).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-tqdm`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Hides the progress bar.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-stream`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Disables streaming mode (waits for full response).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--return-logprob`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Requests logprobs from the server.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tag`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>An arbitrary string tag added to the output file for identification.</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.6 Advanced
+
+#### 2.6.1 Image / Multi-modal
+
+Only applicable when --dataset-name is set to image.
+
+- `--image-count`: Number of images per request.
+- `--image-resolution`: Resolution (e.g., 1080p, 4k, or custom 1080x1920).
+- `--image-format`: jpeg or png.
+- `--image-content`: random (noise) or blank.
+
+#### 2.6.2 LoRA Benchmarking
+
+Used to simulate multi-LoRA serving scenarios.
+
+- `--lora-name`: A list of LoRA adapter names (e.g., `--lora-name` adapter1 adapter2).
+- `--lora-request-distribution`: How requests are assigned to adapters:
+  - `uniform`: Equal probability.
+  - `distinct`: New adapter for every request.
+  - `skewed`: Follows a Zipf distribution (simulating hot/cold adapters).
+- `--lora-zipf-alpha`: The alpha parameter for the Zipf distribution (if `skewed` is used).
+
+#### 2.6.3 Profiling
+
+Tools for deep performance analysis.
+
+- `--profile`: Enables Torch Profiler (Requires `SGLANG_TORCH_PROFILER_DIR` env var on server).
+- `--plot-throughput`: Generates throughput/concurrency plots (requires `termplotlib` and `gnuplot`).
+- `--profile-activities`: Activities to profile (CPU, GPU, CUDA_PROFILER).
+- `--profile-num-steps`: Number of steps to profile.
+- `--profile-by-stage` / `--profile-stages`: Profile specific processing stages.
+
+#### 2.6.4 PD Disaggregation
+
+For benchmarking Prefill-Decode (PD) separated architectures.
+
+- `--pd-separated`: Enable PD disaggregation benchmarking.
+- `--profile-prefill-url`: URL(s) of prefill workers for profiling.
+- `--profile-decode-url`: URL(s) of decode workers for profiling.
+
+<span style={{color:"red"}}>Note</span>: In PD mode, `prefill` and `decode` must be profiled separately.
+
+### 2.7 Specialized Datasets
+
+#### 2.7.1 Generated Shared Prefix (GSP):
+
+Designed to test system prompt caching/prefix sharing performance.
+
+- `--gsp-num-groups`: Number of unique system prompts.
+- `--gsp-prompts-per-group`: How many user questions share the same system prompt.
+- `--gsp-system-prompt-len`: Length of the shared prefix.
+- `--gsp-fast-prepare`: Skips some statistics calculation for faster startup.
+
+#### 2.7.2 Mooncake
+
+Designed for trace replay.
+
+- `--mooncake-slowdown-factor`: Slows down the trace replay (e.g., 2.0 = 2x slower).
+- `--mooncake-num-rounds`: Number of conversation rounds (supports multi-turn).
+- `--use-trace-timestamps`: Schedules requests based on timestamps found in the trace file.
+
+## 3. Metrics
+
+After running the benchmark, the tool generally reports:
+
+- `E2E` (End-to-End Latency): The total time from sending the request to receiving the final token.
+- `TTFT` (Time To First Token): The time between sending the request and seeing the first word appear. This represents the Prefill time (processing the image and text prompt).
+- `TPOT` (Time per Output Token): The average time it takes to generate one token (excluding the first one). This is calculated per request.
+- `ITL` (Inter-Token Latency): The time gap between two distinct streaming packets. While TPOT is an average, ITL measures the "jitter" or smoothness of the stream.

docs_new/cookbook/base/benchmarks/diffusion_model_benchmark.mdx

@@ -0,0 +1,235 @@
+---
+title: Diffusion Models Benchmark Documentation
+metatags:
+    description: "Benchmark diffusion model serving throughput and latency with SGLang - supports image and video generation with flexible configurations."
+---
+
+`sglang.multimodal_gen.benchmarks.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of Diffusion Models. It supports two backends (`sglang-image`, `sglang-video`) and offers flexible configurations for request rates, dataset types, and profiling.
+
+## 1. Quick Start
+
+### 1.1 Benchmarking in Low Concurrency
+
+Run a benchmark on a local server (port 30000) generating 1 videos/images from the `vbench` dataset.
+
+```bash Command
+# For text to video: such as Wan2.2-T2V-A14B-Diffusers
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-video --dataset vbench --task t2v --num-prompts 1 --max-concurrency 1
+
+# For image to video: such as Wan2.2-I2V-A14B-Diffusers
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-video --dataset vbench --task i2v --num-prompts 1 --max-concurrency 1
+
+# For image-text to video: such as Wan2.2-TI2V-5B-Diffusers
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-video --dataset vbench --task ti2v --num-prompts 1 --max-concurrency 1
+
+# For text to image: such as Qwen-Image
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-image --dataset vbench --task t2i --num-prompts 1 --max-concurrency 1
+
+# For image-text to image: such as Qwen-Image-Edit
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-image --dataset vbench --task ti2i --num-prompts 1 --max-concurrency 1
+```
+
+### 1.2 Benchmarking in High Concurrency
+
+Run a benchmark on a local server (port 30000) generating 20 videos/images from the `vbench` dataset.
+
+```bash Command
+# For text to video: such as Wan2.2-T2V-A14B-Diffusers
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-video --dataset vbench --task t2v --num-prompts 20 --max-concurrency 20
+
+# For image to video: such as Wan2.2-I2V-A14B-Diffusers
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-video --dataset vbench --task i2v --num-prompts 20 --max-concurrency 20
+
+# For image-text to video: such as Wan2.2-TI2V-5B-Diffusers
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-video --dataset vbench --task ti2v --num-prompts 20 --max-concurrency 20
+
+# For text to image: such as Qwen-Image
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-image --dataset vbench --task t2i --num-prompts 20 --max-concurrency 20
+
+# For image-text to image: such as Qwen-Image-Edit
+python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
+    --backend sglang-image --dataset vbench --task ti2i --num-prompts 20 --max-concurrency 20
+```
+
+## 2. Parameter Reference
+
+### 2.1 Connection & Backend Settings
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Default</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--backend`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**Required**</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The backend type to use. Choices: `sglang-image`, `sglang-video`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--base-url`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Base URL of the server (e.g., `http://localhost:30000`). If specified, this overrides `--host` and `--port`.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--host`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The server host (e.g., `127.0.0.1`).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--port`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The server port.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--model`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Model name or path.</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.2 Workload & Task Configuration
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "34%"}} />
+    <col style={{width: "33%"}} />
+    <col style={{width: "33%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Choices</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--task`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`t2v`, `i2v`, `ti2v`, `t2i`, `ti2i`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Defines the generation task: `t2v` (Text-to-Video), `i2v` (Image-to-Video), `ti2v` (Text+Image-to-Video), `t2i` (Text-to-image), `ti2i` (Text+Image-to-Image).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dataset`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`vbench`, `random`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The source of prompts/inputs.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dataset-path`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>(Optional) Path to a local dataset file if not using built-in presets.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--num-prompts`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>The total number of prompts/requests to execute during the benchmark.</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.3 Generation Parameters
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--width`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The target width for the generated image or video.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--height`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The target height for the generated image or video.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--num-frames`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Number of frames to generate (Specific to Video backends).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--fps`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Frames Per Second configuration (Specific to Video backends).</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.4 Concurrency & Load Control
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--request-rate`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The number of requests initiated per second. If set to `inf`, all requests are sent immediately (burst). If set to a number, request arrival times follow a Poisson process.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--max-concurrency`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The maximum number of requests allowed to execute simultaneously. This simulates a semaphore or upstream limit. Even if `request-rate` is high, the actual processing rate is capped by this value.</td>
+    </tr>
+  </tbody>
+</table>
+
+### 2.5 Logging & Output
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "50%"}} />
+    <col style={{width: "50%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Argument</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--output-file`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Path to save the benchmark metrics (JSON format).</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--disable-tqdm`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>If set, disables the progress bar in the console.</td>
+    </tr>
+  </tbody>
+</table>
+
+## 3. Metrics
+
+- `Request Throughput` (req/s), Output Throughput (tok/s)
+- `Latency Mean` (ms): Time to Per Step
+- `Peak Memory Max` (ms): Max Memory Usage during running