[Docs] Rename docs_new/ to docs/ (#32123)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
zijiexia
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c949e91f18
commit
b819d2fb5b
@@ -0,0 +1,30 @@
|
||||
# Node
|
||||
node_modules/
|
||||
.env
|
||||
.DS_Store
|
||||
.cache/
|
||||
dist/
|
||||
.next/
|
||||
*.log
|
||||
|
||||
# OS
|
||||
Thumbs.db
|
||||
Desktop.ini
|
||||
|
||||
# VSCode
|
||||
.vscode/
|
||||
|
||||
# Mintlify
|
||||
.mintlify/
|
||||
|
||||
# Python (if any)
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Misc
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
.agents
|
||||
.claude
|
||||
skills-lock.json
|
||||
@@ -0,0 +1,7 @@
|
||||
# Mintlify automatically ignores these files and directories:
|
||||
# .git, .github, .claude, .agents, .idea, node_modules,
|
||||
# README.md, LICENSE.md, CHANGELOG.md, CONTRIBUTING.md
|
||||
|
||||
# Draft content
|
||||
drafts/
|
||||
*.draft.mdx
|
||||
+381
@@ -0,0 +1,381 @@
|
||||
---
|
||||
name: sglang-docs-mintlify
|
||||
description: Build and maintain the SGLang documentation site and integrated cookbook using Mintlify. Use when
|
||||
creating docs pages, configuring navigation, adding components, or setting up
|
||||
API references.
|
||||
license: Apache-2.0
|
||||
compatibility: Requires Node.js for CLI. Works with any Git-based workflow.
|
||||
metadata:
|
||||
author: SGLang Team
|
||||
version: "1.0"
|
||||
mintlify-proj: mintlify
|
||||
---
|
||||
|
||||
# SGLang Mintlify documentation guide for agents
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
- **Do not guess flags, defaults, or behavior.**
|
||||
If you’re documenting CLI args, env vars, APIs, or performance behavior, verify against:
|
||||
- the upstream codebase (`sgl-project/sglang`)
|
||||
- the current public docs (`docs.sglang.io`) until the migration is complete
|
||||
- or an authoritative vendor doc when platform-specific (ROCm, CANN/Ascend, Intel XPU).
|
||||
- **Prefer fixing the docs-site version of an internal link** instead of copying links from older docs.
|
||||
- **Keep examples copy/pasteable.** Use placeholders consistently (e.g., `MODEL_PATH`, `HF_TOKEN`, `HOST`, `PORT`).
|
||||
|
||||
## Source of truth hierarchy
|
||||
|
||||
1. **This repo**
|
||||
- `docs.json` for site structure + navigation
|
||||
- existing MDX pages for voice + conventions
|
||||
2. **Canonical current docs**
|
||||
- `docs.sglang.io` (Sphinx site) is currently the reference structure and content baseline.
|
||||
3. **Implementation**
|
||||
- `sgl-project/sglang` for anything that can change with releases (flags, env vars, defaults, supported models).
|
||||
4. **Cookbook**
|
||||
- `cookbook.sglang.io` / `sgl-project/sgl-cookbook` for recipe patterns and model-specific operational guidance.
|
||||
|
||||
## Writing standards (SGLang-specific)
|
||||
|
||||
### Voice and structure
|
||||
|
||||
* Second person (“you”), active voice.
|
||||
* Prefer **short, scannable sections** with clear outcomes.
|
||||
* Headings in **sentence case**.
|
||||
* Put prerequisites before commands.
|
||||
|
||||
### Technical accuracy patterns
|
||||
|
||||
For pages that include commands/configs, always specify:
|
||||
|
||||
* **Platform** (NVIDIA CUDA / AMD ROCm / Intel XPU / Ascend NPU / CPU)
|
||||
* **OS** (if relevant) and **version constraints**
|
||||
* **Model identifier** format (e.g., Hugging Face repo id) and where it goes (`--model-path`, `--model`, etc.)
|
||||
* **Parallelism knobs** used in the example (`--tp`, `--dp`, node count, etc.)
|
||||
* Any required secrets/tokens (`HF_TOKEN`) and where they are used.
|
||||
|
||||
# Mintlify best practices
|
||||
|
||||
**Always consult [mintlify.com/docs](https://mintlify.com/docs) for components, configuration, and latest features.**
|
||||
|
||||
If you are not already connected to the Mintlify MCP server, [https://mintlify.com/docs/mcp](https://mintlify.com/docs/mcp), add it so that you can search more efficiently.
|
||||
|
||||
**Always** favor searching the current Mintlify documentation over whatever is in your training data about Mintlify.
|
||||
|
||||
Mintlify is a documentation platform that transforms MDX files into documentation sites. Configure site-wide settings in the `docs.json` file, write content in MDX with YAML frontmatter, and favor built-in components over custom components.
|
||||
|
||||
Full schema at [mintlify.com/docs.json](https://mintlify.com/docs.json).
|
||||
|
||||
## Before you write
|
||||
|
||||
### Understand the project
|
||||
|
||||
Read `docs.json` in the project root. This file defines the entire site: navigation structure, theme, colors, links, API and specs.
|
||||
|
||||
Understanding the project tells you:
|
||||
|
||||
* What pages exist and how they're organized
|
||||
* What navigation groups are used (and their naming conventions)
|
||||
* How the site navigation is structured
|
||||
* What theme and configuration the site uses
|
||||
|
||||
### Check for existing content
|
||||
|
||||
Search the docs before creating new pages. You may need to:
|
||||
|
||||
* Update an existing page instead of creating a new one
|
||||
* Add a section to an existing page
|
||||
* Link to existing content rather than duplicating
|
||||
|
||||
### Read surrounding content
|
||||
|
||||
Before writing, read 2-3 similar pages to understand the site's voice, structure, formatting conventions, and level of detail.
|
||||
|
||||
### Understand Mintlify components
|
||||
|
||||
Review the Mintlify [components](https://www.mintlify.com/docs/components) to select and use any relevant components for the documentation request that you are working on.
|
||||
|
||||
## Quick reference
|
||||
|
||||
### CLI commands
|
||||
|
||||
* `npm i -g mint` - Install the Mintlify CLI
|
||||
* `mint dev` - Local preview at localhost:3000
|
||||
* `mint broken-links` - Check internal links
|
||||
* `mint a11y` - Check for accessibility issues in content
|
||||
* `mint rename` - Rename/move files and update references
|
||||
* `mint validate` - Validate documentation builds
|
||||
|
||||
### Required files
|
||||
|
||||
* `docs.json` - Site configuration (navigation, theme, integrations, etc.). See [global settings](https://www.mintlify.com/docs/organize/settings) for all options.
|
||||
* `*.mdx` files - Documentation pages with YAML frontmatter
|
||||
|
||||
### Example file structure
|
||||
|
||||
```
|
||||
project/
|
||||
├── docs.json # Site configuration
|
||||
├── introduction.mdx
|
||||
├── quickstart.mdx
|
||||
├── guides/
|
||||
│ └── example.mdx
|
||||
├── openapi.yml # API specification
|
||||
├── images/ # Static assets
|
||||
│ └── example.png
|
||||
└── snippets/ # Reusable components
|
||||
└── component.jsx
|
||||
```
|
||||
|
||||
## Page frontmatter
|
||||
|
||||
Every page requires `title` in its frontmatter. Include `description` for SEO and navigation.
|
||||
|
||||
```yaml theme={null}
|
||||
---
|
||||
title: "Clear, descriptive title"
|
||||
description: "Concise summary for SEO and navigation."
|
||||
---
|
||||
```
|
||||
|
||||
Optional frontmatter fields:
|
||||
|
||||
* `sidebarTitle`: Short title for sidebar navigation.
|
||||
* `icon`: Lucide or Font Awesome icon name, URL, or file path.
|
||||
* `tag`: Label next to the page title in the sidebar (for example, "NEW").
|
||||
* `mode`: Page layout mode (`default`, `wide`, `custom`).
|
||||
* `keywords`: Array of terms related to the page content for local search and SEO.
|
||||
* Any custom YAML fields for use with personalization or conditional content.
|
||||
|
||||
## File conventions
|
||||
|
||||
* Match existing naming patterns in the directory
|
||||
* If there are no existing files or inconsistent file naming patterns, use kebab-case: `getting-started.mdx`, `api-reference.mdx`
|
||||
* Use root-relative paths without file extensions for internal links: `/getting-started/quickstart`
|
||||
* Do not use relative paths (`../`) or absolute URLs for internal pages
|
||||
* When you create a new page, add it to `docs.json` navigation or it won't appear in the sidebar
|
||||
|
||||
## Organize content
|
||||
|
||||
When a user asks about anything related to site-wide configurations, start by understanding the [global settings](https://www.mintlify.com/docs/organize/settings). See if a setting in the `docs.json` file can be updated to achieve what the user wants.
|
||||
|
||||
### Navigation
|
||||
|
||||
The `navigation` property in `docs.json` controls site structure. Choose one primary pattern at the root level, then nest others within it.
|
||||
|
||||
**Choose your primary pattern:**
|
||||
|
||||
| Pattern | When to use |
|
||||
| ------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| **Groups** | Default. Single audience, straightforward hierarchy |
|
||||
| **Tabs** | Distinct sections with different audiences (Guides vs API Reference) or content types |
|
||||
| **Anchors** | Want persistent section links at sidebar top. Good for separating docs from external resources |
|
||||
| **Dropdowns** | Multiple doc sections users switch between, but not distinct enough for tabs |
|
||||
| **Products** | Multi-product company with separate documentation per product |
|
||||
| **Versions** | Maintaining docs for multiple API/product versions simultaneously |
|
||||
| **Languages** | Localized content |
|
||||
|
||||
**Within your primary pattern:**
|
||||
|
||||
* **Groups** - Organize related pages. Can nest groups within groups, but keep hierarchy shallow
|
||||
* **Menus** - Add dropdown navigation within tabs for quick jumps to specific pages
|
||||
* **`expanded: false`** - Collapse nested groups by default. Use for reference sections users browse selectively
|
||||
* **`openapi`** - Auto-generate pages from OpenAPI spec. Add at group/tab level to inherit
|
||||
|
||||
**Common combinations:**
|
||||
|
||||
* Tabs containing groups (most common for docs with API reference)
|
||||
* Products containing tabs (multi-product SaaS)
|
||||
* Versions containing tabs (versioned API docs)
|
||||
* Anchors containing groups (simple docs with external resource links)
|
||||
|
||||
### Links and paths
|
||||
|
||||
* **Internal links:** Root-relative, no extension: `/getting-started/quickstart`
|
||||
* **Images:** Store in `/images`, reference as `/images/example.png`
|
||||
* **External links:** Use full URLs, they open in new tabs automatically
|
||||
|
||||
## Customize docs sites
|
||||
|
||||
**What to customize where:**
|
||||
|
||||
* **Brand colors, fonts, logo** → `docs.json`. See [global settings](https://www.mintlify.com/docs/organize/settings)
|
||||
* **Component styling, layout tweaks** → `custom.css` at project root
|
||||
* **Dark mode** → Enabled by default. Only disable with `"appearance": "light"` in `docs.json` if brand requires it
|
||||
|
||||
Start with `docs.json`. Only add `custom.css` when you need styling that config doesn't support.
|
||||
|
||||
## Write content
|
||||
|
||||
### Components
|
||||
|
||||
The [components overview](https://mintlify.com/docs/components) organizes all components by purpose: structure content, draw attention, show/hide content, document APIs, link to pages, and add visual context. Start there to find the right component.
|
||||
|
||||
**Common decision points:**
|
||||
|
||||
| Need | Use |
|
||||
| -------------------------- | ----------------------- |
|
||||
| Hide optional details | `<Accordion>` |
|
||||
| Long code examples | `<Expandable>` |
|
||||
| User chooses one option | `<Tabs>` |
|
||||
| Linked navigation cards | `<Card>` in `<Columns>` |
|
||||
| Sequential instructions | `<Steps>` |
|
||||
| Code in multiple languages | `<CodeGroup>` |
|
||||
| API parameters | `<ParamField>` |
|
||||
| API response fields | `<ResponseField>` |
|
||||
|
||||
**Callouts by severity:**
|
||||
|
||||
* `<Note>` - Supplementary info, safe to skip
|
||||
* `<Info>` - Helpful context such as permissions
|
||||
* `<Tip>` - Recommendations or best practices
|
||||
* `<Warning>` - Potentially destructive actions
|
||||
* `<Check>` - Success confirmation
|
||||
|
||||
### Reusable content
|
||||
|
||||
**When to use snippets:**
|
||||
|
||||
* Exact content appears on more than one page
|
||||
* Complex components you want to maintain in one place
|
||||
* Shared content across teams/repos
|
||||
|
||||
**When NOT to use snippets:**
|
||||
|
||||
* Slight variations needed per page (leads to complex props)
|
||||
|
||||
Import snippets with `import { Component } from "/path/to/snippet-name.jsx"`.
|
||||
|
||||
## Writing standards
|
||||
|
||||
### Voice and structure
|
||||
|
||||
* Second-person voice ("you")
|
||||
* Active voice, direct language
|
||||
* Sentence case for headings ("Getting started", not "Getting Started")
|
||||
* Sentence case for code block titles ("Expandable example", not "Expandable Example")
|
||||
* Lead with context: explain what something is before how to use it
|
||||
* Prerequisites at the start of procedural content
|
||||
|
||||
### What to avoid
|
||||
|
||||
**Never use:**
|
||||
|
||||
* Marketing language ("powerful", "seamless", "robust", "cutting-edge")
|
||||
* Filler phrases ("it's important to note", "in order to")
|
||||
* Excessive conjunctions ("moreover", "furthermore", "additionally")
|
||||
* Editorializing ("obviously", "simply", "just", "easily")
|
||||
|
||||
**Watch for AI-typical patterns:**
|
||||
|
||||
* Overly formal or stilted phrasing
|
||||
* Unnecessary repetition of concepts
|
||||
* Generic introductions that don't add value
|
||||
* Concluding summaries that restate what was just said
|
||||
|
||||
### Formatting
|
||||
|
||||
* All code blocks must have language tags
|
||||
* All images and media must have descriptive alt text
|
||||
* Use bold and italics only when they serve the reader's understanding--never use text styling just for decoration
|
||||
* No decorative formatting or emoji
|
||||
|
||||
### Code examples
|
||||
|
||||
* Keep examples simple and practical
|
||||
* Use realistic values (not "foo" or "bar")
|
||||
* One clear example is better than multiple variations
|
||||
* Test that code works before including it
|
||||
|
||||
## Deploy
|
||||
|
||||
Mintlify deploys automatically when changes are pushed to the connected Git repository.
|
||||
|
||||
**What agents can configure:**
|
||||
|
||||
* **Redirects** → Add to `docs.json` with `"redirects": [{"source": "/old", "destination": "/new"}]`
|
||||
* **SEO indexing** → Control with `"seo": {"indexing": "all"}` to include hidden pages in search
|
||||
|
||||
**Requires dashboard setup (human task):**
|
||||
|
||||
* Custom domains and subdomains
|
||||
* Preview deployment settings
|
||||
* DNS configuration
|
||||
|
||||
For `/docs` subpath hosting with Vercel or Cloudflare, agents can help configure rewrite rules. See [/docs subpath](https://mintlify.com/docs/deploy/vercel).
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Understand the task
|
||||
|
||||
Identify what needs to be documented, which pages are affected, and what the reader should accomplish afterward. If any of these are unclear, ask.
|
||||
|
||||
### 2. Research
|
||||
|
||||
* Read `docs.json` to understand the site structure
|
||||
* Search existing docs for related content
|
||||
* Read similar pages to match the site's style
|
||||
|
||||
### 3. Plan
|
||||
|
||||
* Synthesize what the reader should accomplish after reading the docs and the current content
|
||||
* Propose any updates or new content
|
||||
* Verify that your proposed changes will help readers be successful
|
||||
|
||||
### 4. Write
|
||||
|
||||
* Start with the most important information
|
||||
* Keep sections focused and scannable
|
||||
* Use components appropriately (don't overuse them)
|
||||
* Mark anything uncertain with a TODO comment:
|
||||
|
||||
```mdx theme={null}
|
||||
{/* TODO: Verify the default timeout value */}
|
||||
```
|
||||
|
||||
### 5. Update navigation
|
||||
|
||||
If you created a new page, add it to the appropriate group in `docs.json`.
|
||||
|
||||
### 6. Verify
|
||||
|
||||
Before submitting:
|
||||
|
||||
* [ ] Frontmatter includes title and description
|
||||
* [ ] All code blocks have language tags
|
||||
* [ ] Internal links use root-relative paths without file extensions
|
||||
* [ ] New pages are added to `docs.json` navigation
|
||||
* [ ] Content matches the style of surrounding pages
|
||||
* [ ] No marketing language or filler phrases
|
||||
* [ ] TODOs are clearly marked for anything uncertain
|
||||
* [ ] Run `mint broken-links` to check links
|
||||
* [ ] Run `mint validate` to find any errors
|
||||
|
||||
## Edge cases
|
||||
|
||||
### Migrations
|
||||
|
||||
If a user asks about migrating to Mintlify, ask if they are using ReadMe or Docusaurus. If they are, use the [@mintlify/scraping](https://www.npmjs.com/package/@mintlify/scraping) CLI to migrate content. If they are using a different platform to host their documentation, help them manually convert their content to MDX pages using Mintlify components.
|
||||
|
||||
### Hidden pages
|
||||
|
||||
Any page that is not included in the `docs.json` navigation is hidden. Use hidden pages for content that should be accessible by URL or indexed for the assistant or search, but not discoverable through the sidebar navigation.
|
||||
|
||||
### Exclude pages
|
||||
|
||||
The `.mintignore` file is used to exclude files from a documentation repository from being processed.
|
||||
|
||||
## Common gotchas
|
||||
|
||||
1. **Component imports** - JSX components need explicit import, MDX components don't
|
||||
2. **Frontmatter required** - Every MDX file needs `title` at minimum
|
||||
3. **Code block language** - Always specify language identifier
|
||||
4. **Never use `mint.json`** - `mint.json` is deprecated. Only ever use `docs.json`
|
||||
|
||||
## Resources
|
||||
|
||||
* [Documentation](https://mintlify.com/docs)
|
||||
* [https://github.com/sgl-project/sglang](https://github.com/sgl-project/sglang)
|
||||
* [Configuration schema](https://mintlify.com/docs.json)
|
||||
* [Feature requests](https://github.com/orgs/mintlify/discussions/categories/feature-requests)
|
||||
* [Bugs and feedback](https://github.com/orgs/mintlify/discussions/categories/bugs-feedback)
|
||||
@@ -0,0 +1,34 @@
|
||||
> **Customize this file**: Tailor this template to your project by noting specific contribution types you're looking for, adding a Code of Conduct, or adjusting the writing guidelines to match your style.
|
||||
|
||||
# Contribute to the documentation
|
||||
|
||||
Thank you for your interest in contributing to our documentation! This guide will help you get started.
|
||||
|
||||
## How to contribute
|
||||
|
||||
### Option 1: Edit directly on GitHub
|
||||
|
||||
1. Navigate to the page you want to edit
|
||||
2. Click the "Edit this file" button (the pencil icon)
|
||||
3. Make your changes and submit a pull request
|
||||
|
||||
### Option 2: Local development
|
||||
|
||||
1. Fork and clone this repository
|
||||
2. Install the Mintlify CLI: `npm i -g mint`
|
||||
3. Create a branch for your changes
|
||||
4. Make changes
|
||||
5. Run `mint dev`
|
||||
6. Preview your changes at `http://localhost:3000`
|
||||
7. Commit your changes and submit a pull request
|
||||
|
||||
For more details on local development, see our [development guide](development.mdx).
|
||||
|
||||
## Writing guidelines
|
||||
|
||||
- **Use active voice**: "Run the command" not "The command should be run"
|
||||
- **Address the reader directly**: Use "you" instead of "the user"
|
||||
- **Keep sentences concise**: Aim for one idea per sentence
|
||||
- **Lead with the goal**: Start instructions with what the user wants to accomplish
|
||||
- **Use consistent terminology**: Don't alternate between synonyms for the same concept
|
||||
- **Include examples**: Show, don't just tell
|
||||
+201
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
+166
@@ -0,0 +1,166 @@
|
||||
# SGLang Documentation
|
||||
|
||||
The official documentation and cookbook for [SGLang](https://github.com/sgl-project/sglang) — a high-performance serving framework for large language models and vision-language models.
|
||||
|
||||
- **Docs**: Getting started guides, installation, and reference
|
||||
- **Cookbook**: Battle-tested recipes for deploying specific models (Qwen, DeepSeek, Llama, GLM, etc.) on various hardware
|
||||
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
.
|
||||
├── docs.json # Site configuration (navigation, theme, metadata)
|
||||
├── index.mdx # Homepage
|
||||
├── docs/ # Documentation pages
|
||||
│ └── get-started/
|
||||
│ └── install.mdx # Installation guide
|
||||
├── cookbook/ # Model deployment recipes (one .mdx page per model)
|
||||
│ ├── intro.mdx # Cookbook overview and recipe index
|
||||
│ └── autoregressive/ # Autoregressive recipes (also: diffusion/, omni/, …)
|
||||
│ └── DeepSeek/
|
||||
│ └── DeepSeek-V4.mdx
|
||||
└── src/snippets/ # Config-driven cookbook engine
|
||||
├── _deployment.jsx # Shared deploy-matrix engine (no model-specific code)
|
||||
├── _playground.jsx # Shared override-playground engine
|
||||
└── configs/
|
||||
└── deepseek-ai/ # Per-model config + benchmarks (HF-org folder)
|
||||
└── deepseek-v4.jsx
|
||||
```
|
||||
|
||||
Pages are `.mdx` files with YAML frontmatter. Navigation is defined in `docs.json`.
|
||||
|
||||
## Local development
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Node.js >= 20
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Install the CLI
|
||||
npm i -g mint
|
||||
|
||||
# From docs/ (where docs.json lives), start the dev server (hot reload)
|
||||
mint dev
|
||||
```
|
||||
|
||||
Preview at `http://localhost:3000`.
|
||||
|
||||
### Useful commands
|
||||
|
||||
```bash
|
||||
mint dev # Start local preview server
|
||||
mint broken-links # Check for broken links
|
||||
mint update # Update the CLI
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! Whether you want to add a recipe for a new model, improve existing docs, or fix a typo — PRs are appreciated.
|
||||
|
||||
### Quick edit (GitHub)
|
||||
|
||||
1. Navigate to the file you want to edit on GitHub
|
||||
2. Click the pencil icon to edit
|
||||
3. Submit a pull request
|
||||
|
||||
### Local development workflow
|
||||
|
||||
```bash
|
||||
# 1. Fork sgl-project/sglang and clone your fork
|
||||
git clone https://github.com/<YOUR_USERNAME>/sglang.git
|
||||
cd sglang/docs
|
||||
|
||||
# 2. Create a branch
|
||||
git checkout -b my-changes
|
||||
|
||||
# 3. Start the dev server and make your changes
|
||||
mint dev
|
||||
|
||||
# 4. Verify links aren't broken
|
||||
mint broken-links
|
||||
|
||||
# 5. Commit and push
|
||||
git add <files>
|
||||
git commit -m "docs: describe your change"
|
||||
git push origin my-changes
|
||||
|
||||
# 6. Open a pull request on GitHub
|
||||
```
|
||||
|
||||
### Adding a new cookbook recipe
|
||||
|
||||
The autoregressive cookbook is **config-driven**: two shared engines —
|
||||
`src/snippets/_deployment.jsx` (the deploy matrix) and `src/snippets/_playground.jsx`
|
||||
(the override playground) — contain **no** model-specific code. Adding a model means adding
|
||||
*data*: a per-model config (plus optional benchmarks) that both engines consume, and an
|
||||
`.mdx` page that imports them. Copy [`DeepSeek-V4`](cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx)
|
||||
as the reference instance.
|
||||
|
||||
**Recommended — use the Claude Code skill `/cookbook-add-model`.** It walks the whole flow
|
||||
interactively: collect the model card + verified `sglang serve` recipes → instantiate the
|
||||
template → wire up the nav/card → validate → fill in measured benchmarks. Related skills:
|
||||
`/cookbook-migrate-model` (port an existing legacy-template page) and `/cookbook-review-pr`
|
||||
(review a cookbook PR against the checklist).
|
||||
|
||||
**The files it creates / edits — using DeepSeek-V4 as the example:**
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `src/snippets/configs/deepseek-ai/deepseek-v4.jsx` | Per-model config: `supportedHardware`, `variants`, `quantizations`, `strategies`, the `cells[]` deploy matrix (verified env + flags per `hw × variant × quant × strategy × nodes`), and `playgroundFeatures`. |
|
||||
| `src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx` | One entry per cell with measured speed/accuracy + the `sglang_version` it ran on. Optional — skip until you have numbers. |
|
||||
| `cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx` | The page: imports the engines + config, renders `<Deployment>` / `<Playground>`, and adds prose (intro + specs + license, config tips, advanced usage). |
|
||||
| `docs.json` | Nav entry under Cookbook → category → vendor. |
|
||||
| `cookbook/autoregressive/intro.mdx` | Vendor `<Card>` on the category homepage. |
|
||||
|
||||
Note the two folder conventions: under `configs/` the folder is the **HuggingFace org**
|
||||
(`deepseek-ai`); under `cookbook/` it's the **display vendor** (`DeepSeek`). The page wires
|
||||
everything together with data only — no engine edits:
|
||||
|
||||
```mdx
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
<Playground config={config} />
|
||||
```
|
||||
|
||||
Keep `tag: NEW` on the new page and strip it from same-vendor siblings (≤1 per vendor), then
|
||||
validate from `docs/`: `mint validate`, `mint broken-links`, and `mint dev` for a visual
|
||||
smoke test.
|
||||
|
||||
> Diffusion / omni / specbundle pages follow their own category structure — don't force the
|
||||
> autoregressive config-driven template on them.
|
||||
|
||||
### Writing guidelines
|
||||
|
||||
- Use active voice: "Run the command" not "The command should be run"
|
||||
- Address the reader as "you"
|
||||
- Keep sentences concise — one idea per sentence
|
||||
- Lead with the goal, then the steps
|
||||
- Use consistent terminology
|
||||
- Include concrete examples and code snippets
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Thank you to all the authors who contributed to the original documentation in `sglang/docs/` and the original cookbook in [`sgl-cookbook`](https://github.com/sgl-project/sgl-cookbook). The migration to the new Mintlify-based documentation was led by the following [ACM-VIT](https://github.com/ACM-VIT) students:
|
||||
|
||||
[@Adhyan Jain](https://github.com/Adhyan-Jain), [@Maitri-shah29](https://github.com/Maitri-shah29), [@architnigam](https://github.com/architnigam), [@Nakul-Sinha](https://github.com/Nakul-Sinha), [@divyamagrawal06](https://github.com/divyamagrawal06), [@A-Taman](https://github.com/A-Taman), [@nimeshas](https://github.com/nimeshas), [@IshhanKheria](https://github.com/IshhanKheria), [@Krishang-Zinzuwadia](https://github.com/Krishang-Zinzuwadia), [@pokymono](https://github.com/pokymono), [@Ishitajoshii](https://github.com/Ishitajoshii), [@AdityaVKochar](https://github.com/AdityaVKochar)
|
||||
|
||||
Advised by [@adarshxs](https://github.com/adarshxs) (ACM-VIT) and [@wisclmy0611](https://github.com/wisclmy0611), [@Richardczl98](https://github.com/Richardczl98) (LMSYS).
|
||||
|
||||
## Community
|
||||
|
||||
- [GitHub](https://github.com/sgl-project/sglang)
|
||||
- [Slack](https://slack.sglang.io/)
|
||||
- [Discord](https://discord.gg/4ugb2t6YY2)
|
||||
- [X / Twitter](https://x.com/lmsysorg)
|
||||
- [LinkedIn](https://www.linkedin.com/company/sgl-project/)
|
||||
|
||||
## License
|
||||
|
||||
Apache License 2.0 — see the [LICENSE](LICENSE) for details.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 683 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 646 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 702 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 175 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 704 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 618 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 2.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 714 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 637 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 951 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 475 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 349 KiB |
@@ -0,0 +1,6 @@
|
||||
<svg width="940" height="525" viewBox="0 0 940 525" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<rect width="940" height="525" fill="none"/>
|
||||
<text x="470" y="286" text-anchor="middle" font-family="Arial, Helvetica, sans-serif" font-size="128" font-weight="700" letter-spacing="0">
|
||||
<tspan fill="#2B5BFF">Bai</tspan><tspan fill="#D5001C">du</tspan>
|
||||
</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 376 B |
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.4 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 8.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.1 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 98 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.3 KiB |
@@ -0,0 +1,9 @@
|
||||
<svg width="940" height="525" viewBox="0 0 940 525" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="JoyAI-Echo logo">
|
||||
<rect width="940" height="525" rx="28" fill="none"/>
|
||||
<circle cx="354" cy="262" r="86" fill="#FF4D2E"/>
|
||||
<circle cx="410" cy="262" r="86" fill="#111111" fill-opacity="0.92"/>
|
||||
<path d="M345 231C362 213 393 213 410 231" stroke="white" stroke-width="16" stroke-linecap="round"/>
|
||||
<path d="M345 293C362 311 393 311 410 293" stroke="white" stroke-width="16" stroke-linecap="round"/>
|
||||
<text x="494" y="249" fill="#111111" font-family="Arial, Helvetica, sans-serif" font-size="72" font-weight="700" letter-spacing="0">JoyAI</text>
|
||||
<text x="494" y="318" fill="#FF4D2E" font-family="Arial, Helvetica, sans-serif" font-size="58" font-weight="700" letter-spacing="0">Echo</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 825 B |
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 8.7 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
@@ -0,0 +1,8 @@
|
||||
<svg width="940" height="525" viewBox="0 0 940 525" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="LTX logo">
|
||||
<rect width="940" height="525" rx="28" fill="transparent"/>
|
||||
<g transform="translate(251 169) scale(5.86)" fill="#111111">
|
||||
<path d="M0 30.0087V7.50057C0 7.09765 0.154254 6.69973 0.460162 6.43869C0.708822 6.22729 0.987356 6.12029 1.29316 6.12029H8.2339C8.63671 6.12029 9.03463 6.26205 9.31316 6.55308C9.53944 6.79174 9.65133 7.07777 9.65133 7.41345V23.198C9.65133 23.6108 9.98462 23.944 10.3974 23.944H21.4638C21.8666 23.944 22.267 24.0858 22.5431 24.3767C22.7668 24.6155 22.8812 24.9015 22.8812 25.2372V30.5856C22.8812 31.0457 22.6823 31.4982 22.3018 31.7569C22.078 31.9086 21.8244 31.9832 21.5383 31.9832L1.99199 31.9956C0.890348 31.9981 0 31.1078 0 30.0087Z"/>
|
||||
<path d="M36.5888 31.9926C34.4062 31.9876 32.492 31.6543 30.8413 30.9878C29.1906 30.3214 27.9104 29.2346 26.9981 27.7227C26.0856 26.2132 25.6333 24.2137 25.6382 21.7269L25.6532 13.7194L21.7016 13.7119C21.3486 13.7119 21.0528 13.5876 20.8116 13.3365C20.5705 13.0853 20.4512 12.7819 20.4537 12.4164L20.4636 7.39299C20.4636 7.02744 20.5854 6.72154 20.8265 6.47288C21.0677 6.22422 21.3635 6.09983 21.7165 6.10233L25.6681 6.10983L25.6779 1.29066C25.6779 0.925114 25.7998 0.619208 26.041 0.370548C26.2821 0.121887 26.5779 0 26.9309 0L33.9065 0.0124913C34.2595 0.0124913 34.5554 0.136772 34.7965 0.387931C35.0376 0.639089 35.1569 0.944995 35.1545 1.30805L35.1445 6.12721L41.2078 6.13959C41.5608 6.13959 41.8566 6.26398 42.0977 6.51514C42.3389 6.76629 42.4582 7.0722 42.4557 7.43525L42.4458 12.4586C42.4458 12.8242 42.3239 13.1301 42.0829 13.3787C41.8417 13.6274 41.5434 13.7518 41.1928 13.7493L35.1296 13.7368L35.1171 20.8988C35.1171 21.8613 35.3061 22.6148 35.6914 23.1618C36.0767 23.7089 36.6833 23.985 37.5186 23.985L41.5608 23.9925C41.9138 23.9925 42.2096 24.1168 42.4507 24.368C42.6919 24.6192 42.8113 24.9251 42.8088 25.2881L42.7988 30.7093C42.7988 31.075 42.677 31.3808 42.4358 31.6294C42.1947 31.8782 41.8963 32.0025 41.5459 32L36.5913 31.9901L36.5888 31.9926Z"/>
|
||||
<path d="M47.5486 31.9851C47.2282 31.9851 46.965 31.8682 46.7589 31.6369C46.5503 31.4056 46.4485 31.1395 46.4485 30.841C46.4485 30.7416 46.4634 30.6248 46.4957 30.4929C46.5279 30.3611 46.5926 30.2268 46.6869 30.0951L54.3506 18.9342C54.4648 18.7675 54.4673 18.5463 54.3556 18.3771L47.4543 8.01457C47.3896 7.91517 47.335 7.79827 47.2854 7.6664C47.2382 7.53463 47.2133 7.40036 47.2133 7.26859C47.2133 6.97017 47.3251 6.70403 47.5486 6.47275C47.7722 6.24147 48.0279 6.12458 48.316 6.12458H55.6444C56.0914 6.12458 56.4267 6.23158 56.6501 6.44787C56.8737 6.66426 57.0327 6.85328 57.1295 7.01993L60.3082 11.8169C60.5043 12.1128 60.939 12.1128 61.1352 11.8169L64.3139 7.01993C64.4405 6.85328 64.6094 6.66426 64.8156 6.44787C65.0216 6.23158 65.3494 6.12458 65.7964 6.12458H72.7896C73.0778 6.12458 73.331 6.24147 73.557 6.47275C73.7805 6.70403 73.8922 6.95268 73.8922 7.21883C73.8922 7.38547 73.8748 7.53463 73.8451 7.6664C73.8128 7.79827 73.7482 7.91517 73.6539 8.01457L66.6159 18.3747C66.4992 18.5463 66.5017 18.77 66.6209 18.9392L74.4212 30.0975C74.5181 30.2293 74.5801 30.3636 74.6124 30.4954C74.6448 30.6273 74.6596 30.744 74.6596 30.8435C74.6596 31.142 74.5479 31.4081 74.3244 31.6394C74.1008 31.8707 73.8451 31.9874 73.557 31.9874H65.8934C65.4786 31.9874 65.1756 31.888 64.9844 31.689C64.7932 31.4901 64.6317 31.3086 64.5051 31.142L60.9886 25.9544C60.7924 25.671 60.3753 25.6685 60.1766 25.9471L56.4118 31.1395C56.3149 31.3061 56.1634 31.4876 55.9573 31.6865C55.7488 31.8855 55.4383 31.9851 55.0236 31.9851H47.5486Z"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 30 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 3.7 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 9.5 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 84 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 25 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 30 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: Unlimited-OCR
|
||||
description: "Deploy Baidu Unlimited-OCR with SGLang for long document OCR using prefill-aware sliding-window attention."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Unlimited-OCR support is in [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186). Until that PR is included in a tagged SGLang release, install from a build that contains the PR.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git fetch origin pull/29186/head && git checkout FETCH_HEAD
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware to generate the launch command. The recipe uses FlashAttention-3 with `--page-size 1`, which is required by the current prefill-aware sliding-window attention path. It also disables radix cache by default, which is the better fit for batch OCR workloads where each request usually contains a different image.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/baidu/unlimited-ocr.jsx";
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
## Playground
|
||||
|
||||
Use the Playground to adjust tensor parallelism on top of the selected deployment cell.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's multimodal OCR model for document parsing. It uses a sliding-window language backbone, but SGLang serves it with a prefill-aware sliding-window path so image and prompt tokens remain visible during long decode.
|
||||
|
||||
The SGLang integration loads the standalone Unlimited-OCR architecture with SAM and CLIP vision encoders plus a DeepSeek-style language backbone. It supports OpenAI-compatible image requests and model-specific image processing options through `images_config`.
|
||||
|
||||
**Resources:** [Hugging Face](https://huggingface.co/baidu/Unlimited-OCR) · [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Attention backend**: use `--attention-backend fa3 --page-size 1`. The prefill-aware SWA page table is built with token-level locations, so page size 1 is required.
|
||||
- **Radix cache**: keep `--disable-radix-cache` for batch OCR over different documents. If your workload repeatedly asks about the same image and prompt, remove this flag to allow prefix reuse through `PureSWARadixCache`.
|
||||
- **Long OCR generations**: keep the default prefill-aware SWA path enabled. It retains prompt and image KV while still applying a sliding window to generated text.
|
||||
- **Custom logit processor**: keep `--enable-custom-logit-processor` in the launch command.
|
||||
- **Image modes**: pass `images_config.image_mode` per request. Supported modes are `tiny`, `small`, `base`, `large`, and `gundam`. Multiple images are supported only for `tiny`, `small`, and `base`.
|
||||
- **Default image mode**: when `images_config.image_mode` is omitted, SGLang uses `gundam`.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 OCR request
|
||||
|
||||
<Accordion title="OCR Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/Unlimited-OCR",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "document parsing."},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://example.com/your_document.png"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=2048,
|
||||
temperature=0,
|
||||
extra_body={"images_config": {"image_mode": "gundam"}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Choosing an image mode
|
||||
|
||||
Use lower modes to reduce prefill cost for simple images, and use `gundam` for high-detail document parsing.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Use</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Multiple images</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>tiny</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Lowest prefill cost.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>small</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Lightweight OCR requests.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>base</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Balanced quality and cost.</td>
|
||||
<td style={{padding: "9px 12px"}}>Yes</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>large</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Higher resolution single-image OCR.</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><code>gundam</code></td>
|
||||
<td style={{padding: "9px 12px"}}>Default high-detail document parsing mode.</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,224 @@
|
||||
---
|
||||
title: Ornith-1.0
|
||||
description: "Deploy DeepReinforce Ornith-1.0 with SGLang - a self-improving agentic-coding model family with 397B, 35B, and 9B checkpoints plus FP8 and GGUF variants."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
Ornith-1.0 model cards recommend SGLang `>=0.5.9`. The Deploy panel below emits the base serve command; the reasoning and tool-call parsers from the model-card quickstarts (`--reasoning-parser qwen3` for `<think>...</think>` traces, `--tool-call-parser qwen3_coder` for Qwen-style XML tool calls) are added on top via the [Playground](#playground).
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install "sglang>=0.5.9"
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick an Ornith checkpoint to generate the launch command. The non-FP8 397B recipe requires an H200 single node in this matrix. The 397B-FP8 recipe is available on H100 and H200 with TP=8; H100 also supports the 35B and 9B checkpoints. The 35B recipes use tensor parallelism 2 in this matrix. The 9B checkpoint is dense and serves on a single GPU by default; the command panel makes that default explicit with `--tp 1`.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/deepreinforce-ai/ornith-1.0.jsx";
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground layers SGLang features on top of whichever cell the Deploy panel is showing — only your overrides change, and any change flips the badge to **Not Verified** until the new configuration is run end-to-end.
|
||||
|
||||
For Ornith-1.0 the knobs are the reasoning and tool-call parsers:
|
||||
|
||||
- **Reasoning Parser** appends `--reasoning-parser qwen3`. Ornith emits `<think>...</think>` traces; with this on, SGLang surfaces them as `message.reasoning_content` instead of leaving the tags inline in `content`.
|
||||
- **Tool Call Parser** appends `--tool-call-parser qwen3_coder`, so Qwen-style XML tool calls are returned as OpenAI-compatible `tool_calls`.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ornith-1.0](https://huggingface.co/collections/deepreinforce-ai/ornith-10) is DeepReinforce's self-improving open-source model family for agentic coding. The model cards describe the family as post-trained on top of Gemma 4 and Qwen 3.5, and the collection currently includes 397B, 35B, and 9B repos plus FP8 and GGUF variants. The model cards report results on Terminal-Bench 2.1, SWE-Bench, NL2Repo, ClawEval, and SWE Atlas benchmarks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Agentic coding specialization**: the model cards describe Ornith-1.0 as specialized for agentic coding and report coding-agent benchmark results.
|
||||
- **Self-improving training**: the model cards state that Ornith-1.0 uses reinforcement learning to optimize both solution rollouts and the scaffold that drives those rollouts.
|
||||
- **Reasoning model behavior**: assistant responses begin with a `<think>...</think>` reasoning block before the final answer; enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) to split it into `reasoning_content`.
|
||||
- **Tool calling**: emits Qwen-style XML tool calls; enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground).
|
||||
- **Long context**: model-card recipes use `--context-length 262144`.
|
||||
- **MIT license**: the Hugging Face repos are released under MIT.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "32%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "32%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Format</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Deploy Panel</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Notes</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B">deepreinforce-ai/Ornith-1.0-397B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>H200 only</td>
|
||||
<td style={{padding: "9px 12px"}}>Flagship 397B MoE checkpoint; model-card baseline uses TP=8 on an H200 single node.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B-FP8">deepreinforce-ai/Ornith-1.0-397B-FP8</a></td>
|
||||
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 repo in the collection; the deploy command uses this repo id with TP=8.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B">deepreinforce-ai/Ornith-1.0-35B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>35B MoE checkpoint; the deploy command uses TP=2.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-FP8">deepreinforce-ai/Ornith-1.0-35B-FP8</a></td>
|
||||
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 repo in the collection; the deploy command uses this repo id with TP=2.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B">deepreinforce-ai/Ornith-1.0-9B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||
<td style={{padding: "9px 12px"}}>Dense 9B checkpoint; the model card describes it as designed for efficient single-GPU deployment.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF">deepreinforce-ai/Ornith-1.0-35B-GGUF</a></td>
|
||||
<td style={{padding: "9px 12px"}}>GGUF</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
<td style={{padding: "9px 12px"}}>Listed for completeness; GGUF targets llama.cpp-style local inference, not the SGLang server recipe here.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B-GGUF">deepreinforce-ai/Ornith-1.0-9B-GGUF</a></td>
|
||||
<td style={{padding: "9px 12px"}}>GGUF</td>
|
||||
<td style={{padding: "9px 12px"}}>No</td>
|
||||
<td style={{padding: "9px 12px"}}>Listed for completeness; the model card shows llama.cpp and Ollama examples for the GGUF build.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:** [MIT](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B/blob/main/LICENSE)
|
||||
|
||||
**Resources:** [Hugging Face collection](https://huggingface.co/collections/deepreinforce-ai/ornith-10) · [Ornith blog](https://deep-reinforce.com/ornith_1_0.html)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Reasoning parser**: Ornith responses include `<think>...</think>`. Enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) so OpenAI-compatible responses expose the reasoning trace as `message.reasoning_content`.
|
||||
- **Tool-call parser**: enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) so `<tool_call>` blocks are returned as OpenAI-compatible tool calls.
|
||||
- **Context length**: the model-card SGLang recipes use `--context-length 262144`. Lower it if you need more memory headroom.
|
||||
- **Tensor parallelism**: the 397B model-card recipes use `--tp 8`; in this single-node matrix, non-FP8 397B is H200-only, while 397B-FP8 is available on both H100 and H200. The 35B deploy commands use `--tp 2`. The 9B model-card recipe is single-GPU by default; the command panel makes that explicit with `--tp 1`. Adjust TP to match your node and memory budget.
|
||||
- **Sampling**: model cards recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20` for normal use. Their reported benchmark setup may use different task-specific sampling parameters.
|
||||
- **Benchmarks**: benchmark numbers in the model cards are reported by DeepReinforce. They are useful for context, but the command panel leaves recipes unverified until exact runs are signed off.
|
||||
|
||||
## 3. Usage Examples
|
||||
|
||||
### 3.1 Basic Chat Completion
|
||||
|
||||
`message.reasoning_content` is only populated when the server was launched with the `--reasoning-parser qwen3` toggle (see the [Playground](#playground)); otherwise the `<think>...</think>` trace stays inline in `message.content`.
|
||||
|
||||
<Accordion title="Python client">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Ornith-1.0-9B",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a compact Python function is_prime(n)."}
|
||||
],
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=1024,
|
||||
extra_body={"top_k": 20},
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) and launch with the resulting command. Then use the standard OpenAI-compatible `tools` field:
|
||||
|
||||
<Accordion title="Tool-call request">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run_tests",
|
||||
"description": "Run the project's test suite.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"target": {"type": "string", "description": "Test target or command"}
|
||||
},
|
||||
"required": ["target"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Ornith-1.0-9B",
|
||||
messages=[{"role": "user", "content": "Run the unit tests for the parser module."}],
|
||||
tools=tools,
|
||||
tool_choice="auto",
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -0,0 +1,526 @@
|
||||
---
|
||||
title: DeepSeek-Math-V2
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-Math-V2 with SGLang - advanced mathematical reasoning model with gold-level IMO/CMO performance and theorem-proving capabilities."
|
||||
---
|
||||
|
||||
import { DeepSeekMathV2Deployment } from '/src/snippets/autoregressive/deepseek-math-v2-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024
|
||||
- **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy
|
||||
- **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024
|
||||
- **Large MoE Model**: ~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
|
||||
|
||||
<DeepSeekMathV2Deployment />
|
||||
|
||||
<Warning>
|
||||
DeepSeek-Math-V2 is built on DeepSeek-V3.2 and uses DSA sparse attention. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on this model.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Hardware Requirements:**
|
||||
|
||||
- **B200 (183GB)**: BF16 tp=8
|
||||
- **B300 (275GB)**: BF16 tp=8
|
||||
|
||||
**DP Attention:**
|
||||
|
||||
- Enable DP attention for high-throughput scenarios
|
||||
- The `--dp` value commonly matches the `--tp` value
|
||||
- Trade-off: Higher throughput at the cost of slightly increased latency
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Deployment Command
|
||||
|
||||
Deploy the model using the command generated above. Example for B200:
|
||||
|
||||
```shell Command
|
||||
sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 Mathematical Reasoning
|
||||
|
||||
DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning.
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Mathematical reasoning problem
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-Math-V2",
|
||||
messages=[
|
||||
{"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"}
|
||||
],
|
||||
max_tokens=4096,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2.
|
||||
|
||||
This is a classic formula for the sum of the first n natural numbers. We can prove by induction.
|
||||
|
||||
Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds.
|
||||
|
||||
Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1
|
||||
)((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n.
|
||||
|
||||
...
|
||||
=============== Content =================
|
||||
We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below.
|
||||
|
||||
---
|
||||
|
||||
### 1. Proof by mathematical induction
|
||||
|
||||
**Base case (\(n=1\))**:
|
||||
\[
|
||||
1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1,
|
||||
\]
|
||||
so the formula holds for \(n=1\).
|
||||
|
||||
**Inductive hypothesis:**
|
||||
Assume that for some positive integer \(k\) the formula is true, i.e.
|
||||
\[
|
||||
1+2+\dots+k = \frac{k(k+1)}{2}.
|
||||
\]
|
||||
|
||||
**Inductive step (\(k \to k+1\))**:
|
||||
Consider the sum up to \(k+1\):
|
||||
\[
|
||||
\begin{aligned}
|
||||
1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt]
|
||||
&= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt]
|
||||
&= (k+1)\left(\frac{k}{2}+1\right)\\[4pt]
|
||||
&= (k+1)\frac{k+2}{2}\\[4pt]
|
||||
&= \frac{(k+1)(k+2)}{2}\\[4pt]
|
||||
&= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}.
|
||||
\end{aligned}
|
||||
\]
|
||||
Thus the formula also holds for \(n=k+1\).
|
||||
|
||||
By the principle of mathematical induction,
|
||||
\[
|
||||
1+2+3+\dots+n = \frac{n(n+1)}{2}
|
||||
\]
|
||||
for every positive integer \(n\).
|
||||
|
||||
---
|
||||
|
||||
### 2. Proof by pairing (Gauss’s trick)
|
||||
|
||||
Let
|
||||
\[
|
||||
S = 1 + 2 + 3 + \dots + n.
|
||||
\]
|
||||
|
||||
Write the same sum in reverse order:
|
||||
\[
|
||||
S = n + (n-1) + (n-2) + \dots + 1.
|
||||
\]
|
||||
|
||||
Add the two equalities term‑by‑term:
|
||||
\[
|
||||
\begin{aligned}
|
||||
2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\
|
||||
&= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\
|
||||
&= n\,(n+1).
|
||||
\end{aligned}
|
||||
\]
|
||||
|
||||
Therefore
|
||||
\[
|
||||
S = \frac{n(n+1)}{2}.
|
||||
\]
|
||||
|
||||
Both proofs are rigorous and show that the formula holds for all positive integers \(n\).
|
||||
```
|
||||
|
||||
### 4.3 Competition-Level Problems
|
||||
|
||||
**Example: IMO-style Problem:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# IMO-style problem
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-Math-V2",
|
||||
messages=[
|
||||
{"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."}
|
||||
],
|
||||
max_tokens=8192,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
We need to prove that for positive real numbers a,b,c with abc = 1, we have:
|
||||
|
||||
\[
|
||||
(a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1.
|
||||
\]
|
||||
|
||||
We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite:
|
||||
|
||||
...
|
||||
=============== Content =================
|
||||
|
||||
We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove
|
||||
|
||||
\[
|
||||
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
|
||||
\]
|
||||
|
||||
---
|
||||
|
||||
### 1. A convenient substitution
|
||||
|
||||
Because \(abc=1\), we can write
|
||||
|
||||
\[
|
||||
a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x}
|
||||
\]
|
||||
|
||||
with positive numbers \(x,y,z\).
|
||||
(For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).)
|
||||
|
||||
---
|
||||
|
||||
### 2. Rewriting the factors
|
||||
|
||||
\[
|
||||
\begin{aligned}
|
||||
a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm]
|
||||
b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm]
|
||||
c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}.
|
||||
\end{aligned}
|
||||
\]
|
||||
|
||||
Hence the product becomes
|
||||
|
||||
\[
|
||||
P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)
|
||||
=\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}.
|
||||
\]
|
||||
|
||||
---
|
||||
|
||||
### 3. Reducing to a known inequality
|
||||
|
||||
We have to show \(P\le1\), i.e.
|
||||
|
||||
\[
|
||||
(x+z-y)(x+y-z)(y+z-x)\le xyz .
|
||||
\tag{1}
|
||||
\]
|
||||
|
||||
Set
|
||||
|
||||
\[
|
||||
p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz .
|
||||
\]
|
||||
|
||||
Notice that
|
||||
|
||||
\[
|
||||
x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x .
|
||||
\]
|
||||
|
||||
Therefore
|
||||
|
||||
\[
|
||||
\begin{aligned}
|
||||
(x+z-y)(x+y-z)(y+z-x)
|
||||
&=(p-2x)(p-2y)(p-2z)\\
|
||||
&=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\
|
||||
&=-p^{3}+4pq-8r .
|
||||
\end{aligned}
|
||||
\]
|
||||
|
||||
Inequality (1) is thus equivalent to
|
||||
|
||||
\[
|
||||
-p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r .
|
||||
\tag{2}
|
||||
\]
|
||||
|
||||
---
|
||||
|
||||
### 4. Applying Schur’s inequality
|
||||
|
||||
Schur’s inequality of third degree states that for any non‑negative \(x,y,z\)
|
||||
|
||||
\[
|
||||
p^{3}+9r\ge 4pq .
|
||||
\]
|
||||
|
||||
Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2).
|
||||
Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds.
|
||||
|
||||
Consequently (1) is true, and we obtain \(P\le1\).
|
||||
|
||||
---
|
||||
|
||||
### 5. Equality case
|
||||
|
||||
Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\).
|
||||
Then \(a=b=c=1\), and indeed the product equals \(1\).
|
||||
|
||||
---
|
||||
|
||||
Thus for all positive \(a,b,c\) with \(abc=1\),
|
||||
|
||||
\[
|
||||
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
|
||||
\]
|
||||
|
||||
∎
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
#### 5.1.1 GSM8K Benchmark
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --port 30000
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.975
|
||||
Invalid: 0.000
|
||||
Latency: 34.358 s
|
||||
Output throughput: 540.162 token/s
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x, 183GB each)
|
||||
- Model: DeepSeek-Math-V2
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.8
|
||||
|
||||
#### 5.2.1 Latency Benchmark
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-Math-V2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 53.34
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2778
|
||||
Request throughput (req/s): 0.19
|
||||
Input token throughput (tok/s): 36.97
|
||||
Output token throughput (tok/s): 52.19
|
||||
Peak output token throughput (tok/s): 56.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 89.16
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5330.72
|
||||
Median E2E Latency (ms): 5879.28
|
||||
P90 E2E Latency (ms): 8320.33
|
||||
P99 E2E Latency (ms): 9921.29
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 183.38
|
||||
Median TTFT (ms): 177.92
|
||||
P99 TTFT (ms): 217.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 17.96
|
||||
Median TPOT (ms): 18.39
|
||||
P99 TPOT (ms): 19.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.57
|
||||
Median ITL (ms): 18.63
|
||||
P95 ITL (ms): 19.26
|
||||
P99 ITL (ms): 19.48
|
||||
Max ITL (ms): 24.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.2 Throughput Benchmark
|
||||
|
||||
**Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-Math-V2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 217.36
|
||||
Total input tokens: 301701
|
||||
Total input text tokens: 301701
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 187456
|
||||
Request throughput (req/s): 4.60
|
||||
Input token throughput (tok/s): 1388.05
|
||||
Output token throughput (tok/s): 866.67
|
||||
Peak output token throughput (tok/s): 2589.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 2254.72
|
||||
Concurrency: 89.81
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 19521.73
|
||||
Median E2E Latency (ms): 12076.76
|
||||
P90 E2E Latency (ms): 47248.87
|
||||
P99 E2E Latency (ms): 86862.79
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 790.40
|
||||
Median TTFT (ms): 456.81
|
||||
P99 TTFT (ms): 4223.33
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 106.52
|
||||
Median TPOT (ms): 107.24
|
||||
P99 TPOT (ms): 238.33
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 100.29
|
||||
Median ITL (ms): 38.34
|
||||
P95 ITL (ms): 237.00
|
||||
P99 ITL (ms): 347.49
|
||||
Max ITL (ms): 3642.56
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,255 @@
|
||||
---
|
||||
title: DeepSeek-OCR-2
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-OCR-2 with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
|
||||
---
|
||||
|
||||
import { DeepSeekOCR2Deployment } from '/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents).
|
||||
- **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version.
|
||||
- **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks.
|
||||
- **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details.
|
||||
|
||||
For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs.
|
||||
|
||||
<DeepSeekOCR2Deployment />
|
||||
|
||||
**Note**: DeepSeek-OCR-2 has ~3B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details.
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Single GPU Deployment:** DeepSeek-OCR-2 (~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving.
|
||||
- **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway).
|
||||
- **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`.
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
**OpenAI-compatible request example**
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = "http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "deepseek-ai/DeepSeek-OCR-2",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "<image>\n<|grounding|>Convert the document to markdown."},
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 512,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
**Reference**
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Recommended Prompts
|
||||
|
||||
The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts).
|
||||
|
||||
**Structured document conversion** — extracts text while preserving layout:
|
||||
|
||||
```text Example
|
||||
<image>
|
||||
<|grounding|>Convert the document to markdown.
|
||||
```
|
||||
|
||||
**Free-form OCR** — extracts without layouts:
|
||||
|
||||
```text Example
|
||||
<image>
|
||||
Free OCR.
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA H200 GPU (1x)
|
||||
- Model: DeepSeek-OCR-2
|
||||
- Tensor Parallelism: 1
|
||||
- sglang version: 0.0.0.dev1+g93fca0bbc
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models).
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-OCR-2 \
|
||||
--enable-multimodal \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-OCR-2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 3.54
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2710
|
||||
Request throughput (req/s): 2.83
|
||||
Input token throughput (tok/s): 557.53
|
||||
Output token throughput (tok/s): 787.10
|
||||
Peak output token throughput (tok/s): 818.00
|
||||
Peak concurrent requests: 5
|
||||
Total token throughput (tok/s): 1344.63
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 352.69
|
||||
Median E2E Latency (ms): 392.34
|
||||
P90 E2E Latency (ms): 540.64
|
||||
P99 E2E Latency (ms): 639.01
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 18.08
|
||||
Median TTFT (ms): 16.57
|
||||
P99 TTFT (ms): 25.67
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 1.18
|
||||
Median TPOT (ms): 1.21
|
||||
P99 TPOT (ms): 1.22
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.21
|
||||
Median ITL (ms): 1.21
|
||||
P95 ITL (ms): 1.28
|
||||
P99 ITL (ms): 1.44
|
||||
Max ITL (ms): 4.32
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-OCR-2 \
|
||||
--enable-multimodal \
|
||||
--tp 1 \
|
||||
--ep 1 \
|
||||
--dp 1 \
|
||||
--enable-dp-attention \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--model deepseek-ai/DeepSeek-OCR-2 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 14.79
|
||||
Total input tokens: 301698
|
||||
Total input text tokens: 301698
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 185236
|
||||
Request throughput (req/s): 67.63
|
||||
Input token throughput (tok/s): 20402.54
|
||||
Output token throughput (tok/s): 12738.99
|
||||
Peak output token throughput (tok/s): 17508.00
|
||||
Peak concurrent requests: 187
|
||||
Total token throughput (tok/s): 33141.53
|
||||
Concurrency: 86.87
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1284.50
|
||||
Median E2E Latency (ms): 866.07
|
||||
P90 E2E Latency (ms): 3027.32
|
||||
P99 E2E Latency (ms): 5490.63
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 86.08
|
||||
Median TTFT (ms): 50.09
|
||||
P99 TTFT (ms): 613.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.79
|
||||
Median TPOT (ms): 6.54
|
||||
P99 TPOT (ms): 50.10
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.42
|
||||
Median ITL (ms): 4.64
|
||||
P95 ITL (ms): 23.65
|
||||
P99 ITL (ms): 39.62
|
||||
Max ITL (ms): 452.65
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,247 @@
|
||||
---
|
||||
title: DeepSeek-OCR
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-OCR with SGLang - high-accuracy text extraction from images and documents for OCR tasks."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced OCR**: High-accuracy text recognition from images and documents
|
||||
- **Multi-Modality**: Supports various image formats and document types
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details.
|
||||
|
||||
For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.
|
||||
|
||||
import { DeepSeekOCRDeployment } from "/src/snippets/autoregressive/deepseek-ocr-deployment.jsx";
|
||||
|
||||
<DeepSeekOCRDeployment />
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 OCR-Specific Prompts
|
||||
|
||||
DeepSeek-OCR accepts recommended prompts from the model card:
|
||||
|
||||
```text
|
||||
<image>
|
||||
<|grounding|>Convert the document to markdown.
|
||||
```
|
||||
|
||||
```text
|
||||
<image>
|
||||
Free OCR.
|
||||
```
|
||||
|
||||
**OpenAI-compatible image request example:**
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = "http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "deepseek-ai/DeepSeek-OCR",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "<image>\n<|grounding|>Convert the document to markdown."
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://example.com/your_image.jpg"}
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
"max_tokens": 512,
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.text)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (1x)
|
||||
- Model: DeepSeek-OCR
|
||||
- Tensor Parallelism: 1
|
||||
- sglang version: 0.5.7
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-OCR \
|
||||
--tp 1 \
|
||||
--dtype float16 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-OCR \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 4.45
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2770
|
||||
Request throughput (req/s): 2.25
|
||||
Input token throughput (tok/s): 442.89
|
||||
Output token throughput (tok/s): 625.26
|
||||
Peak output token throughput (tok/s): 635.00
|
||||
Peak concurrent requests: 4
|
||||
Total token throughput (tok/s): 1068.16
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 443.32
|
||||
Median E2E Latency (ms): 493.29
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 21.59
|
||||
Median TTFT (ms): 20.89
|
||||
P99 TTFT (ms): 24.81
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 1.47
|
||||
Median TPOT (ms): 1.52
|
||||
P99 TPOT (ms): 1.53
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.52
|
||||
Median ITL (ms): 1.51
|
||||
P95 ITL (ms): 1.76
|
||||
P99 ITL (ms): 1.93
|
||||
Max ITL (ms): 8.28
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-OCR \
|
||||
--tp 1 \
|
||||
--ep 1 \
|
||||
--dp 1 \
|
||||
--enable-dp-attention \
|
||||
--dtype float16 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-OCR \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 16.24
|
||||
Total input tokens: 301698
|
||||
Total input text tokens: 301698
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 186927
|
||||
Request throughput (req/s): 61.59
|
||||
Input token throughput (tok/s): 18582.90
|
||||
Output token throughput (tok/s): 11602.84
|
||||
Peak output token throughput (tok/s): 15479.00
|
||||
Peak concurrent requests: 179
|
||||
Total token throughput (tok/s): 30185.75
|
||||
Concurrency: 85.53
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1388.60
|
||||
Median E2E Latency (ms): 901.43
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 73.36
|
||||
Median TTFT (ms): 50.21
|
||||
P99 TTFT (ms): 349.53
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.42
|
||||
Median TPOT (ms): 7.31
|
||||
P99 TPOT (ms): 27.99
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.04
|
||||
Median ITL (ms): 4.62
|
||||
P95 ITL (ms): 21.11
|
||||
P99 ITL (ms): 36.92
|
||||
Max ITL (ms): 172.15
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,993 @@
|
||||
---
|
||||
title: DeepSeek-R1
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-R1 reasoning model with SGLang - advanced step-by-step reasoning with FP8/FP4 quantization for NVIDIA and AMD GPUs."
|
||||
---
|
||||
|
||||
import { DeepSeekR1BasicDeployment } from '/src/snippets/autoregressive/deepseek-r1-basic-deployment.jsx';
|
||||
import { DeepSeekR1AdvancedDeployment } from '/src/snippets/autoregressive/deepseek-r1-advanced-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X
|
||||
- **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X
|
||||
- **BF16 (upcast from FP8)**: [unsloth/DeepSeek-R1-0528-BF16](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16)
|
||||
- **INT8 (channel-wise)**: [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8)
|
||||
- **W4A8**: [novita/Deepseek-R1-0528-W4AFP8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8)
|
||||
- **AWQ (4-bit)**: [QuixiAI/DeepSeek-R1-0528-AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ)
|
||||
- **MXFP4**: [amd/DeepSeek-R1-MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4)
|
||||
|
||||
**License:**
|
||||
To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details.
|
||||
|
||||
For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy.
|
||||
|
||||
<DeepSeekR1BasicDeployment />
|
||||
|
||||
### 3.2 Optimal Configurations
|
||||
|
||||
Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware.
|
||||
|
||||
<DeepSeekR1AdvancedDeployment />
|
||||
|
||||
### 3.3 Configuration Tips
|
||||
|
||||
DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.
|
||||
|
||||
**Recommended GPU configurations by weight type:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> The official DeepSeek-R1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
|
||||
|
||||
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels to avoid JIT overhead (~10 min):
|
||||
```bash
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --trust-remote-code
|
||||
```
|
||||
|
||||
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node.
|
||||
|
||||
**NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`.
|
||||
|
||||
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-0528",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--tool-call-parser deepseekv3 \
|
||||
--chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-0528",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments:
|
||||
🔧 Tool Call: None
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1-0528",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
|
||||
|
||||
DeepSeek-R1 supports EAGLE-based Multi-Token Prediction (MTP), the same mechanism as DeepSeek-V3. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the complete launch command, flag reference, tuning guidance (`--speculative-num-steps`, `--speculative-eagle-topk`, `--max-running-requests`), and `bench_speculative.py` link. R1's speed benchmark commands that include `--speculative-*` flags use this mechanism.
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Limit the model's thinking token budget using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-R1 \
|
||||
--tp 8 \
|
||||
--port 30000 \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--enable-custom-logit-processor
|
||||
```
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-R1",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: B200 GPU (8x)
|
||||
- Model: DeepSeek-R1-0528
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-0528 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 40.00
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4205
|
||||
Request throughput (req/s): 0.25
|
||||
Input token throughput (tok/s): 152.52
|
||||
Output token throughput (tok/s): 105.24
|
||||
Peak output token throughput (tok/s): 110.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 257.76
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3998.40
|
||||
Median E2E Latency (ms): 3207.53
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 153.00
|
||||
Median TTFT (ms): 140.76
|
||||
P99 TTFT (ms): 214.66
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.16
|
||||
Median TPOT (ms): 9.15
|
||||
P99 TPOT (ms): 9.21
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.16
|
||||
Median ITL (ms): 9.15
|
||||
P95 ITL (ms): 9.47
|
||||
P99 ITL (ms): 9.63
|
||||
Max ITL (ms): 15.45
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 51.21
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40725
|
||||
Total generated tokens (retokenized): 40458
|
||||
Request throughput (req/s): 1.56
|
||||
Input token throughput (tok/s): 774.66
|
||||
Output token throughput (tok/s): 795.30
|
||||
Peak output token throughput (tok/s): 1088.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1569.96
|
||||
Concurrency: 13.93
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8918.33
|
||||
Median E2E Latency (ms): 9466.16
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 273.51
|
||||
Median TTFT (ms): 131.71
|
||||
P99 TTFT (ms): 839.57
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 17.56
|
||||
Median TPOT (ms): 17.46
|
||||
P99 TPOT (ms): 28.68
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 17.02
|
||||
Median ITL (ms): 14.70
|
||||
P95 ITL (ms): 16.41
|
||||
P99 ITL (ms): 112.38
|
||||
Max ITL (ms): 461.90
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 110.46
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252162
|
||||
Total generated tokens (retokenized): 251441
|
||||
Request throughput (req/s): 4.53
|
||||
Input token throughput (tok/s): 2261.80
|
||||
Output token throughput (tok/s): 2282.90
|
||||
Peak output token throughput (tok/s): 3900.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 4544.71
|
||||
Concurrency: 92.26
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20380.71
|
||||
Median E2E Latency (ms): 19391.65
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 563.14
|
||||
Median TTFT (ms): 147.62
|
||||
P99 TTFT (ms): 2632.11
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 40.11
|
||||
Median TPOT (ms): 41.98
|
||||
P99 TPOT (ms): 50.10
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 39.37
|
||||
Median ITL (ms): 26.36
|
||||
P95 ITL (ms): 98.16
|
||||
P99 ITL (ms): 150.08
|
||||
Max ITL (ms): 2052.85
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 411.34
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44452
|
||||
Total generated tokens (retokenized): 44390
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 14.83
|
||||
Output token throughput (tok/s): 108.07
|
||||
Peak output token throughput (tok/s): 110.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 122.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 41132.04
|
||||
Median E2E Latency (ms): 44288.71
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 125.76
|
||||
Median TTFT (ms): 126.19
|
||||
P99 TTFT (ms): 137.69
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.21
|
||||
Median TPOT (ms): 9.20
|
||||
P99 TPOT (ms): 9.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.23
|
||||
Median ITL (ms): 9.22
|
||||
P95 ITL (ms): 9.64
|
||||
P99 ITL (ms): 9.86
|
||||
Max ITL (ms): 15.18
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 348.93
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318226
|
||||
Total generated tokens (retokenized): 317630
|
||||
Request throughput (req/s): 0.23
|
||||
Input token throughput (tok/s): 113.69
|
||||
Output token throughput (tok/s): 912.02
|
||||
Peak output token throughput (tok/s): 1088.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 1025.70
|
||||
Concurrency: 14.07
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 61360.70
|
||||
Median E2E Latency (ms): 62071.20
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 176.02
|
||||
Median TTFT (ms): 153.75
|
||||
P99 TTFT (ms): 268.44
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 15.42
|
||||
Median TPOT (ms): 15.59
|
||||
P99 TPOT (ms): 16.07
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 15.39
|
||||
Median ITL (ms): 15.17
|
||||
P95 ITL (ms): 16.62
|
||||
P99 ITL (ms): 18.13
|
||||
Max ITL (ms): 226.59
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 589.31
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1300705
|
||||
Total generated tokens (retokenized): 1297658
|
||||
Request throughput (req/s): 0.54
|
||||
Input token throughput (tok/s): 269.70
|
||||
Output token throughput (tok/s): 2207.16
|
||||
Peak output token throughput (tok/s): 2944.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 2476.86
|
||||
Concurrency: 57.03
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 105032.36
|
||||
Median E2E Latency (ms): 108229.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 223.91
|
||||
Median TTFT (ms): 158.15
|
||||
P99 TTFT (ms): 474.86
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.94
|
||||
Median TPOT (ms): 26.72
|
||||
P99 TPOT (ms): 27.99
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.79
|
||||
Median ITL (ms): 25.37
|
||||
P95 ITL (ms): 26.70
|
||||
P99 ITL (ms): 105.49
|
||||
Max ITL (ms): 237.91
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 40.65
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4195
|
||||
Request throughput (req/s): 0.25
|
||||
Input token throughput (tok/s): 1031.65
|
||||
Output token throughput (tok/s): 103.56
|
||||
Peak output token throughput (tok/s): 110.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 1135.20
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4063.62
|
||||
Median E2E Latency (ms): 3296.13
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 165.91
|
||||
Median TTFT (ms): 154.96
|
||||
P99 TTFT (ms): 240.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.26
|
||||
Median TPOT (ms): 9.27
|
||||
P99 TPOT (ms): 9.42
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.28
|
||||
Median ITL (ms): 9.28
|
||||
P95 ITL (ms): 9.66
|
||||
P99 ITL (ms): 9.83
|
||||
Max ITL (ms): 14.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 56.71
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41589
|
||||
Total generated tokens (retokenized): 41490
|
||||
Request throughput (req/s): 1.41
|
||||
Input token throughput (tok/s): 5290.75
|
||||
Output token throughput (tok/s): 733.41
|
||||
Peak output token throughput (tok/s): 1024.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 6024.16
|
||||
Concurrency: 14.25
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10098.99
|
||||
Median E2E Latency (ms): 10623.46
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 486.80
|
||||
Median TTFT (ms): 189.59
|
||||
P99 TTFT (ms): 2138.73
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 19.06
|
||||
Median TPOT (ms): 19.23
|
||||
P99 TPOT (ms): 30.69
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.53
|
||||
Median ITL (ms): 15.63
|
||||
P95 ITL (ms): 16.64
|
||||
P99 ITL (ms): 109.71
|
||||
Max ITL (ms): 1471.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 115.55
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 169680
|
||||
Total generated tokens (retokenized): 169275
|
||||
Request throughput (req/s): 2.77
|
||||
Input token throughput (tok/s): 11024.93
|
||||
Output token throughput (tok/s): 1468.50
|
||||
Peak output token throughput (tok/s): 2254.00
|
||||
Peak concurrent requests: 70
|
||||
Total token throughput (tok/s): 12493.43
|
||||
Concurrency: 59.45
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 21465.98
|
||||
Median E2E Latency (ms): 20686.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 913.93
|
||||
Median TTFT (ms): 224.92
|
||||
P99 TTFT (ms): 6257.83
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 39.93
|
||||
Median TPOT (ms): 40.99
|
||||
P99 TPOT (ms): 60.91
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.83
|
||||
Median ITL (ms): 26.29
|
||||
P95 ITL (ms): 113.81
|
||||
P99 ITL (ms): 176.94
|
||||
Max ITL (ms): 5521.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.959
|
||||
Invalid: 0.000
|
||||
Latency: 29.185 s
|
||||
Output throughput: 4854.672 token/s
|
||||
```
|
||||
@@ -0,0 +1,637 @@
|
||||
---
|
||||
title: "DeepSeek-V3"
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-V3 MoE model with SGLang - efficient architecture with strong reasoning, coding, and tool-augmented capabilities."
|
||||
---
|
||||
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost.
|
||||
|
||||
Key highlights include:
|
||||
|
||||
- **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable.
|
||||
- **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies.
|
||||
- **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively.
|
||||
- **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { DeepSeekV3Deployment } from "/src/snippets/autoregressive/deepseek-v3-deployment.jsx";
|
||||
|
||||
<DeepSeekV3Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Recommended GPU configurations by weight type:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> The official DeepSeek-V3 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
|
||||
|
||||
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (~10 min):
|
||||
```bash
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code
|
||||
```
|
||||
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
|
||||
|
||||
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
|
||||
|
||||
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
|
||||
|
||||
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [Basic API Usage](../../../docs/get-started/quickstart)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To determine 15% of a number, follow these steps:
|
||||
|
||||
**Step 1: Understand the Problem**
|
||||
You need to find 15% of a given number. Let's assume the number is 240 for this example.
|
||||
|
||||
**Step 2: Convert the Percentage to a Decimal**
|
||||
To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100.
|
||||
|
||||
\[ 15\% = \frac{15}{100} = 0.15 \]
|
||||
|
||||
**Step 3: Multiply the Decimal by the Number**
|
||||
Now, multiply the decimal form of the percentage by the number you want to find the percentage of.
|
||||
|
||||
\[ 0.15 \times 240 \]
|
||||
|
||||
**Step 4: Perform the Multiplication**
|
||||
Calculate the product:
|
||||
|
||||
\[ 0.15 \times 240 = 36 \]
|
||||
|
||||
**Step 5: Conclusion**
|
||||
Therefore, 15% of 240 is:
|
||||
|
||||
\boxed{36}
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--tool-call-parser deepseekv3 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
|
||||
**Quick Test (curl):**
|
||||
|
||||
```shell Command
|
||||
curl "http://127.0.0.1:30000/v1/chat/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"temperature": 0,
|
||||
"max_tokens": 100,
|
||||
"model": "deepseek-ai/DeepSeek-V3",
|
||||
"tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
|
||||
"messages": [{"role": "user", "content": "How'\''s the weather in Beijing today?"}]
|
||||
}'
|
||||
```
|
||||
|
||||
<Note>
|
||||
Use a low `temperature` (e.g. `0`) for more consistent tool call results. The `--chat-template` flag above provides an improved unified prompt for tool use.
|
||||
</Note>
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
Please attach the code blocks below to the previous Python script.
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
|
||||
|
||||
SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). With this optimization, decoding speed improves by up to **1.8×** at batch size 1 and **1.5×** at batch size 32 on H200 TP8.
|
||||
|
||||
**Enable with:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3-0324 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--trust-remote-code \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`.
|
||||
|
||||
<Note>
|
||||
For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs` to include your target batch sizes (default captured sizes for speculative decoding: 48).
|
||||
</Note>
|
||||
|
||||
<Tip>
|
||||
The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable.
|
||||
</Tip>
|
||||
|
||||
#### 4.2.4 MLA Optimizations
|
||||
|
||||
DeepSeek V3 uses [Multi-head Latent Attention (MLA)](https://arxiv.org/pdf/2405.04434), an attention mechanism that improves inference efficiency. SGLang implements several optimizations:
|
||||
|
||||
- **Weight Absorption:** Reorders matrix multiplications to improve decoding phase efficiency.
|
||||
- **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, CutlassMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default.
|
||||
- **FP8 Quantization:** W8A8 FP8 and KV Cache FP8, with BMM operators for weight-absorbed MLA in FP8.
|
||||
- **CUDA Graph & Torch.compile:** Both MLA and MoE support CUDA Graph and Torch.compile for reduced decoding latency.
|
||||
- **Chunked Prefix Cache:** Increases throughput for long-sequence chunked prefill (FlashAttention3 backend only).
|
||||
|
||||
Overall, these optimizations achieve up to **7×** output throughput improvement vs. the baseline.
|
||||
|
||||
**Reference:** See [SGLang v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for details.
|
||||
|
||||
#### 4.2.5 Multi-Node Deployment
|
||||
|
||||
For multi-node serving and hardware-specific examples:
|
||||
|
||||
- [8× H200 / 4–8× B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
|
||||
- [8× MI300X](../../../docs/hardware-platforms/amd_gpu#running-deepseek-v3)
|
||||
- [2×8× H200 with Docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker)
|
||||
- [4×8× A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes)
|
||||
- [8× A100 AWQ](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization)
|
||||
- [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
|
||||
- [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
|
||||
- [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus)
|
||||
- [4× Atlas 800I A3 (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment)
|
||||
|
||||
**Blog references for large-scale deployment:**
|
||||
|
||||
- [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/))
|
||||
- [PD Disaggregation and Large-Scale Expert Parallelism on 96× H100](https://lmsys.org/blog/2025-05-05-large-scale-ep/)
|
||||
- [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/)
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (8x)
|
||||
- Model: DeepSeek-V3
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.7
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 81.27
|
||||
Total input tokens: 1972
|
||||
Total input text tokens: 1972
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 2784
|
||||
Total generated tokens (retokenized): 2774
|
||||
Request throughput (req/s): 0.12
|
||||
Input token throughput (tok/s): 24.27
|
||||
Output token throughput (tok/s): 34.26
|
||||
Peak output token throughput (tok/s): 65.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 58.52
|
||||
Concurrency: 1.00
|
||||
Accept length: 2.61
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8123.17
|
||||
Median E2E Latency (ms): 7982.65
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1080.76
|
||||
Median TTFT (ms): 1248.82
|
||||
P99 TTFT (ms): 1896.37
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.04
|
||||
Median TPOT (ms): 24.76
|
||||
P99 TPOT (ms): 32.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.41
|
||||
Median ITL (ms): 20.14
|
||||
P95 ITL (ms): 60.28
|
||||
P99 ITL (ms): 60.99
|
||||
Max ITL (ms): 61.49
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--ep 8 \
|
||||
--dp 8 \
|
||||
--enable-dp-attention \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- **Test Results:**
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 406.16
|
||||
Total input tokens: 301701
|
||||
Total input text tokens: 301701
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 188375
|
||||
Total generated tokens (retokenized): 187542
|
||||
Request throughput (req/s): 2.46
|
||||
Input token throughput (tok/s): 742.81
|
||||
Output token throughput (tok/s): 463.80
|
||||
Peak output token throughput (tok/s): 1299.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 1206.61
|
||||
Concurrency: 87.53
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 35552.98
|
||||
Median E2E Latency (ms): 21466.07
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1521.51
|
||||
Median TTFT (ms): 476.80
|
||||
P99 TTFT (ms): 8329.50
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 214.73
|
||||
Median TPOT (ms): 152.00
|
||||
P99 TPOT (ms): 1155.85
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 182.10
|
||||
Median ITL (ms): 79.18
|
||||
P95 ITL (ms): 398.60
|
||||
P99 ITL (ms): 1488.96
|
||||
Max ITL (ms): 43465.60
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
- DeepSeek-V3
|
||||
```text Output
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 32.450 s
|
||||
Output throughput: 614.211 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
cd sglang
|
||||
bash benchmark/mmlu/download_data.sh
|
||||
python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 8000
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
- DeepSeek-V3
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.800
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.928
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.928
|
||||
subject: college_biology, #q:144, acc: 0.965
|
||||
subject: college_chemistry, #q:100, acc: 0.670
|
||||
subject: college_computer_science, #q:100, acc: 0.840
|
||||
subject: college_mathematics, #q:100, acc: 0.800
|
||||
subject: college_medicine, #q:173, acc: 0.861
|
||||
Total latency: 58.339
|
||||
Average accuracy: 0.871
|
||||
```
|
||||
@@ -0,0 +1,989 @@
|
||||
---
|
||||
title: DeepSeek-V3.1
|
||||
metatags:
|
||||
description: "Deploy DeepSeek-V3.1 MoE model with SGLang - hybrid reasoning, improved tool calling, and agentic behavior for complex multi-step tasks."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture.
|
||||
|
||||
**[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases.
|
||||
|
||||
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { DeepSeekV31Deployment } from "/src/snippets/autoregressive/deepseek-v31-deployment.jsx";
|
||||
|
||||
<DeepSeekV31Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.
|
||||
|
||||
**Recommended GPU configurations by weight type:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Weight Type</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Supported Hardware</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>FP8</strong> (recommended)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>BF16</strong> (upcast from FP8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> The official DeepSeek-V3.1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.
|
||||
|
||||
**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (~10 min):
|
||||
```bash
|
||||
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --trust-remote-code
|
||||
```
|
||||
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
|
||||
|
||||
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
|
||||
|
||||
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
|
||||
|
||||
**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [Basic API Usage](../../../docs/get-started/quickstart)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3.1-Terminus \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.1-Terminus",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100.
|
||||
|
||||
To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15.
|
||||
|
||||
Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240.
|
||||
|
||||
I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36.
|
||||
|
||||
I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer.
|
||||
|
||||
So, 15% of 240 is 36.
|
||||
|
||||
The problem says "step by step," so I should present it clearly.
|
||||
=============== Content =================
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \).
|
||||
2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \).
|
||||
3. Multiply the decimal by 240: \( 0.15 \times 240 \).
|
||||
4. Perform the multiplication:
|
||||
- \( 0.15 \times 200 = 30 \)
|
||||
- \( 0.15 \times 40 = 6 \)
|
||||
- Add the results: \( 30 + 6 = 36 \).
|
||||
|
||||
Alternatively, you can find 15% by breaking it into parts:
|
||||
- 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \).
|
||||
- 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \).
|
||||
- Add 10% and 5%: \( 24 + 12 = 36 \).
|
||||
|
||||
Thus, 15% of 240 is 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3.1-Terminus \
|
||||
--tool-call-parser deepseekv31 \
|
||||
--reasoning-parser deepseek-v3 \
|
||||
--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.1-Terminus",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
extra_body = {"chat_template_kwargs": {"thinking": True}},
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide.
|
||||
|
||||
I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world.
|
||||
|
||||
The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you.
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
Please attach the code blocks below to the previous Python script.
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.1-Terminus",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "Currently, it is **22°C and sunny** in Beijing."
|
||||
```
|
||||
|
||||
#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)
|
||||
|
||||
DeepSeek-V3.1 shares the same architecture as DeepSeek-V3 and supports the same EAGLE-based MTP speculative decoding path. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the full configuration, tuning guidance, and `bench_speculative.py` reference. The `--speculative-num-steps`, `--speculative-eagle-topk`, and `--max-running-requests` recommendations apply equally to V3.1.
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (8x)
|
||||
- Model: DeepSeek-V3.1-Terminus
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.7
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3.1 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 106.24
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4201
|
||||
Request throughput (req/s): 0.09
|
||||
Input token throughput (tok/s): 57.43
|
||||
Output token throughput (tok/s): 39.72
|
||||
Peak output token throughput (tok/s): 43.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 97.15
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10620.29
|
||||
Median E2E Latency (ms): 8868.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 557.85
|
||||
Median TTFT (ms): 213.58
|
||||
P99 TTFT (ms): 1625.28
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 23.84
|
||||
Median TPOT (ms): 23.90
|
||||
P99 TPOT (ms): 24.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.90
|
||||
Median ITL (ms): 23.92
|
||||
P95 ITL (ms): 24.15
|
||||
P99 ITL (ms): 24.25
|
||||
Max ITL (ms): 25.44
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 107.71
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40625
|
||||
Request throughput (req/s): 0.74
|
||||
Input token throughput (tok/s): 368.28
|
||||
Output token throughput (tok/s): 378.84
|
||||
Peak output token throughput (tok/s): 508.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 747.12
|
||||
Concurrency: 13.72
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18473.65
|
||||
Median E2E Latency (ms): 19558.42
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 607.91
|
||||
Median TTFT (ms): 191.32
|
||||
P99 TTFT (ms): 2135.13
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 35.50
|
||||
Median TPOT (ms): 35.99
|
||||
P99 TPOT (ms): 43.62
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 35.10
|
||||
Median ITL (ms): 32.18
|
||||
P95 ITL (ms): 33.03
|
||||
P99 ITL (ms): 159.99
|
||||
Max ITL (ms): 453.99
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 207.65
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 251238
|
||||
Request throughput (req/s): 2.41
|
||||
Input token throughput (tok/s): 1203.15
|
||||
Output token throughput (tok/s): 1216.79
|
||||
Peak output token throughput (tok/s): 2100.00
|
||||
Peak concurrent requests: 106
|
||||
Total token throughput (tok/s): 2419.94
|
||||
Concurrency: 91.02
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 37800.20
|
||||
Median E2E Latency (ms): 35921.56
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 835.15
|
||||
Median TTFT (ms): 236.88
|
||||
P99 TTFT (ms): 2868.52
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 73.33
|
||||
Median TPOT (ms): 76.35
|
||||
P99 TPOT (ms): 97.63
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 73.30
|
||||
Median ITL (ms): 50.82
|
||||
P95 ITL (ms): 180.67
|
||||
P99 ITL (ms): 186.83
|
||||
Max ITL (ms): 1661.39
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 1097.29
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44313
|
||||
Request throughput (req/s): 0.01
|
||||
Input token throughput (tok/s): 5.56
|
||||
Output token throughput (tok/s): 40.52
|
||||
Peak output token throughput (tok/s): 43.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 46.08
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 109725.52
|
||||
Median E2E Latency (ms): 117748.67
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 156.67
|
||||
Median TTFT (ms): 156.19
|
||||
P99 TTFT (ms): 159.87
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.41
|
||||
Median TPOT (ms): 24.51
|
||||
P99 TPOT (ms): 24.96
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 24.65
|
||||
Median ITL (ms): 24.58
|
||||
P95 ITL (ms): 25.68
|
||||
P99 ITL (ms): 25.93
|
||||
Max ITL (ms): 29.80
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 775.02
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 317426
|
||||
Request throughput (req/s): 0.10
|
||||
Input token throughput (tok/s): 51.18
|
||||
Output token throughput (tok/s): 410.70
|
||||
Peak output token throughput (tok/s): 512.00
|
||||
Peak concurrent requests: 18
|
||||
Total token throughput (tok/s): 461.89
|
||||
Concurrency: 13.86
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 134236.65
|
||||
Median E2E Latency (ms): 135181.28
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 214.35
|
||||
Median TTFT (ms): 194.12
|
||||
P99 TTFT (ms): 300.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.72
|
||||
Median TPOT (ms): 34.00
|
||||
P99 TPOT (ms): 34.75
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.69
|
||||
Median ITL (ms): 33.71
|
||||
P95 ITL (ms): 34.50
|
||||
P99 ITL (ms): 34.92
|
||||
Max ITL (ms): 164.76
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 1231.97
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1296845
|
||||
Request throughput (req/s): 0.26
|
||||
Input token throughput (tok/s): 129.01
|
||||
Output token throughput (tok/s): 1056.05
|
||||
Peak output token throughput (tok/s): 1472.00
|
||||
Peak concurrent requests: 67
|
||||
Total token throughput (tok/s): 1185.07
|
||||
Concurrency: 56.17
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 216256.25
|
||||
Median E2E Latency (ms): 224192.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 317.68
|
||||
Median TTFT (ms): 235.28
|
||||
P99 TTFT (ms): 649.39
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 53.30
|
||||
Median TPOT (ms): 55.10
|
||||
P99 TPOT (ms): 56.58
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 53.13
|
||||
Median ITL (ms): 52.95
|
||||
P95 ITL (ms): 56.23
|
||||
P99 ITL (ms): 181.04
|
||||
Max ITL (ms): 208.61
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 114.47
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4194
|
||||
Request throughput (req/s): 0.09
|
||||
Input token throughput (tok/s): 366.39
|
||||
Output token throughput (tok/s): 36.87
|
||||
Peak output token throughput (tok/s): 42.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 403.26
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 11442.86
|
||||
Median E2E Latency (ms): 9508.87
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 883.78
|
||||
Median TTFT (ms): 481.38
|
||||
P99 TTFT (ms): 2217.45
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.93
|
||||
Median TPOT (ms): 25.05
|
||||
P99 TPOT (ms): 26.11
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.08
|
||||
Median ITL (ms): 25.08
|
||||
P95 ITL (ms): 26.18
|
||||
P99 ITL (ms): 26.28
|
||||
Max ITL (ms): 27.41
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 162.33
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41443
|
||||
Request throughput (req/s): 0.49
|
||||
Input token throughput (tok/s): 1848.27
|
||||
Output token throughput (tok/s): 256.70
|
||||
Peak output token throughput (tok/s): 467.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 2104.97
|
||||
Concurrency: 14.52
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 29456.89
|
||||
Median E2E Latency (ms): 27628.16
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1784.30
|
||||
Median TTFT (ms): 1347.21
|
||||
P99 TTFT (ms): 5384.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 53.65
|
||||
Median TPOT (ms): 52.09
|
||||
P99 TPOT (ms): 74.39
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 53.23
|
||||
Median ITL (ms): 34.52
|
||||
P95 ITL (ms): 35.81
|
||||
P99 ITL (ms): 513.25
|
||||
Max ITL (ms): 2865.73
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model deepseek-ai/DeepSeek-V3.1 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 282.55
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169081
|
||||
Request throughput (req/s): 1.13
|
||||
Input token throughput (tok/s): 4508.6
|
||||
Output token throughput (tok/s): 601.67
|
||||
Peak output token throughput (tok/s): 1216
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 5110.27
|
||||
Concurrency: 59.81
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 52810.32
|
||||
Median E2E Latency (ms): 50981.81
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 786.69
|
||||
Median TTFT (ms): 499.38
|
||||
P99 TTFT (ms): 2925.98
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 97.93
|
||||
Median TPOT (ms): 103.45
|
||||
P99 TPOT (ms): 157.84
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 98.11
|
||||
Median ITL (ms): 55.7
|
||||
P95 ITL (ms): 240.71
|
||||
P99 ITL (ms): 1114.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
**Test Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.959
|
||||
Invalid: 0.000
|
||||
Latency: 29.185 s
|
||||
Output throughput: 4854.672 token/s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,593 @@
|
||||
---
|
||||
title: DeepSeek-V4
|
||||
description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, and Pro."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
|
||||
|
||||
**NVIDIA GPUs**
|
||||
|
||||
A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000).
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
**AMD GPUs (ROCm)**
|
||||
|
||||
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
|
||||
|
||||
For example:
|
||||
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710`
|
||||
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.13.post1-rocm720-mi30x-20260623`
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710
|
||||
|
||||
docker run \
|
||||
--device=/dev/kfd --device=/dev/dri \
|
||||
--group-add video \
|
||||
--cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
|
||||
--shm-size 32g --ipc=host \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Note>
|
||||
For a runnable end-to-end example, see the [DeepSeek-V4-Flash demo notebook](https://github.com/sgl-project/sglang/blob/main/docs/demo/deepseek_v4_flash.ipynb).
|
||||
</Note>
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the per-hardware image from the <a href="#install">Install SGLang</a> panel above.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>, <code>NODE_RANK</code>, <code>NODE0_IP</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
The knobs come in two flavors:
|
||||
|
||||
- **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with `off` to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
|
||||
- **DeepSeek-V4 specific features** — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below).
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash refresh adds a checkpoint with a bundled DSpark draft head:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "30%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "15%"}} />
|
||||
<col style={{width: "40%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variant</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Total params</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Active (MoE)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash">DeepSeek-V4-Flash</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>284B</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731">DeepSeek-V4-Flash-0731</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>304</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>13B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro">DeepSeek-V4-Pro</a></strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>1.6T</strong></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>49B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>high-capacity: B200 / B300 (TP=8) · GB300 (TP=4) · H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H200 FP8 (2-node, TP=16) · H100 (2-node, TP=16)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling.
|
||||
|
||||
**Highlights:** hybrid CSA + HCA attention (~27% inference FLOPs / ~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar.
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=1.0`.
|
||||
|
||||
**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
{/* TODO: expand this section as more recipes are validated end-to-end. */}
|
||||
|
||||
**Concurrency & DeepEP dispatch buffer**
|
||||
|
||||
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together.
|
||||
|
||||
The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.
|
||||
|
||||
**Speculative decoding**
|
||||
|
||||
The original Flash and Pro recipes use EAGLE. Flash Official (0731) uses the bundled DSpark draft head; see [DSpark](#3-4-dspark-speculative-decoding) for its launch and tuning notes.
|
||||
|
||||
For the original Flash and Pro checkpoints:
|
||||
|
||||
- `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
|
||||
- `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
|
||||
- `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
|
||||
- MTP runs on the v2 speculative path.
|
||||
|
||||
**Compressed attention state dtype**
|
||||
|
||||
DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path:
|
||||
|
||||
```bash Command
|
||||
SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Flash \
|
||||
<other args>
|
||||
```
|
||||
|
||||
This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior.
|
||||
|
||||
**EPLB + Waterfill (Experimental)**
|
||||
|
||||
For recorded/static EPLB reproduction, first record an expert-distribution file by following
|
||||
[Capture expert selection distribution in MoE models](../../../docs/basic_usage/native_api.mdx#capture-expert-selection-distribution-in-moe-models).
|
||||
For reproduction runs, use the generated `expert_distribution_recorder_*.pt` as
|
||||
the initial expert location. **Please checkout to latest main branch for this feature.**
|
||||
|
||||
For non-PD reproduction, use:
|
||||
|
||||
```bash Command
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
|
||||
--enable-waterfill
|
||||
```
|
||||
|
||||
For PD-Disagg reproduction, use `normal` mode on the prefill server and
|
||||
`low_latency` mode on the decode server. Add the same `--init-expert-location`
|
||||
flag to both commands:
|
||||
|
||||
```bash Command
|
||||
# prefill
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal \
|
||||
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
|
||||
--enable-waterfill
|
||||
|
||||
# decode
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
|
||||
--enable-waterfill
|
||||
```
|
||||
|
||||
You can also add `--ep-num-redundant-experts` and `--eplb-algorithm` to customize
|
||||
EPLB placement.
|
||||
|
||||
Waterfill also supports MegaMOE. Use `--moe-a2a-backend megamoe
|
||||
--enable-waterfill` to keep the MegaMOE backend while applying Waterfill to the
|
||||
fused shared expert slot.
|
||||
|
||||
**FP4 Indexer (Experimental)**
|
||||
|
||||
DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.
|
||||
|
||||
```bash Command
|
||||
# Please use the latest main branch for this feature.
|
||||
sglang serve \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Flash \
|
||||
--tp 4 \
|
||||
--moe-runner-backend flashinfer_mxfp4 \
|
||||
--enable-deepseek-v4-fp4-indexer
|
||||
```
|
||||
|
||||
**NVFP4 Hybrid Checkpoints**
|
||||
|
||||
The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and
|
||||
[`nvidia/DeepSeek-V4-Flash-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4) checkpoints
|
||||
quantize MoE experts to **NVFP4** while keeping attention and dense layers in
|
||||
**FP8**. It requires `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided.
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path nvidia/DeepSeek-V4-Flash-NVFP4 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
Requires Blackwell (SM100+). The MTP layer in this checkpoint stays
|
||||
MXFP4-packed and is routed through the `Mxfp4FlashinferTrtllmMoEMethod` path
|
||||
automatically.
|
||||
|
||||
<a id="hopper-note" />
|
||||
|
||||
**Hopper (H100 / H200) note**
|
||||
|
||||
Two options are available for running DeepSeek-V4 on Hopper:
|
||||
|
||||
- **Original FP4 checkpoints** — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
|
||||
- **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).
|
||||
|
||||
On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below.
|
||||
|
||||
PD-Disagg recipes on H200 may require `docker run --privileged --ulimit memlock=-1`
|
||||
(or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) so mooncake
|
||||
can discover the IB HCAs; without IB exposure mooncake silently falls back to
|
||||
TCP, which can lead to garbled KV transfer on large checkpoints.
|
||||
|
||||
**RTX PRO 6000 (SM120 / Blackwell Desktop) note**
|
||||
|
||||
RTX PRO 6000 (96 GB) runs **Flash only** with the FlashInfer MXFP4 MoE runner.
|
||||
V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes.
|
||||
HiCache and MegaMoE are **not** supported on RTX PRO 6000.
|
||||
|
||||
**AMD (MI300X / MI355X) note**
|
||||
|
||||
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
|
||||
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
|
||||
- **TP / DP setting** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP**, which additionally needs `--dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000`.
|
||||
- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.
|
||||
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
|
||||
|
||||
**MegaMoE**
|
||||
|
||||
MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput
|
||||
on MoE layers. To enable it, use the **MegaMoE** chip in the Playground
|
||||
below — the playground will swap `--moe-a2a-backend deepep` for
|
||||
`--moe-a2a-backend megamoe` and add the relevant env vars automatically.
|
||||
|
||||
Two variants are exposed:
|
||||
- **W4A8** — default MegaMoE kernel (FP4 weights, FP8 activations).
|
||||
- **W4A4** — adds `SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1` and
|
||||
`SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1` to run the custom W4A4
|
||||
kernel (FP4 activations). Higher throughput with negligible accuracy drop
|
||||
(~89.5 GPQA on Pro).
|
||||
|
||||
Notes:
|
||||
- The W4A8 / W4A4 variants above are **Blackwell-only** (B200 / B300 / GB200 / GB300). On **Hopper (SM90, H100 / H200)** use the all-FP8 MegaMoE path described below instead.
|
||||
- MegaMoE is **only wired into the `high-throughput` recipe** on Blackwell (per [sgl-project/sglang#26451](https://github.com/sgl-project/sglang/pull/26451)). The chip is hidden on `low-latency` and `balanced` — switch to `high-throughput` to expose it.
|
||||
- When running MegaMoE, don't set `--moe-runner-backend` manually.
|
||||
- Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).
|
||||
|
||||
**SM90 (Hopper) FP8 MegaMoE (Experimental)**
|
||||
|
||||
On SM90 (Hopper, H100 / H200), the all-FP8 MegaMoE path routes MoE through the
|
||||
DeepGEMM `mega_moe` runner for higher long-context / large-decode throughput on
|
||||
the FP8 checkpoints. Unlike the Blackwell W4A8 / W4A4 variants above, experts
|
||||
stay in **FP8** — keep `SGLANG_DSV4_FP4_EXPERTS=0`. It requires a `sgl-deep-gemm`
|
||||
build with SM90 FP8 MegaMoE support. **Please use the latest image for this
|
||||
feature.**
|
||||
|
||||
Enable the MegaMoE path with `--moe-a2a-backend megamoe` — or equivalently set
|
||||
`SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1`, which auto-configures the same backend:
|
||||
|
||||
```bash Command
|
||||
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \
|
||||
SGLANG_DSV4_FP4_EXPERTS=0 \
|
||||
sglang serve \
|
||||
--model-path sgl-project/DeepSeek-V4-Flash-FP8 \
|
||||
--tp 8 \
|
||||
--moe-a2a-backend megamoe \
|
||||
--chunked-prefill-size 4096
|
||||
```
|
||||
|
||||
`SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` caps the number of tokens
|
||||
the MegaMoE path processes per rank (i.e. per GPU); the MegaMoE path is only used
|
||||
for batches at or below this cap. The right value depends on your parallelism /
|
||||
token-split scheme, and larger values reserve more HBM.
|
||||
|
||||
**GB300 PD-Disagg cross-pod MNNVL**
|
||||
|
||||
On some GB300 clusters with cross-pod KV transfer over NVLink, mooncake may
|
||||
fail with `nvlink_transport.cpp:497 Requested address ... not found!`. If
|
||||
this happens, prepend `MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`
|
||||
to both prefill and decode `sglang serve` commands.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Enable the `deepseek-v4` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.
|
||||
|
||||
<Accordion title="Streaming with Thinking Process (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.</think>To find 15% of 240, follow these steps:
|
||||
|
||||
**Step 1: Understand the meaning of percent**
|
||||
"Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \).
|
||||
|
||||
**Step2: Convert the percentage to a decimal or fraction**
|
||||
\( 15\% = \frac{15}{100} = 0.15 \)
|
||||
|
||||
**Step3: Multiply by the given number**
|
||||
Multiply the decimal form by 240:
|
||||
\( 0.15 \times 240 \)
|
||||
|
||||
**Step4: Perform the multiplication**
|
||||
\( 0.15 \times 240 = 36 \)
|
||||
|
||||
**Answer:** 15% of 240 is **36**.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `deepseekv4` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.
|
||||
|
||||
<Accordion title="Python Example with Thinking Process">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if getattr(delta, "tool_calls", None):
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {"name": None, "arguments": ""}
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]["name"] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default.</think>
|
||||
|
||||
<｜DSML｜tool_calls>
|
||||
<｜DSML｜invoke name="get_weather">
|
||||
<｜DSML｜parameter name="location" string="true">Beijing</｜DSML｜parameter>
|
||||
</｜DSML｜invoke>
|
||||
</｜DSML｜tool_calls>
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers.
|
||||
|
||||
To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**:
|
||||
|
||||
- **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only.
|
||||
- **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file` / `mooncake` / `hf3fs` / `nixl`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).
|
||||
|
||||
For AMD devices,
|
||||
- **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. Use `direct` IO backend + `page_first_direct` or `layer-first` mem-layout.
|
||||
- **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).
|
||||
|
||||
The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow.
|
||||
|
||||
For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache).
|
||||
|
||||
### 3.4 DSpark (Speculative Decoding)
|
||||
|
||||
Flash Official (0731) bundles a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`.
|
||||
|
||||
Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint.
|
||||
|
||||
The verified 4×GB300 FP4 low-latency command is:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
|
||||
--tp 4 \
|
||||
--moe-runner-backend flashinfer_mxfp4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
--mem-fraction-static 0.90 \
|
||||
--chunked-prefill-size 4096 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
Keep `--mem-fraction-static 0.90` on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16.
|
||||
|
||||
**Tune proposed draft tokens.** `--speculative-dspark-block-size N` asks DSpark to propose `N` tokens per step; the target verifies a window of `N + 1`. If the flag is omitted, SGLang reads the value from the checkpoint. The current 0731 checkpoint resolves to five proposed tokens, which is the verified default. Use the **DSpark Proposed Draft Tokens** slider in the [Playground](#playground) to sweep one through five.
|
||||
|
||||
Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone.
|
||||
|
||||
For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.
|
||||
|
||||
DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; turn DSpark off before selecting a prefill or decode role. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Ernie4.5-VL
|
||||
metatags:
|
||||
description: "Deploy Ernie4.5-VL vision-language model with SGLang - community contribution guide for Baidu's multimodal model."
|
||||
---
|
||||
|
||||
## 📝 Community Contribution Welcome
|
||||
|
||||
This guide is currently under development. We welcome community contributions!
|
||||
|
||||
If you have experience deploying **Ernie4.5-VL** with SGLang, please help us complete this documentation.
|
||||
|
||||
## 🚀 How to Contribute
|
||||
|
||||
```shell Command
|
||||
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
|
||||
cd sglang-cookbook
|
||||
git checkout -b add-ernie4-5-vl-guide
|
||||
# Edit this file and submit a PR
|
||||
```
|
||||
|
||||
## 📚 Reference
|
||||
|
||||
- [GLM-4.6V](../GLM/GLM-4.6V)
|
||||
|
||||
---
|
||||
|
||||
**Let's build this together!** 🌟
|
||||
@@ -0,0 +1,696 @@
|
||||
---
|
||||
title: Ernie4.5
|
||||
metatags:
|
||||
description: "Deploy Ernie4.5 with SGLang - community contribution guide for Baidu's Ernie 4.5 model deployment."
|
||||
---
|
||||
|
||||
import { Ernie45Deployment } from '/src/snippets/autoregressive/ernie-45-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
The **ERNIE-4.5** series is a family of large language models developed by Baidu. ERNIE (Enhanced Representation through Knowledge Integration) 4.5 represents an advanced version of the ERNIE series, optimized for general-purpose tasks and conversational scenarios.
|
||||
|
||||
ERNIE-4.5 delivers advanced features as below:
|
||||
- **Heterogeneous Modality Structure**: MoE architecture that supports parameter sharing across modalities while allowing dedicated parameters for each individual modality, enhancing multimodal understanding without compromising, and even improving, performance on text-related tasks.
|
||||
- **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing.
|
||||
- **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations.
|
||||
- **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions.
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Ernie45Deployment />
|
||||
|
||||
## 4. API Usage
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
The following example demonstrates deployment using ERNIE-4.5-21B-A3B-PT.
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--tp 1
|
||||
```
|
||||
|
||||
**Basic Python Client Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/ERNIE-4.5-21B-A3B-PT",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is artificial intelligence?"}
|
||||
],
|
||||
temperature=1.0,
|
||||
top_p=0.95,
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
```text Output
|
||||
**Artificial Intelligence (AI)** is the simulation of human intelligence processes by machines, particularly computer systems. These processes include **learning** (acquiring information and rules for using the information), **reasoning** (using rules to reach approximate or definite conclusions), and **self-correction**. AI encompasses a wide range of techniques, algorithms, and methodologies designed to enable machines to perform tasks that typically require human intelligence.
|
||||
|
||||
### Key Characteristics of AI:
|
||||
...
|
||||
|
||||
### In Summary:
|
||||
AI represents a transformative force with the potential to revolutionize industries and enhance human capabilities. However, its development requires careful consideration of ethical, legal, and social implications to ensure that it benefits society as a whole. As AI continues to evolve, ongoing dialogue among stakeholders will be crucial to balancing innovation with responsibility.
|
||||
```
|
||||
|
||||
**Streaming Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/ERNIE-4.5-21B-A3B-PT",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain quantum computing in simple terms."}
|
||||
],
|
||||
temperature=1.0,
|
||||
top_p=0.95,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Sure! Here’s a simple explanation of quantum computing:
|
||||
|
||||
### **Quantum Computing: Making Computers Super Fast (But Weird) Using Quantum Rules**
|
||||
|
||||
1. **Classic vs. Quantum Computers**
|
||||
- **Normal computers** use **bits** (0s and 1s) to store and process information.
|
||||
- **Quantum computers** use **qubits** (short for quantum bits). Unlike bits, qubits can be **0, 1, or both at the same time** (this is called **superposition**).
|
||||
|
||||
2. **Superposition: The Magic Behind Speed**
|
||||
- A single qubit can represent **0 and 1 simultaneously**, like a coin spinning in the air.
|
||||
- Many qubits working together (in something called **quantum parallelism**) can **check multiple possibilities at once**, making quantum computers much faster for certain problems.
|
||||
|
||||
3. **Entanglement: Making Qubits Link**
|
||||
- When qubits are **entangled**, their states are linked—changing one instantly affects the other, no matter how far apart they are (this is called **spooky action at a distance** by Einstein).
|
||||
- Entanglement allows quantum computers to process information in **very efficient ways**.
|
||||
|
||||
4. **What Quantum Computers Are Good At**
|
||||
- **Cracking encryption** (like RSA).
|
||||
- **Factoring large numbers** (used in encryption and cryptography).
|
||||
- **Searching unsorted databases** (way faster than classical computers).
|
||||
- **Simulating quantum systems** (like molecules for drug discovery).
|
||||
- **Optimizing problems** (like logistics or finance).
|
||||
|
||||
5. **Challenges & Current State**
|
||||
- Qubits are **fragile** and easily disturbed (called **decoherence**).
|
||||
- Engineers are working to keep qubits stable long enough to do useful calculations.
|
||||
- Today’s quantum computers are **small and experimental**, but the goal is to build powerful ones that outperform classical supercomputers.
|
||||
|
||||
### **Final Thought**
|
||||
Quantum computing isn’t just a faster calculator—it’s a **new way of thinking about problems** using the weird laws of physics. While still new, it has the potential to revolutionize fields like medicine, AI, and cybersecurity.
|
||||
|
||||
Would you like an example of how a quantum computer might solve a problem? 😊
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X GPU (1x)
|
||||
- Model: ERNIE-4.5-21B-A3B-PT
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--tp 1
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 58.72
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4219
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 103.90
|
||||
Output token throughput (tok/s): 71.87
|
||||
Peak output token throughput (tok/s): 245.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 175.77
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5869.86
|
||||
Median E2E Latency (ms): 1870.80
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4152.58
|
||||
Median TTFT (ms): 36.81
|
||||
P99 TTFT (ms): 37498.23
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.07
|
||||
Median TPOT (ms): 4.09
|
||||
P99 TPOT (ms): 4.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.08
|
||||
Median ITL (ms): 4.08
|
||||
P95 ITL (ms): 4.14
|
||||
P99 ITL (ms): 4.20
|
||||
Max ITL (ms): 4.67
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 34.30
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40773
|
||||
Request throughput (req/s): 2.33
|
||||
Input token throughput (tok/s): 1156.62
|
||||
Output token throughput (tok/s): 1189.77
|
||||
Peak output token throughput (tok/s): 1392.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 2346.39
|
||||
Concurrency: 14.14
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6060.62
|
||||
Median E2E Latency (ms): 6496.70
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 78.90
|
||||
Median TTFT (ms): 45.90
|
||||
P99 TTFT (ms): 234.33
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 11.99
|
||||
Median TPOT (ms): 12.16
|
||||
P99 TPOT (ms): 14.81
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.75
|
||||
Median ITL (ms): 11.48
|
||||
P95 ITL (ms): 12.24
|
||||
P99 ITL (ms): 34.85
|
||||
Max ITL (ms): 105.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 66.63
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252449
|
||||
Request throughput (req/s): 7.50
|
||||
Input token throughput (tok/s): 3749.79
|
||||
Output token throughput (tok/s): 3792.28
|
||||
Peak output token throughput (tok/s): 4902.00
|
||||
Peak concurrent requests: 113
|
||||
Total token throughput (tok/s): 7542.06
|
||||
Concurrency: 90.33
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 12036.90
|
||||
Median E2E Latency (ms): 11782.16
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 104.86
|
||||
Median TTFT (ms): 84.62
|
||||
P99 TTFT (ms): 297.85
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 23.89
|
||||
Median TPOT (ms): 24.62
|
||||
P99 TPOT (ms): 26.91
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.66
|
||||
Median ITL (ms): 20.48
|
||||
P95 ITL (ms): 45.57
|
||||
P99 ITL (ms): 54.31
|
||||
Max ITL (ms): 185.12
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Reasoning Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 185.11
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44423
|
||||
Request throughput (req/s): 0.05
|
||||
Input token throughput (tok/s): 32.96
|
||||
Output token throughput (tok/s): 240.19
|
||||
Peak output token throughput (tok/s): 245.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 273.15
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18508.84
|
||||
Median E2E Latency (ms): 19866.81
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 32.59
|
||||
Median TTFT (ms): 32.14
|
||||
P99 TTFT (ms): 38.58
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.13
|
||||
Median TPOT (ms): 4.13
|
||||
P99 TPOT (ms): 4.20
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.16
|
||||
Median ITL (ms): 4.12
|
||||
P95 ITL (ms): 4.31
|
||||
P99 ITL (ms): 4.36
|
||||
Max ITL (ms): 7.28
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 263.48
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 317984
|
||||
Request throughput (req/s): 0.30
|
||||
Input token throughput (tok/s): 150.55
|
||||
Output token throughput (tok/s): 1208.09
|
||||
Peak output token throughput (tok/s): 1408.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 1358.64
|
||||
Concurrency: 14.35
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 47249.55
|
||||
Median E2E Latency (ms): 47828.67
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 62.77
|
||||
Median TTFT (ms): 57.10
|
||||
P99 TTFT (ms): 93.70
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 11.92
|
||||
Median TPOT (ms): 12.09
|
||||
P99 TPOT (ms): 12.50
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.86
|
||||
Median ITL (ms): 12.04
|
||||
P95 ITL (ms): 12.68
|
||||
P99 ITL (ms): 13.61
|
||||
Max ITL (ms): 39.94
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 428.30
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1299877
|
||||
Request throughput (req/s): 0.75
|
||||
Input token throughput (tok/s): 371.09
|
||||
Output token throughput (tok/s): 3037.63
|
||||
Peak output token throughput (tok/s): 3880.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 3408.73
|
||||
Concurrency: 57.08
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 76392.58
|
||||
Median E2E Latency (ms): 79698.73
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 92.79
|
||||
Median TTFT (ms): 78.71
|
||||
P99 TTFT (ms): 168.89
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.81
|
||||
Median TPOT (ms): 19.15
|
||||
P99 TPOT (ms): 19.81
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.77
|
||||
Median ITL (ms): 18.77
|
||||
P95 ITL (ms): 19.86
|
||||
P99 ITL (ms): 42.08
|
||||
Max ITL (ms): 74.36
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.3 Summarization Scenario Benchmark
|
||||
|
||||
##### 5.1.3.1 Low Concurrency
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 18.59
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4216
|
||||
Request throughput (req/s): 0.54
|
||||
Input token throughput (tok/s): 2256.43
|
||||
Output token throughput (tok/s): 227.04
|
||||
Peak output token throughput (tok/s): 245.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 2483.46
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1856.72
|
||||
Median E2E Latency (ms): 1513.87
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 86.66
|
||||
Median TTFT (ms): 72.30
|
||||
P99 TTFT (ms): 167.13
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.19
|
||||
Median TPOT (ms): 4.22
|
||||
P99 TPOT (ms): 4.30
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.20
|
||||
Median ITL (ms): 4.23
|
||||
P95 ITL (ms): 4.34
|
||||
P99 ITL (ms): 4.42
|
||||
Max ITL (ms): 5.68
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.2 Medium Concurrency
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 40.25
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41646
|
||||
Request throughput (req/s): 1.99
|
||||
Input token throughput (tok/s): 7454.72
|
||||
Output token throughput (tok/s): 1035.37
|
||||
Peak output token throughput (tok/s): 1310.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 8490.09
|
||||
Concurrency: 14.37
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7229.56
|
||||
Median E2E Latency (ms): 7578.95
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 137.38
|
||||
Median TTFT (ms): 122.59
|
||||
P99 TTFT (ms): 485.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.04
|
||||
Median TPOT (ms): 14.24
|
||||
P99 TPOT (ms): 20.77
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.64
|
||||
Median ITL (ms): 12.36
|
||||
P95 ITL (ms): 14.72
|
||||
P99 ITL (ms): 57.39
|
||||
Max ITL (ms): 411.31
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.3 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 78.33
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169888
|
||||
Request throughput (req/s): 4.09
|
||||
Input token throughput (tok/s): 16262.33
|
||||
Output token throughput (tok/s): 2170.20
|
||||
Peak output token throughput (tok/s): 3005.00
|
||||
Peak concurrent requests: 73
|
||||
Total token throughput (tok/s): 18432.53
|
||||
Concurrency: 58.79
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 14392.52
|
||||
Median E2E Latency (ms): 14460.70
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 184.82
|
||||
Median TTFT (ms): 155.24
|
||||
P99 TTFT (ms): 379.82
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 26.97
|
||||
Median TPOT (ms): 28.31
|
||||
P99 TPOT (ms): 33.61
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 26.79
|
||||
Median ITL (ms): 20.55
|
||||
P95 ITL (ms): 47.55
|
||||
P99 ITL (ms): 145.64
|
||||
Max ITL (ms): 287.62
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py \
|
||||
--num-shots 8 \
|
||||
--num-questions 1316 \
|
||||
--parallel 1316
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
- ERNIE-4.5-21B-A3B-PT
|
||||
```
|
||||
Accuracy: 0.865
|
||||
Invalid: 0.000
|
||||
Latency: 21.669 s
|
||||
Output throughput: 10359.790 token/s
|
||||
```
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: Chroma-1.0
|
||||
metatags:
|
||||
description: "Deploy Chroma-1.0 end-to-end speech conversation model with SGLang - real-time speech generation, voice cloning, and speech reasoning."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Chroma-1.0](https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma) is an open-source end-to-end speech conversation model developed by FlashLabs, focusing on the following core capabilities:
|
||||
|
||||
- **Real-time Speech Generation**: Supports low-latency speech synthesis, suitable for real-time conversational scenarios.
|
||||
- **Customized Voice Cloning**: Capable of cloning and replicating specific speaker voice characteristics.
|
||||
- **End-to-End Architecture**: Provides a complete processing workflow from speech to speech.
|
||||
- **Speech Reasoning**: Equipped with reasoning capabilities to understand and process speech content.
|
||||
|
||||
## 2. Architecture Overview
|
||||
|
||||
**Chroma-1.0** utilizes a hybrid serving architecture rather than a direct SGLang deployment. This design choice is driven by:
|
||||
|
||||
1. **Complex Model Architecture**: The end-to-end speech processing pipeline involves specialized components that go beyond standard text generation loops.
|
||||
2. **KV Cache & State Management**: The model requires custom handling of KV caches that differs from standard implementations.
|
||||
3. **Batching Limitations**: The current implementation supports a batch size of 1, meaning SGLang's advanced continuous batching capabilities are not yet fully applicable.
|
||||
|
||||
Therefore, you will start the **FlashLabs Server**, which manages the overall workflow and selectively leverages SGLang for specific inference components where supported.
|
||||
|
||||
- **Outer Layer**: FlashLabs Server (Handles Audio I/O, State, and Model Logic)
|
||||
- **Inner Engine**: SGLang Instance (Utilized for specific acceleration where applicable)
|
||||
|
||||
## 3. Installation & Setup
|
||||
|
||||
We recommend following these steps to set up the environment and prepare the model.
|
||||
|
||||
### Step 1: Get the Docker Image
|
||||
|
||||
Pull the official pre-built image from Docker Hub to ensure all dependencies are correctly configured.
|
||||
|
||||
```bash Command
|
||||
docker pull flashlabs/chroma:latest
|
||||
```
|
||||
|
||||
### Step 2: Download Model Weights
|
||||
|
||||
Download the **Chroma-4B** weights from Hugging Face. You can choose one of the following methods:
|
||||
|
||||
**Method 1: Using Python (Recommended)**
|
||||
|
||||
```bash Command
|
||||
huggingface-cli download FlashLabs/Chroma-4B --local-dir Chroma-4B
|
||||
```
|
||||
|
||||
**Method 2: Using Git Clone**
|
||||
|
||||
Make sure you have Git LFS installed before cloning.
|
||||
|
||||
```bash Command
|
||||
# Install Git LFS first
|
||||
git lfs install
|
||||
|
||||
# Clone the repository
|
||||
git clone https://huggingface.co/FlashLabs/Chroma-4B Chroma-4B
|
||||
```
|
||||
|
||||
### Step 3: Download Chroma Codes (SGLang version)
|
||||
|
||||
```bash Command
|
||||
git clone https://github.com/FlashLabs-AI-Corp/Chroma-SGLang.git
|
||||
|
||||
cd Chroma-SGLang
|
||||
```
|
||||
|
||||
### Step 4: Run the Server
|
||||
|
||||
```bash Command
|
||||
docker run -d \
|
||||
--gpus all \
|
||||
-p 8000:8000 \
|
||||
-w /app/Chroma-SGLang \
|
||||
-v "your_Chroma-SGLang_path":/app/Chroma-SGLang \
|
||||
-v "your_chroma_path":/model \
|
||||
-e CHROMA_MODEL_PATH=/model \
|
||||
-e DP_SIZE="1" \
|
||||
flashlabs/chroma:latest \
|
||||
/opt/conda/bin/python -m uvicorn api_server:app \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--workers 1
|
||||
```
|
||||
|
||||
or run simply the following one line command
|
||||
|
||||
```bash Command
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
## 5. Client Usage Example
|
||||
|
||||
Once the server is running, you can interact with it using HTTP requests.
|
||||
|
||||
### Python Client
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
import base64
|
||||
|
||||
url = "http://localhost:8000/v1/chat/completions"
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
payload = {
|
||||
"model": "chroma",
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are Chroma, a voice agent developed by FlashLabs."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "audio", "audio": "assets/question_audio.wav"}
|
||||
]
|
||||
}
|
||||
],
|
||||
"max_tokens": 1000,
|
||||
"return_audio": True
|
||||
}
|
||||
|
||||
response = requests.post(url, json=payload, headers=headers)
|
||||
result = response.json()
|
||||
|
||||
if result.get("audio"):
|
||||
audio_data = base64.b64decode(result["audio"])
|
||||
with open("output.wav", "wb") as f:
|
||||
f.write(audio_data)
|
||||
print("Audio saved to output.wav")
|
||||
```
|
||||
|
||||
### OpenAI SDK Compatible Example
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="dummy",
|
||||
base_url="http://localhost:8000/v1"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="chroma",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "audio", "audio": "assets/question_audio.wav"}
|
||||
]
|
||||
}
|
||||
],
|
||||
extra_body={
|
||||
"prompt_text": "I have not... I'm so exhausted, I haven't slept in a very long time. It could be because... Well, I used our... Uh, I'm, I just use... This is what I use every day. I use our cleanser every day, I use serum in the morning and then the moistu- daily moisturizer. That's what I use every morning.",
|
||||
"prompt_audio": "assets/ref_audio.wav",
|
||||
"return_audio": True
|
||||
}
|
||||
)
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
### CLI (cURL)
|
||||
|
||||
```bash Command
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "chroma",
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are Chroma, a voice agent developed by FlashLabs."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "audio",
|
||||
"audio": "assets/question_audio.wav"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"max_tokens": 1000,
|
||||
"return_audio": true
|
||||
}' | jq -r '.audio' | base64 -d > output.wav
|
||||
```
|
||||
@@ -0,0 +1,516 @@
|
||||
---
|
||||
title: GLM-4.5
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5 with SGLang on AMD GPUs - advanced reasoning, function calling, BF16/FP8 quantization options."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM45Deployment } from "/src/snippets/autoregressive/glm-45-deployment.jsx";
|
||||
|
||||
<GLM45Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.5 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
|
||||
- Model: GLM-4.5
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.5 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
@@ -0,0 +1,582 @@
|
||||
---
|
||||
title: GLM-4.5V
|
||||
metatags:
|
||||
description: "Deploy GLM-4.5V vision-language model with SGLang - SOTA multimodal performance, 64K context, image reasoning and video understanding."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
|
||||
|
||||
GLM-4.5V introduces several key features:
|
||||
|
||||
- **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection.
|
||||
- **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences.
|
||||
- **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces.
|
||||
- **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context.
|
||||
- **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
|
||||
|
||||
import { GLM45VDeployment } from "/src/snippets/autoregressive/glm-45v-deployment.jsx";
|
||||
|
||||
<GLM45VDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||||
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`.
|
||||
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
|
||||
- **Hardware Notes:**
|
||||
- **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
|
||||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||||
- **Additional Multimodal Parameters:**
|
||||
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
|
||||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
|
||||
|
||||
**Example with full multimodal optimizations:**
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
GLM-4.5V supports both image and video inputs. Here's a basic example with image input:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe this image in detail."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.37s
|
||||
Generated text: Auntie Anne's
|
||||
|
||||
CINNAMON SUGAR
|
||||
1 x 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
```
|
||||
|
||||
**Multi-Image Input Example:**
|
||||
|
||||
GLM-4.5V can process multiple images in a single request for comparison or analysis:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.86s
|
||||
Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot).
|
||||
```
|
||||
|
||||
**Video Input Example:**
|
||||
|
||||
GLM-4.5V supports video understanding by processing video URLs:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe what happens in this video."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- For video processing, ensure you have sufficient context length configured (up to 64K tokens)
|
||||
- Video processing may require more memory; adjust `--mem-fraction-static` accordingly
|
||||
- You can also provide local file paths using `file://` protocol
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 3.89s
|
||||
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
|
||||
```
|
||||
|
||||
#### 4.2.2 Thinking Mode
|
||||
|
||||
GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
**Disable Thinking Mode:**
|
||||
|
||||
To disable thinking mode for a specific request:
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[{"role": "user", "content": "What is the capital of France?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
```
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
GLM-4.5V supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.5V \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Beyond enabling/disabling the full reasoning mode (section 4.2.2), you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.5V",
|
||||
messages=[{"role": "user", "content": "Describe this image briefly."}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.1.1 MMMU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
```text Output
|
||||
Benchmark time: 616.6163094160147
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.867, 'num': 30},
|
||||
'Agriculture': {'acc': 0.567, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.667, 'num': 30},
|
||||
'Art': {'acc': 0.667, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.9, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.8, 'num': 30},
|
||||
'Biology': {'acc': 0.6, 'num': 30},
|
||||
'Chemistry': {'acc': 0.533, 'num': 30},
|
||||
'Clinical_Medicine': {'acc': 0.667, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.8, 'num': 30},
|
||||
'Design': {'acc': 0.867, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30},
|
||||
'Economics': {'acc': 0.833, 'num': 30},
|
||||
'Electronics': {'acc': 0.433, 'num': 30},
|
||||
'Energy_and_Power': {'acc': 0.733, 'num': 30},
|
||||
'Finance': {'acc': 0.767, 'num': 30},
|
||||
'Geography': {'acc': 0.667, 'num': 30},
|
||||
'History': {'acc': 0.8, 'num': 30},
|
||||
'Literature': {'acc': 0.9, 'num': 30},
|
||||
'Manage': {'acc': 0.733, 'num': 30},
|
||||
'Marketing': {'acc': 0.9, 'num': 30},
|
||||
'Materials': {'acc': 0.567, 'num': 30},
|
||||
'Math': {'acc': 0.8, 'num': 30},
|
||||
'Mechanical_Engineering': {'acc': 0.767, 'num': 30},
|
||||
'Music': {'acc': 0.3, 'num': 30},
|
||||
'Overall': {'acc': 0.732, 'num': 900},
|
||||
'Overall-Art and Design': {'acc': 0.683, 'num': 120},
|
||||
'Overall-Business': {'acc': 0.82, 'num': 150},
|
||||
'Overall-Health and Medicine': {'acc': 0.787, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120},
|
||||
'Overall-Science': {'acc': 0.707, 'num': 150},
|
||||
'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210},
|
||||
'Pharmacy': {'acc': 0.9, 'num': 30},
|
||||
'Physics': {'acc': 0.933, 'num': 30},
|
||||
'Psychology': {'acc': 0.767, 'num': 30},
|
||||
'Public_Health': {'acc': 0.9, 'num': 30},
|
||||
'Sociology': {'acc': 0.667, 'num': 30}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.732
|
||||
```
|
||||
@@ -0,0 +1,914 @@
|
||||
---
|
||||
title: GLM-4.6
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6 with SGLang - 200K context window, superior coding, advanced reasoning, and enhanced agentic capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows:
|
||||
|
||||
- **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks.
|
||||
- **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages.
|
||||
- **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability.
|
||||
- **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks.
|
||||
- **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios.
|
||||
|
||||
For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM46Deployment } from "/src/snippets/autoregressive/glm-46-deployment.jsx";
|
||||
|
||||
<GLM46Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.6 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"🔧 Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
|
||||
- Model: GLM-4.6
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 63.82
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4209
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 95.60
|
||||
Output token throughput (tok/s): 65.97
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 161.57
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6379.24
|
||||
Median E2E Latency (ms): 5085.00
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 155.57
|
||||
Median TTFT (ms): 149.79
|
||||
P99 TTFT (ms): 207.69
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.81
|
||||
Median TPOT (ms): 14.80
|
||||
P99 TPOT (ms): 14.84
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.82
|
||||
Median ITL (ms): 14.82
|
||||
P95 ITL (ms): 15.17
|
||||
P99 ITL (ms): 15.36
|
||||
Max ITL (ms): 25.05
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 72.06
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40725
|
||||
Total generated tokens (retokenized): 40672
|
||||
Request throughput (req/s): 1.11
|
||||
Input token throughput (tok/s): 550.47
|
||||
Output token throughput (tok/s): 565.14
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1115.61
|
||||
Concurrency: 13.71
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 12348.93
|
||||
Median E2E Latency (ms): 13164.81
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 196.08
|
||||
Median TTFT (ms): 155.22
|
||||
P99 TTFT (ms): 377.98
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 24.24
|
||||
Median TPOT (ms): 24.55
|
||||
P99 TPOT (ms): 30.42
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 23.92
|
||||
Median ITL (ms): 21.40
|
||||
P95 ITL (ms): 22.49
|
||||
P99 ITL (ms): 123.83
|
||||
Max ITL (ms): 486.54
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 138.50
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252162
|
||||
Total generated tokens (retokenized): 251841
|
||||
Request throughput (req/s): 3.61
|
||||
Input token throughput (tok/s): 1803.78
|
||||
Output token throughput (tok/s): 1820.61
|
||||
Peak output token throughput (tok/s): 2900.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3624.40
|
||||
Concurrency: 90.91
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 25183.97
|
||||
Median E2E Latency (ms): 23968.49
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 337.77
|
||||
Median TTFT (ms): 180.65
|
||||
P99 TTFT (ms): 906.14
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 49.97
|
||||
Median TPOT (ms): 52.20
|
||||
P99 TPOT (ms): 61.81
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 49.36
|
||||
Median ITL (ms): 35.05
|
||||
P95 ITL (ms): 124.91
|
||||
P99 ITL (ms): 187.69
|
||||
Max ITL (ms): 440.34
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 666.64
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44452
|
||||
Total generated tokens (retokenized): 44387
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 9.15
|
||||
Output token throughput (tok/s): 66.68
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 75.83
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 66661.35
|
||||
Median E2E Latency (ms): 71902.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 160.21
|
||||
Median TTFT (ms): 140.32
|
||||
P99 TTFT (ms): 295.56
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 14.92
|
||||
Median TPOT (ms): 14.94
|
||||
P99 TPOT (ms): 15.02
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 14.96
|
||||
Median ITL (ms): 14.96
|
||||
P95 ITL (ms): 15.36
|
||||
P99 ITL (ms): 15.57
|
||||
Max ITL (ms): 19.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 503.30
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318226
|
||||
Total generated tokens (retokenized): 318025
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 78.82
|
||||
Output token throughput (tok/s): 632.28
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 711.09
|
||||
Concurrency: 13.88
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 87349.22
|
||||
Median E2E Latency (ms): 88248.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 228.54
|
||||
Median TTFT (ms): 142.78
|
||||
P99 TTFT (ms): 569.84
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 21.97
|
||||
Median TPOT (ms): 22.14
|
||||
P99 TPOT (ms): 22.47
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 21.91
|
||||
Median ITL (ms): 21.80
|
||||
P95 ITL (ms): 22.30
|
||||
P99 ITL (ms): 22.78
|
||||
Max ITL (ms): 137.19
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 772.28
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1300705
|
||||
Total generated tokens (retokenized): 1299924
|
||||
Request throughput (req/s): 0.41
|
||||
Input token throughput (tok/s): 205.80
|
||||
Output token throughput (tok/s): 1684.24
|
||||
Peak output token throughput (tok/s): 2112.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 1890.05
|
||||
Concurrency: 56.17
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 135563.36
|
||||
Median E2E Latency (ms): 140888.88
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 232.45
|
||||
Median TTFT (ms): 145.59
|
||||
P99 TTFT (ms): 576.49
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.47
|
||||
Median TPOT (ms): 34.02
|
||||
P99 TPOT (ms): 35.10
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.30
|
||||
Median ITL (ms): 32.63
|
||||
P95 ITL (ms): 34.27
|
||||
P99 ITL (ms): 104.39
|
||||
Max ITL (ms): 155.65
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 65.11
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4210
|
||||
Total generated tokens (retokenized): 4210
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 644.17
|
||||
Output token throughput (tok/s): 64.66
|
||||
Peak output token throughput (tok/s): 68.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 708.83
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6508.31
|
||||
Median E2E Latency (ms): 5263.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 189.48
|
||||
Median TTFT (ms): 159.23
|
||||
P99 TTFT (ms): 304.09
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 15.02
|
||||
Median TPOT (ms): 15.03
|
||||
P99 TPOT (ms): 15.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 15.04
|
||||
Median ITL (ms): 15.03
|
||||
P95 ITL (ms): 15.46
|
||||
P99 ITL (ms): 15.65
|
||||
Max ITL (ms): 24.20
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 76.43
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41589
|
||||
Total generated tokens (retokenized): 41577
|
||||
Request throughput (req/s): 1.05
|
||||
Input token throughput (tok/s): 3925.47
|
||||
Output token throughput (tok/s): 544.15
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 4469.62
|
||||
Concurrency: 13.95
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13329.63
|
||||
Median E2E Latency (ms): 14141.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 339.88
|
||||
Median TTFT (ms): 252.75
|
||||
P99 TTFT (ms): 906.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 25.37
|
||||
Median TPOT (ms): 25.73
|
||||
P99 TPOT (ms): 30.94
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.04
|
||||
Median ITL (ms): 21.68
|
||||
P95 ITL (ms): 22.69
|
||||
P99 ITL (ms): 146.98
|
||||
Max ITL (ms): 483.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.6 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 136.24
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 169680
|
||||
Total generated tokens (retokenized): 169452
|
||||
Request throughput (req/s): 2.35
|
||||
Input token throughput (tok/s): 9350.32
|
||||
Output token throughput (tok/s): 1245.44
|
||||
Peak output token throughput (tok/s): 1984.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 10595.77
|
||||
Concurrency: 58.46
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 24889.40
|
||||
Median E2E Latency (ms): 25123.37
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 355.82
|
||||
Median TTFT (ms): 268.84
|
||||
P99 TTFT (ms): 858.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 46.62
|
||||
Median TPOT (ms): 49.04
|
||||
P99 TPOT (ms): 58.88
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 46.36
|
||||
Median ITL (ms): 32.46
|
||||
P95 ITL (ms): 135.23
|
||||
P99 ITL (ms): 204.27
|
||||
Max ITL (ms): 508.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.975
|
||||
Invalid: 0.000
|
||||
Latency: 16.574 s
|
||||
Output throughput: 1194.637 token/s
|
||||
```
|
||||
@@ -0,0 +1,512 @@
|
||||
---
|
||||
title: GLM-4.6V
|
||||
metatags:
|
||||
description: "Deploy GLM-4.6V vision-language model with SGLang - native function calling, 128K context, multimodal document understanding and frontend replication."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios.
|
||||
|
||||
Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features:
|
||||
|
||||
- **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#4-2-3-tool-calling).
|
||||
- **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content.
|
||||
- **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text.
|
||||
- **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
### 2.1 Docker Installation (Recommended)
|
||||
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
**Advantages:**
|
||||
|
||||
- Ready to use out of the box, no manual environment configuration needed
|
||||
- Avoids dependency conflict issues
|
||||
- Easy to migrate between different environments
|
||||
|
||||
### 2.2 Build from Source
|
||||
|
||||
If you need to use the latest development version or require custom modifications, you can build from source:
|
||||
|
||||
```bash Command
|
||||
# Install SGLang using UV (recommended)
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -e "python[all]" --index-url=https://pypi.org/simple
|
||||
pip install nvidia-cudnn-cu12==9.16.0.29
|
||||
# Install ffmpeg to support video input
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
**Use Cases:**
|
||||
|
||||
- Need to customize and modify SGLang source code
|
||||
- Want to use the latest development features
|
||||
- Participate in SGLang project development
|
||||
|
||||
For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.
|
||||
|
||||
import { GLM46VDeployment } from "/src/snippets/autoregressive/glm-46v-deployment.jsx";
|
||||
|
||||
<GLM46VDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
- **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size * number of images in current running requests.)
|
||||
- **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically).
|
||||
- **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
|
||||
- **Hardware Notes:**
|
||||
- **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
|
||||
- **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
|
||||
- **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
|
||||
- **Additional Multimodal Parameters:**
|
||||
- `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
|
||||
- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies.
|
||||
- `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency.
|
||||
|
||||
**Example with full multimodal optimizations:**
|
||||
```bash Command
|
||||
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
|
||||
SGLANG_VLM_CACHE_SIZE_MB=0 \
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/GLM-4.6V \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--enable-cache-report \
|
||||
--log-level info \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.65 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--attention-backend fa3 \
|
||||
--mm-attention-backend fa3 \
|
||||
--mm-enable-dp-encoder \
|
||||
--enable-metrics
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Multi-Modal Inputs
|
||||
|
||||
GLM-4.6V supports image and video inputs via the OpenAI-compatible API.
|
||||
|
||||
**Image Input:**
|
||||
|
||||
```python Example
|
||||
import subprocess
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{30000}/v1/chat/completions \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{{
|
||||
"model": "default",
|
||||
"messages": [
|
||||
{{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{{
|
||||
"type": "image_url",
|
||||
"image_url": {{
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}}
|
||||
}},
|
||||
{{
|
||||
"type": "text",
|
||||
"text": "What is the image"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
],
|
||||
"temperature": "0",
|
||||
"max_completion_tokens": "1000",
|
||||
"max_tokens": "1000"
|
||||
}}'
|
||||
"""
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print(response)
|
||||
```
|
||||
|
||||
```text Output
|
||||
{"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"</>\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"</>\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
|
||||
```
|
||||
|
||||
**Video Input:**
|
||||
|
||||
```python Example
|
||||
import subprocess
|
||||
|
||||
curl_command = f"""
|
||||
curl -s http://localhost:{30000}/v1/chat/completions \\
|
||||
-H "Content-Type: application/json" \\
|
||||
-d '{{
|
||||
"model": "default",
|
||||
"messages": [
|
||||
{{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{{
|
||||
"type": "video_url",
|
||||
"video_url": {{
|
||||
"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
|
||||
}}
|
||||
}},
|
||||
{{
|
||||
"type": "text",
|
||||
"text": "What is in the video"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
],
|
||||
"temperature": "0",
|
||||
"max_completion_tokens": "1000",
|
||||
"max_tokens": "1000"
|
||||
}}'
|
||||
"""
|
||||
|
||||
response = subprocess.check_output(curl_command, shell=True).decode()
|
||||
print(response)
|
||||
```
|
||||
|
||||
```text Output
|
||||
{"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n* **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n* **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n* **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n* **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
|
||||
```
|
||||
|
||||
#### 4.2.2 Thinking Mode
|
||||
|
||||
GLM-4.6V supports Thinking mode. Enable the reasoning parser during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.6V \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
GLM-4.6V supports tool calling with vision capabilities. Pass tools in your API request:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
openai_api_key = "EMPTY"
|
||||
openai_api_base = "http://127.0.0.1:30000/v1"
|
||||
client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)
|
||||
|
||||
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current temperature for a given location.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "City and country e.g. Beijing, China",
|
||||
}
|
||||
},
|
||||
"required": ["location"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Please help me check today's weather in Beijing, and tell me whether the tool returned an image."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_bk32t88BGpSdbtDgzT044Rh4",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": 'get_weather',
|
||||
"arguments": '{"location":"Beijing, China"}'
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C."
|
||||
},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=messages,
|
||||
timeout=900,
|
||||
tools=tools
|
||||
)
|
||||
print(response.choices[0].message.content.strip())
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C.
|
||||
|
||||
Yes, the tool returned an image (the SGL logo).
|
||||
```
|
||||
|
||||
#### 4.2.4 Thinking Budget
|
||||
|
||||
Beyond the reasoning parser, you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request — same as the [GLM-4.6 text model approach](./GLM-4.6#4-2-3-thinking-budget):
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.6V",
|
||||
messages=[{"role": "user", "content": "Describe this image briefly."}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1. Text Benchmark: Latency, Throughput and Accuracy
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 ./benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
#### Result Output
|
||||
```text Output
|
||||
Accuracy: 0.925
|
||||
Invalid: 0.000
|
||||
Latency: 15.327 s
|
||||
Output throughput: 1788.375 token/s
|
||||
```
|
||||
|
||||
### 5.2. Multimodal Benchmark - Latency and Throughput
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang-oai-chat \
|
||||
--port 30000 \
|
||||
--model zai-org/GLM-4.6V \
|
||||
--dataset-name image \
|
||||
--image-count 2 \
|
||||
--image-resolution 720p \
|
||||
--random-input-len 128 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 128 \
|
||||
--max-concurrency 8
|
||||
```
|
||||
|
||||
#### Result Output
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang-oai-chat
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 8
|
||||
Successful requests: 128
|
||||
Benchmark duration (s): 89.27
|
||||
Total input tokens: 315390
|
||||
Total input text tokens: 8702
|
||||
Total input vision tokens: 306688
|
||||
Total generated tokens: 66020
|
||||
Total generated tokens (retokenized): 31037
|
||||
Request throughput (req/s): 1.43
|
||||
Input token throughput (tok/s): 3533.17
|
||||
Output token throughput (tok/s): 739.59
|
||||
Peak output token throughput (tok/s): 823.00
|
||||
Peak concurrent requests: 12
|
||||
Total token throughput (tok/s): 4272.76
|
||||
Concurrency: 7.67
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5349.20
|
||||
Median E2E Latency (ms): 5380.98
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1724.04
|
||||
Median TTFT (ms): 1688.16
|
||||
P99 TTFT (ms): 6152.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.15
|
||||
Median TPOT (ms): 7.77
|
||||
P99 TPOT (ms): 23.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 10.00
|
||||
Median ITL (ms): 8.44
|
||||
P95 ITL (ms): 9.23
|
||||
P99 ITL (ms): 116.02
|
||||
Max ITL (ms): 173.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
### 5.3. Multimodal Accuracy Benchmark - MMMU
|
||||
|
||||
#### Command
|
||||
```shell Command
|
||||
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}'
|
||||
```
|
||||
|
||||
#### Result Output
|
||||
```text Output
|
||||
Benchmark time: 487.2229107860476
|
||||
answers saved to: ./answer_sglang.json
|
||||
Evaluating...
|
||||
answers saved to: ./answer_sglang.json
|
||||
{'Accounting': {'acc': 0.962, 'num': 26},
|
||||
'Agriculture': {'acc': 0.5, 'num': 30},
|
||||
'Architecture_and_Engineering': {'acc': 0.733, 'num': 15},
|
||||
'Art': {'acc': 0.833, 'num': 30},
|
||||
'Art_Theory': {'acc': 0.9, 'num': 30},
|
||||
'Basic_Medical_Science': {'acc': 0.733, 'num': 30},
|
||||
'Biology': {'acc': 0.586, 'num': 29},
|
||||
'Chemistry': {'acc': 0.654, 'num': 26},
|
||||
'Clinical_Medicine': {'acc': 0.633, 'num': 30},
|
||||
'Computer_Science': {'acc': 0.76, 'num': 25},
|
||||
'Design': {'acc': 0.867, 'num': 30},
|
||||
'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30},
|
||||
'Economics': {'acc': 0.862, 'num': 29},
|
||||
'Electronics': {'acc': 0.5, 'num': 18},
|
||||
'Energy_and_Power': {'acc': 0.875, 'num': 16},
|
||||
'Finance': {'acc': 0.857, 'num': 28},
|
||||
'Geography': {'acc': 0.714, 'num': 28},
|
||||
'History': {'acc': 0.767, 'num': 30},
|
||||
'Literature': {'acc': 0.897, 'num': 29},
|
||||
'Manage': {'acc': 0.759, 'num': 29},
|
||||
'Marketing': {'acc': 1.0, 'num': 26},
|
||||
'Materials': {'acc': 0.833, 'num': 18},
|
||||
'Math': {'acc': 0.76, 'num': 25},
|
||||
'Mechanical_Engineering': {'acc': 0.619, 'num': 21},
|
||||
'Music': {'acc': 0.286, 'num': 28},
|
||||
'Overall': {'acc': 0.761, 'num': 803},
|
||||
'Overall-Art and Design': {'acc': 0.729, 'num': 118},
|
||||
'Overall-Business': {'acc': 0.884, 'num': 138},
|
||||
'Overall-Health and Medicine': {'acc': 0.773, 'num': 150},
|
||||
'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118},
|
||||
'Overall-Science': {'acc': 0.728, 'num': 136},
|
||||
'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143},
|
||||
'Pharmacy': {'acc': 0.933, 'num': 30},
|
||||
'Physics': {'acc': 0.929, 'num': 28},
|
||||
'Psychology': {'acc': 0.733, 'num': 30},
|
||||
'Public_Health': {'acc': 0.933, 'num': 30},
|
||||
'Sociology': {'acc': 0.724, 'num': 29}}
|
||||
eval out saved to ./val_sglang.json
|
||||
Overall accuracy: 0.761
|
||||
```
|
||||
@@ -0,0 +1,935 @@
|
||||
---
|
||||
title: GLM-4.7-Flash
|
||||
metatags:
|
||||
description: "Deploy GLM-4.7-Flash 30B-A3B MoE model with SGLang - lightweight, efficient inference optimized for single-GPU deployment."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment.
|
||||
|
||||
As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency:
|
||||
|
||||
- **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference
|
||||
- **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance
|
||||
- **Superior Coding**: Strong code generation and understanding capabilities
|
||||
- **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows
|
||||
- **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios
|
||||
|
||||
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM47FlashDeployment } from "/src/snippets/autoregressive/glm-47-flash-deployment.jsx";
|
||||
|
||||
<GLM47FlashDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.7-Flash. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--reasoning-parser glm45 \
|
||||
--attention-backend triton \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47 \
|
||||
--attention-backend triton \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls (tool call deltas may stream in multiple chunks)
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
if tool_calls_accumulator:
|
||||
print("\n=============== Tool Calls =================", flush=True)
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the
|
||||
user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you.
|
||||
=============== Tool Calls =================
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7-Flash",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 (1x)
|
||||
- Model: GLM-4.7-Flash
|
||||
- Tensor Parallelism: 1
|
||||
- SGLang Version: 0.5.7
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Three core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--attention-backend triton \
|
||||
--tp 1
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 38.94
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.26
|
||||
Input token throughput (tok/s): 156.67
|
||||
Output token throughput (tok/s): 108.37
|
||||
Peak output token throughput (tok/s): 125.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 265.03
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3891.12
|
||||
Median E2E Latency (ms): 3061.48
|
||||
P90 E2E Latency (ms): 7172.25
|
||||
P99 E2E Latency (ms): 9042.62
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 131.36
|
||||
Median TTFT (ms): 94.55
|
||||
P99 TTFT (ms): 435.93
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.75
|
||||
Median TPOT (ms): 8.82
|
||||
P99 TPOT (ms): 9.39
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.93
|
||||
Median ITL (ms): 8.98
|
||||
P95 ITL (ms): 9.83
|
||||
P99 ITL (ms): 10.20
|
||||
Max ITL (ms): 18.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 52.73
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40775
|
||||
Request throughput (req/s): 1.52
|
||||
Input token throughput (tok/s): 752.27
|
||||
Output token throughput (tok/s): 773.83
|
||||
Peak output token throughput (tok/s): 1040.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 1526.10
|
||||
Concurrency: 13.98
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 9217.90
|
||||
Median E2E Latency (ms): 9642.50
|
||||
P90 E2E Latency (ms): 15147.02
|
||||
P99 E2E Latency (ms): 18237.06
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 299.02
|
||||
Median TTFT (ms): 105.98
|
||||
P99 TTFT (ms): 1109.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.03
|
||||
Median TPOT (ms): 18.00
|
||||
P99 TPOT (ms): 26.51
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 17.52
|
||||
Median ITL (ms): 16.07
|
||||
P95 ITL (ms): 18.14
|
||||
P99 ITL (ms): 89.43
|
||||
Max ITL (ms): 763.13
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 91.48
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250941
|
||||
Request throughput (req/s): 5.47
|
||||
Input token throughput (tok/s): 2730.87
|
||||
Output token throughput (tok/s): 2761.82
|
||||
Peak output token throughput (tok/s): 4199.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 5492.69
|
||||
Concurrency: 90.54
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16566.04
|
||||
Median E2E Latency (ms): 16134.36
|
||||
P90 E2E Latency (ms): 30167.60
|
||||
P99 E2E Latency (ms): 34034.04
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 433.94
|
||||
Median TTFT (ms): 123.26
|
||||
P99 TTFT (ms): 1760.09
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 32.26
|
||||
Median TPOT (ms): 33.56
|
||||
P99 TPOT (ms): 38.78
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 31.99
|
||||
Median ITL (ms): 24.06
|
||||
P95 ITL (ms): 79.62
|
||||
P99 ITL (ms): 103.03
|
||||
Max ITL (ms): 1369.20
|
||||
==================================================
|
||||
```
|
||||
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 525.43
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44451
|
||||
Request throughput (req/s): 0.02
|
||||
Input token throughput (tok/s): 11.61
|
||||
Output token throughput (tok/s): 84.62
|
||||
Peak output token throughput (tok/s): 125.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 96.23
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 52540.19
|
||||
Median E2E Latency (ms): 53694.45
|
||||
P90 E2E Latency (ms): 94742.08
|
||||
P99 E2E Latency (ms): 101224.18
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 97.45
|
||||
Median TTFT (ms): 95.28
|
||||
P99 TTFT (ms): 105.64
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.94
|
||||
Median TPOT (ms): 11.25
|
||||
P99 TPOT (ms): 13.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.80
|
||||
Median ITL (ms): 11.51
|
||||
P95 ITL (ms): 15.83
|
||||
P99 ITL (ms): 16.86
|
||||
Max ITL (ms): 19.96
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 473.92
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 317860
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 83.70
|
||||
Output token throughput (tok/s): 671.65
|
||||
Peak output token throughput (tok/s): 1040.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 755.35
|
||||
Concurrency: 13.80
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 81746.73
|
||||
Median E2E Latency (ms): 78508.54
|
||||
P90 E2E Latency (ms): 155292.49
|
||||
P99 E2E Latency (ms): 166769.99
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 117.50
|
||||
Median TTFT (ms): 101.97
|
||||
P99 TTFT (ms): 182.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.36
|
||||
Median TPOT (ms): 20.48
|
||||
P99 TPOT (ms): 22.63
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.52
|
||||
Median ITL (ms): 20.42
|
||||
P95 ITL (ms): 23.41
|
||||
P99 ITL (ms): 26.29
|
||||
Max ITL (ms): 90.48
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 714.72
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1289431
|
||||
Request throughput (req/s): 0.45
|
||||
Input token throughput (tok/s): 222.38
|
||||
Output token throughput (tok/s): 1820.33
|
||||
Peak output token throughput (tok/s): 3200.00
|
||||
Peak concurrent requests: 68
|
||||
Total token throughput (tok/s): 2042.71
|
||||
Concurrency: 55.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 124364.58
|
||||
Median E2E Latency (ms): 129250.98
|
||||
P90 E2E Latency (ms): 219175.80
|
||||
P99 E2E Latency (ms): 247741.77
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 149.40
|
||||
Median TTFT (ms): 114.78
|
||||
P99 TTFT (ms): 288.60
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 30.51
|
||||
Median TPOT (ms): 31.75
|
||||
P99 TPOT (ms): 33.32
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 30.56
|
||||
Median ITL (ms): 30.82
|
||||
P95 ITL (ms): 33.20
|
||||
P99 ITL (ms): 80.54
|
||||
Max ITL (ms): 117.72
|
||||
==================================================
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 58.27
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.17
|
||||
Input token throughput (tok/s): 719.73
|
||||
Output token throughput (tok/s): 72.42
|
||||
Peak output token throughput (tok/s): 112.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 792.15
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5825.08
|
||||
Median E2E Latency (ms): 4624.26
|
||||
P90 E2E Latency (ms): 12690.22
|
||||
P99 E2E Latency (ms): 13177.96
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 296.01
|
||||
Median TTFT (ms): 195.59
|
||||
P99 TTFT (ms): 717.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.63
|
||||
Median TPOT (ms): 13.07
|
||||
P99 TPOT (ms): 16.68
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 13.13
|
||||
Median ITL (ms): 13.17
|
||||
P95 ITL (ms): 17.02
|
||||
P99 ITL (ms): 17.47
|
||||
Max ITL (ms): 19.84
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 89.59
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41656
|
||||
Request throughput (req/s): 0.89
|
||||
Input token throughput (tok/s): 3348.77
|
||||
Output token throughput (tok/s): 465.10
|
||||
Peak output token throughput (tok/s): 752.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3813.87
|
||||
Concurrency: 14.39
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16120.74
|
||||
Median E2E Latency (ms): 16246.55
|
||||
P90 E2E Latency (ms): 27279.72
|
||||
P99 E2E Latency (ms): 34577.93
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1943.94
|
||||
Median TTFT (ms): 382.19
|
||||
P99 TTFT (ms): 8980.41
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 27.87
|
||||
Median TPOT (ms): 28.26
|
||||
P99 TPOT (ms): 40.55
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 27.27
|
||||
Median ITL (ms): 21.74
|
||||
P95 ITL (ms): 23.32
|
||||
P99 ITL (ms): 232.65
|
||||
Max ITL (ms): 4282.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7-Flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 167.01
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169226
|
||||
Request throughput (req/s): 1.92
|
||||
Input token throughput (tok/s): 7627.82
|
||||
Output token throughput (tok/s): 1017.93
|
||||
Peak output token throughput (tok/s): 1984.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 8645.75
|
||||
Concurrency: 59.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 31147.52
|
||||
Median E2E Latency (ms): 30603.34
|
||||
P90 E2E Latency (ms): 54889.44
|
||||
P99 E2E Latency (ms): 67665.30
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 428.87
|
||||
Median TTFT (ms): 441.69
|
||||
P99 TTFT (ms): 1232.68
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 58.06
|
||||
Median TPOT (ms): 62.79
|
||||
P99 TPOT (ms): 82.23
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 57.93
|
||||
Median ITL (ms): 33.30
|
||||
P95 ITL (ms): 247.98
|
||||
P99 ITL (ms): 409.63
|
||||
Max ITL (ms): 1421.21
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Result
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.845
|
||||
Invalid: 0.000
|
||||
Latency: 8.431 s
|
||||
Output throughput: 2195.387 token/s
|
||||
```
|
||||
@@ -0,0 +1,962 @@
|
||||
---
|
||||
title: GLM-4.7
|
||||
metatags:
|
||||
description: "Deploy GLM-4.7 with SGLang on NVIDIA Blackwell (B200, GB200) and AMD GPUs - state-of-the-art reasoning, robust tool calling, and NVFP4 weights for Blackwell."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and agent workflows.
|
||||
|
||||
GLM-4.7 brings improvements across all major domains:
|
||||
|
||||
- **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations
|
||||
- **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing
|
||||
- **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance
|
||||
- **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows
|
||||
- **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms
|
||||
|
||||
For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks
|
||||
- **Multiple Quantizations**: BF16, FP8, and NVFP4 variants for different performance/memory trade-offs
|
||||
- **Hardware Optimization**: Tuned for NVIDIA Blackwell (B200, GB200) and AMD MI300X/MI325X/MI355X GPUs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8)
|
||||
- **NVFP4 (4-bit, NVIDIA Blackwell)**: [nvidia/GLM-4.7-NVFP4](https://huggingface.co/nvidia/GLM-4.7-NVFP4)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
**Docker Images by Hardware Platform:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA H100 / H200 / B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA GB200 / B300 / GB300 (aarch64)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-cu130`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-rocm720-mi30x`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.12-rocm720-mi35x`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.
|
||||
|
||||
import { GLM47Deployment } from "/src/snippets/autoregressive/glm-47-deployment.jsx";
|
||||
|
||||
<GLM47Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
Pick a weight format by hardware: **NVFP4** on NVIDIA Blackwell (B200, GB200), **FP8** on H100/H200/AMD, **BF16** as the full-precision fallback. The recommended tensor-parallel size per platform:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>NVFP4</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200 (8×, single node)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=2 / 4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB200 (NVL72, 4× per tray)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=2 / 4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200 (8×)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X / MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=2 / 4 / 8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4 / 8</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **EAGLE Speculative Decoding:** Supported for GLM-4.7. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).
|
||||
|
||||
For general GLM-4.x family launch guidance (AMD ROCm notes and more), see [Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang](/cookbook/autoregressive/GLM/GLM-4.5). Per-hardware bench commands and flags are inline in §5.1 below.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
To solve this problem, I need to calculate 15% of 240.
|
||||
Step 1: Convert 15% to decimal: 15% = 0.15
|
||||
Step 2: Multiply 240 by 0.15
|
||||
Step 3: 240 × 0.15 = 36
|
||||
=============== Content =================
|
||||
|
||||
The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
<Note>
|
||||
**Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.
|
||||
</Note>
|
||||
|
||||
GLM-4.7 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47 \
|
||||
--tp 8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
#### 4.2.3 Thinking Budget
|
||||
|
||||
Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-4.7",
|
||||
messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
|
||||
max_tokens=1024,
|
||||
extra_body={
|
||||
"custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
|
||||
"custom_params": {"thinking_budget": 512},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200, NVIDIA GB200, AMD MI300X/MI325X/MI355X (8x)
|
||||
- Model: GLM-4.7-NVFP4 on NVIDIA Blackwell; GLM-4.7-FP8 or GLM-4.7 (BF16) on AMD
|
||||
- SGLang Version: 0.5.12 (NVIDIA Blackwell), 0.5.6.post1 (AMD)
|
||||
- Best per-GPU throughput config on B200: **TP=2 NVFP4 bf16-KV** (NVFP4 weights, no EP). Numbers below come from this config.
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Test Scenarios
|
||||
|
||||
Four core scenarios reflect real-world usage patterns:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
<col style={{width: "25%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Scenario</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Input Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Output Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Use Case</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Chat**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most common conversational AI workload</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Reasoning**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Long-form generation, complex reasoning tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Summarization**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Document summarization, RAG retrieval</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Throughput**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>4K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mixed RAG / agent / multi-turn conversation (used for the inline B200 / GB200 results below)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 Concurrency Levels
|
||||
|
||||
Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):
|
||||
|
||||
- **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
|
||||
- **Medium Concurrency**: `--max-concurrency 16` (Balanced)
|
||||
- **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) — the Throughput (4K/1K) scenario uses `--max-concurrency 128` to match the inline B200/GB200 results below.
|
||||
|
||||
#### 5.1.3 Number of Prompts
|
||||
|
||||
For each concurrency level, configure `num_prompts` to simulate realistic user loads:
|
||||
|
||||
- **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
|
||||
- **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
|
||||
- **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Benchmark Commands
|
||||
|
||||
**Scenario 1: Chat (1K/1K) - Most Important**
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 2: Reasoning (1K/8K)**
|
||||
|
||||
- Low Concurrency
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 3: Summarization (8K/1K)**
|
||||
|
||||
- Low Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Medium Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- High Concurrency
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-4.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
**Scenario 4: Throughput (4K/1K) — NVIDIA Blackwell with NVFP4**
|
||||
|
||||
The remaining sub-sections (§5.1.4.1 NVIDIA B200, §5.1.4.2 NVIDIA GB200) measure this scenario with `nvidia/GLM-4.7-NVFP4` weights and report the full `bench_serving` output verbatim. The same commands apply to other NVIDIA hardware after substituting the deployment line from §3.1.
|
||||
|
||||
> **Note**: These runs use EOS-enabled generation (no `--disable-ignore-eos`), so generated-token counts reflect natural model behavior rather than a strict fixed-OSL pin. Compare against other EOS-enabled runs at the same workload, not against fixed-output-length benchmarks.
|
||||
|
||||
#### 5.1.4.1 NVIDIA B200
|
||||
|
||||
**Model Deployment (NVIDIA B200, TP=2 NVFP4 — max tok/s/gpu config):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 5 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 1
|
||||
Successful requests: 5
|
||||
Benchmark duration (s): 25.07
|
||||
Total input tokens: 8105
|
||||
Total generated tokens: 2674
|
||||
Request throughput (req/s): 0.20
|
||||
Input token throughput (tok/s): 323.25
|
||||
Output token throughput (tok/s): 106.65
|
||||
Total token throughput (tok/s): 429.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5011.93
|
||||
Median E2E Latency (ms): 6441.44
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 179.61
|
||||
Median TTFT (ms): 169.05
|
||||
P99 TTFT (ms): 238.01
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.05
|
||||
Median TPOT (ms): 9.03
|
||||
P99 TPOT (ms): 9.16
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.05
|
||||
Median ITL (ms): 9.05
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 60.60
|
||||
Total input tokens: 179772
|
||||
Total generated tokens: 39657
|
||||
Request throughput (req/s): 1.32
|
||||
Input token throughput (tok/s): 2966.39
|
||||
Output token throughput (tok/s): 654.37
|
||||
Total token throughput (tok/s): 3620.76
|
||||
Concurrency: 14.01
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10615.87
|
||||
Median E2E Latency (ms): 9985.45
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 267.39
|
||||
Median TTFT (ms): 177.26
|
||||
P99 TTFT (ms): 584.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.98
|
||||
Median TPOT (ms): 21.06
|
||||
P99 TPOT (ms): 24.88
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.92
|
||||
Median ITL (ms): 17.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 640 \
|
||||
--max-concurrency 128 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 128
|
||||
Successful requests: 640
|
||||
Benchmark duration (s): 172.95
|
||||
Total input tokens: 1453591
|
||||
Total generated tokens: 308740
|
||||
Request throughput (req/s): 3.70
|
||||
Input token throughput (tok/s): 8404.67
|
||||
Output token throughput (tok/s): 1785.14
|
||||
Total token throughput (tok/s): 10189.80
|
||||
Concurrency: 117.85
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 31848.20
|
||||
Median E2E Latency (ms): 28554.42
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1598.40
|
||||
Median TTFT (ms): 298.88
|
||||
P99 TTFT (ms): 11015.96
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 65.94
|
||||
Median TPOT (ms): 65.81
|
||||
P99 TPOT (ms): 137.73
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 62.99
|
||||
Median ITL (ms): 35.44
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.4.2 NVIDIA GB200
|
||||
|
||||
**Model Deployment (NVIDIA GB200, TP=2 NVFP4 — max tok/s/gpu config):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--tp-size 2 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm47
|
||||
```
|
||||
|
||||
- Low Concurrency (Latency-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 5 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 1
|
||||
Successful requests: 5
|
||||
Benchmark duration (s): 24.74
|
||||
Total input tokens: 8105
|
||||
Total generated tokens: 2674
|
||||
Request throughput (req/s): 0.20
|
||||
Input token throughput (tok/s): 327.65
|
||||
Output token throughput (tok/s): 108.10
|
||||
Total token throughput (tok/s): 435.75
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4944.47
|
||||
Median E2E Latency (ms): 6347.31
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 211.41
|
||||
Median TTFT (ms): 207.25
|
||||
P99 TTFT (ms): 226.46
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.86
|
||||
Median TPOT (ms): 8.84
|
||||
P99 TPOT (ms): 8.96
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.87
|
||||
Median ITL (ms): 8.85
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Medium Concurrency (Balanced)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 60.40
|
||||
Total input tokens: 179772
|
||||
Total generated tokens: 39657
|
||||
Request throughput (req/s): 1.32
|
||||
Input token throughput (tok/s): 2976.52
|
||||
Output token throughput (tok/s): 656.61
|
||||
Total token throughput (tok/s): 3633.13
|
||||
Concurrency: 13.97
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 10611.51
|
||||
Median E2E Latency (ms): 9956.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 338.14
|
||||
Median TTFT (ms): 215.25
|
||||
P99 TTFT (ms): 915.40
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 20.87
|
||||
Median TPOT (ms): 21.36
|
||||
P99 TPOT (ms): 27.05
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 20.77
|
||||
Median ITL (ms): 16.53
|
||||
==================================================
|
||||
```
|
||||
|
||||
- High Concurrency (Throughput-Optimized)
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model nvidia/GLM-4.7-NVFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 640 \
|
||||
--max-concurrency 128 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Max request concurrency: 128
|
||||
Successful requests: 640
|
||||
Benchmark duration (s): 181.89
|
||||
Total input tokens: 1453591
|
||||
Total generated tokens: 309221
|
||||
Request throughput (req/s): 3.52
|
||||
Input token throughput (tok/s): 7991.59
|
||||
Output token throughput (tok/s): 1700.04
|
||||
Total token throughput (tok/s): 9691.63
|
||||
Concurrency: 118.86
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 33690.47
|
||||
Median E2E Latency (ms): 30421.55
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 1353.16
|
||||
Median TTFT (ms): 383.52
|
||||
P99 TTFT (ms): 8940.53
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 69.88
|
||||
Median TPOT (ms): 71.77
|
||||
P99 TPOT (ms): 131.75
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 67.23
|
||||
Median ITL (ms): 33.46
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.5 Understanding the Results
|
||||
|
||||
**Key Metrics:**
|
||||
|
||||
- **Request Throughput (req/s)**: Number of requests processed per second
|
||||
- **Output Token Throughput (tok/s)**: Total tokens generated per second
|
||||
- **Mean TTFT (ms)**: Time to First Token - measures responsiveness
|
||||
- **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
|
||||
- **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency
|
||||
|
||||
**Why These Configurations Matter:**
|
||||
|
||||
- **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
|
||||
- **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
|
||||
- **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q&A, and summarization tasks.
|
||||
- **4K/1K (Throughput)**: Realistic mixed workload typical of production deployments (RAG context + medium response). Long enough input that prefill matters, long enough output that decode steady-state dominates. Used for the inline B200 / GB200 results above.
|
||||
- **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.
|
||||
|
||||
**Interpreting Results:**
|
||||
|
||||
- Compare your results against baseline numbers for your hardware
|
||||
- Higher throughput at same latency = better performance
|
||||
- Lower TTFT = more responsive user experience
|
||||
- Lower TPOT = faster generation speed
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-shots 5 \
|
||||
--num-questions 1319 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
- Test Result (NVIDIA B200, TP=2 NVFP4)
|
||||
```text Output
|
||||
Accuracy: 0.946
|
||||
Latency: 178.284 s
|
||||
Output throughput: 769.204 token/s
|
||||
```
|
||||
|
||||
- Test Result (NVIDIA GB200, TP=2 NVFP4)
|
||||
```text Output
|
||||
Accuracy: 0.951
|
||||
Latency: 175.190 s
|
||||
Invalid: 0.000
|
||||
```
|
||||
@@ -0,0 +1,740 @@
|
||||
---
|
||||
title: GLM-5.1
|
||||
metatags:
|
||||
description: "Deploy GLM-5.1 with SGLang on NVIDIA H100/H200/B300/GB300 and AMD MI300X/MI325X/MI355X."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8)
|
||||
- **NVFP4 (4-bit quantized)**: [nvidia/GLM-5.1-NVFP4](https://huggingface.co/nvidia/GLM-5.1-NVFP4)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B300, GB300, and AMD MI300X/MI325X/MI355X GPUs.
|
||||
|
||||
import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment.jsx'
|
||||
|
||||
<GLM51Deployment />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.1.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
|
||||
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>NVFP4</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>MXFP4</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=4</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **H100 and H200**: FP8 is the recommended deployment path.
|
||||
- **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300.
|
||||
- **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang.
|
||||
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
|
||||
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1-FP8 \
|
||||
--tp 8 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 B300/GB300 (NVFP4) Server Command
|
||||
|
||||
#### B300
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/GLM-5.1-NVFP4 \
|
||||
--tp 8 \
|
||||
--quantization modelopt_fp4 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.80 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### GB300
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path nvidia/GLM-5.1-NVFP4 \
|
||||
--tp 4 \
|
||||
--quantization modelopt_fp4 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.80 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 MI300X/MI325X/MI355X (ROCm) Server Command
|
||||
|
||||
The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above.
|
||||
|
||||
#### MXFP4 (MI355X / gfx950)
|
||||
|
||||
On MI355X (gfx950), set `SGLANG_DSA_TRITON_PREFILL=1` to enable a faster Triton attention kernel for the prefill phase (opt-in, off by default). Keep `--dsa-prefill-backend tilelang` as shown. The EAGLE speculative-decoding flags below are optional but recommended on gfx950.
|
||||
|
||||
```shell Command
|
||||
# SGLANG_DSA_TRITON_PREFILL=1 is optional; it enables a faster Triton prefill kernel on gfx950
|
||||
SGLANG_DSA_TRITON_PREFILL=1 sglang serve \
|
||||
--model-path amd/GLM-5.1-MXFP4 \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### FP8 (Recommended)
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1-FP8 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
#### BF16
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5.1 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-custom-all-reduce \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.3 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.4 Advanced Usage
|
||||
|
||||
#### 4.4.1 Reasoning Parser
|
||||
|
||||
GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
|
||||
|
||||
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
|
||||
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
|
||||
|
||||
**Example 1: Thinking Mode (Default)**
|
||||
|
||||
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Thinking mode is enabled by default, no extra parameters needed
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
1. **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step.
|
||||
|
||||
2. **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$.
|
||||
|
||||
3. **Step-by-Step Breakdown:**
|
||||
* *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$.
|
||||
* *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$.
|
||||
* *Step 3: Perform the calculation.*
|
||||
* $0.15 \times 240$
|
||||
* I can break this down further to make it easy to follow:
|
||||
* $0.10 \times 240 = 24$ (which is 10%)
|
||||
* $0.05 \times 240 = 12$ (which is 5%, half of 10%)
|
||||
* $24 + 12 = 36$
|
||||
* Alternatively, standard multiplication:
|
||||
* $240 \times 15 = 3600$
|
||||
* Move decimal two places left -> $36$
|
||||
* *Step 4: State the final answer clearly.*
|
||||
|
||||
4. **Draft the Response (incorporating the steps clearly):**
|
||||
* *Introduction:* State the problem clearly.
|
||||
* *Step 1:* Explain how to convert 15% to a decimal.
|
||||
* *Step 2:* Explain the multiplication step.
|
||||
* *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value).
|
||||
* *Conclusion:* Give the final answer.
|
||||
|
||||
5. **Refine the Output (Self-Correction/Polishing during drafting):**
|
||||
* *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15.
|
||||
* *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240.
|
||||
* *Drafting Step 3 (Standard way):* 0.15 × 240 = 36.
|
||||
* *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%.
|
||||
* 10% of 240 = 24 (move the decimal point one place to the left)
|
||||
* 5% is half of 10%, so half of 24 = 12
|
||||
* Add them together: 24 + 12 = 36.
|
||||
* *Final Answer:* 15% of 240 is 36.
|
||||
|
||||
6. **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output).
|
||||
=============== Content =================
|
||||
Here is the step-by-step solution to find 15% of 240:
|
||||
|
||||
**Step 1: Convert the percentage to a decimal.**
|
||||
To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left).
|
||||
* 15% = 15 ÷ 100 = **0.15**
|
||||
|
||||
**Step 2: Multiply the decimal by the number.**
|
||||
Now, multiply the decimal (0.15) by the number you are finding the percentage of (240).
|
||||
* 0.15 × 240 = **36**
|
||||
|
||||
*(Alternative mental math method for Step 2)*:
|
||||
If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%:
|
||||
* **10% of 240** = 24 (just move the decimal point one place to the left)
|
||||
* **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2)
|
||||
* **Add them together**: 24 + 12 = **36**
|
||||
|
||||
**Answer:**
|
||||
15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Example 2: Instruct Mode (Thinking Off)**
|
||||
|
||||
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Disable thinking mode via chat_template_kwargs
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is 15% of 240?"}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# In Instruct mode, the model responds directly without reasoning_content
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
15% of 240 is 36.
|
||||
|
||||
Here is how to calculate it:
|
||||
1. Convert the percentage to a decimal: 15% = 0.15
|
||||
2. Multiply the decimal by the number: 0.15 × 240 = 36
|
||||
```
|
||||
|
||||
#### 4.4.2 Tool Calling
|
||||
|
||||
GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.1-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location.
|
||||
=============== Content =================
|
||||
Tool Call: get_weather
|
||||
Arguments:
|
||||
Tool Call: None
|
||||
Arguments: {
|
||||
Tool Call: None
|
||||
Arguments: "location": "Be
|
||||
Tool Call: None
|
||||
Arguments: ijing"
|
||||
Tool Call: None
|
||||
Arguments: }
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: H200 (8x)
|
||||
- Model: GLM-5.1-FP8
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: commit 947927bdb
|
||||
|
||||
#### 5.1.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5.1-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 35.78
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4213
|
||||
Request throughput (req/s): 0.28
|
||||
Input token throughput (tok/s): 170.54
|
||||
Output token throughput (tok/s): 117.96
|
||||
Peak output token throughput (tok/s): 148.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 288.50
|
||||
Concurrency: 1.00
|
||||
Accept length: 3.48
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3576.31
|
||||
Median E2E Latency (ms): 2935.97
|
||||
P90 E2E Latency (ms): 5908.97
|
||||
P99 E2E Latency (ms): 8588.08
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 290.88
|
||||
Median TTFT (ms): 282.34
|
||||
P99 TTFT (ms): 332.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.54
|
||||
Median TPOT (ms): 6.97
|
||||
P99 TPOT (ms): 9.04
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.80
|
||||
Median ITL (ms): 6.81
|
||||
P95 ITL (ms): 13.51
|
||||
P99 ITL (ms): 26.99
|
||||
Max ITL (ms): 29.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5.1-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 411.74
|
||||
Total input tokens: 502493
|
||||
Total input text tokens: 502493
|
||||
Total generated tokens: 500251
|
||||
Total generated tokens (retokenized): 499614
|
||||
Request throughput (req/s): 2.43
|
||||
Input token throughput (tok/s): 1220.41
|
||||
Output token throughput (tok/s): 1214.97
|
||||
Peak output token throughput (tok/s): 2648.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 2435.38
|
||||
Concurrency: 96.30
|
||||
Accept length: 3.50
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 39648.76
|
||||
Median E2E Latency (ms): 39058.12
|
||||
P90 E2E Latency (ms): 57009.82
|
||||
P99 E2E Latency (ms): 68880.33
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 20613.80
|
||||
Median TTFT (ms): 21429.21
|
||||
P99 TTFT (ms): 29543.17
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 38.73
|
||||
Median TPOT (ms): 36.52
|
||||
P99 TPOT (ms): 67.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.13
|
||||
Median ITL (ms): 16.57
|
||||
P95 ITL (ms): 86.01
|
||||
P99 ITL (ms): 164.88
|
||||
Max ITL (ms): 1307.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
<Note>
|
||||
The accuracy benchmark results below are shared with GLM-5, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate benchmark run is planned.
|
||||
</Note>
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.000
|
||||
Latency: 32.470 s
|
||||
Output throughput: 642.044 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.860
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.941
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.932
|
||||
subject: college_biology, #q:144, acc: 0.972
|
||||
subject: college_chemistry, #q:100, acc: 0.640
|
||||
subject: college_computer_science, #q:100, acc: 0.900
|
||||
subject: college_mathematics, #q:100, acc: 0.810
|
||||
subject: college_medicine, #q:173, acc: 0.873
|
||||
subject: college_physics, #q:102, acc: 0.912
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.928
|
||||
subject: econometrics, #q:114, acc: 0.807
|
||||
subject: electrical_engineering, #q:145, acc: 0.897
|
||||
subject: elementary_mathematics, #q:378, acc: 0.937
|
||||
subject: formal_logic, #q:126, acc: 0.778
|
||||
subject: global_facts, #q:100, acc: 0.710
|
||||
subject: high_school_biology, #q:310, acc: 0.961
|
||||
subject: high_school_chemistry, #q:203, acc: 0.847
|
||||
subject: high_school_computer_science, #q:100, acc: 0.960
|
||||
subject: high_school_european_history, #q:165, acc: 0.891
|
||||
subject: high_school_geography, #q:198, acc: 0.960
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.984
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.923
|
||||
subject: high_school_mathematics, #q:270, acc: 0.696
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.962
|
||||
subject: high_school_physics, #q:151, acc: 0.821
|
||||
subject: high_school_psychology, #q:545, acc: 0.956
|
||||
subject: high_school_statistics, #q:216, acc: 0.889
|
||||
subject: high_school_us_history, #q:204, acc: 0.941
|
||||
subject: high_school_world_history, #q:237, acc: 0.945
|
||||
subject: human_aging, #q:223, acc: 0.857
|
||||
subject: human_sexuality, #q:131, acc: 0.908
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.907
|
||||
subject: logical_fallacies, #q:163, acc: 0.933
|
||||
subject: machine_learning, #q:112, acc: 0.830
|
||||
subject: management, #q:103, acc: 0.942
|
||||
subject: marketing, #q:234, acc: 0.940
|
||||
subject: medical_genetics, #q:100, acc: 0.990
|
||||
subject: miscellaneous, #q:783, acc: 0.959
|
||||
subject: moral_disputes, #q:346, acc: 0.873
|
||||
subject: moral_scenarios, #q:895, acc: 0.837
|
||||
subject: nutrition, #q:306, acc: 0.922
|
||||
subject: philosophy, #q:311, acc: 0.897
|
||||
subject: prehistory, #q:324, acc: 0.929
|
||||
subject: professional_accounting, #q:282, acc: 0.844
|
||||
subject: professional_law, #q:1534, acc: 0.714
|
||||
subject: professional_medicine, #q:272, acc: 0.941
|
||||
subject: professional_psychology, #q:612, acc: 0.913
|
||||
subject: public_relations, #q:110, acc: 0.791
|
||||
subject: security_studies, #q:245, acc: 0.878
|
||||
subject: sociology, #q:201, acc: 0.940
|
||||
subject: us_foreign_policy, #q:100, acc: 0.920
|
||||
subject: virology, #q:166, acc: 0.596
|
||||
subject: world_religions, #q:171, acc: 0.936
|
||||
Total latency: 165.275
|
||||
Average accuracy: 0.877
|
||||
```
|
||||
|
||||
### 5.3 AMD GPU Benchmarks
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
|
||||
|
||||
- MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang DSA backends)
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.970
|
||||
Invalid: 0.000
|
||||
```
|
||||
|
||||
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
title: GLM-5.2
|
||||
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
- **Balanced** — good speed with several users at once. Use for typical multi-user serving.
|
||||
- **High-Throughput** — most tokens per second across many users. Best for batch jobs.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/zai-org/glm-5.2.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/zai-org/glm-5.2-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.2.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on `main @ 09ca4fc`. Spec cells pin the EAGLE acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN` (low-latency 5-1-6 = 3.5, balanced 2-1-3 = 2); high-throughput has no spec.
|
||||
</Note>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within ~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Context</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.2-FP8">GLM-5.2-FP8</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · FP8</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/zai-org/GLM-5.2">GLM-5.2</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · BF16</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/nvidia/GLM-5.2-NVFP4">GLM-5.2-NVFP4</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>MoE · DSA · 256 experts (top-8) · MTP · NVFP4</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right"}}>1,048,576</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code).
|
||||
|
||||
**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.2's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`.
|
||||
- **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute.
|
||||
- **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).
|
||||
- **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.
|
||||
- **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points.
|
||||
- **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP.
|
||||
- **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider.
|
||||
|
||||
- **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).
|
||||
|
||||
<Note>
|
||||
**gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).
|
||||
</Note>
|
||||
|
||||
- **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel. The block-FP8 accuracy bug that previously degraded it is now fixed (see note above), but MTP on gfx950 still depends on the spec-decode draft kernel, which is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
GLM-5.2 is a hybrid-reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Thinking is on by default; turn it off with `chat_template_kwargs: {"enable_thinking": False}` (the template variable is `enable_thinking`, not `thinking`).
|
||||
|
||||
**Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to inject a `Reasoning Effort: <level>` system line (only while thinking is on). **The template wires only two effective levels — `Max` and `High` — and if you don't pass `reasoning_effort` at all you get `Max`, the highest.** `"high"` is the *only* value that lowers effort; every other value (including `"low"` and `"medium"`) falls through to `Max`:
|
||||
|
||||
| `reasoning_effort` | Injected system line | Effect |
|
||||
|---|---|---|
|
||||
| *(not passed / unset)* | `Reasoning Effort: Max` | **default — highest reasoning** |
|
||||
| `"high"` | `Reasoning Effort: High` | dials reasoning **down** |
|
||||
| `"low"`, `"medium"`, any other value | `Reasoning Effort: Max` | falls through to `Max` (not a distinct level) |
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
resp = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.2-FP8",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}},
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240.
|
||||
2. **Convert the percentage to a decimal:** 15% = 0.15
|
||||
3. **Multiply by the total:** 0.15 * 240 = 36
|
||||
(Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)
|
||||
|
||||
Answer: 15% of 240 is **36**.
|
||||
|
||||
Here is how you can calculate it:
|
||||
0.15 × 240 = 36
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.2 emits the newer `<tool_call>…<arg_key>…<arg_value>…` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields.
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
resp = client.chat.completions.create(
|
||||
model="zai-org/GLM-5.2-FP8",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=tools,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", getattr(msg, "reasoning_content", None))
|
||||
print("Tool calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.
|
||||
|
||||
Tool calls: [
|
||||
{
|
||||
"id": "call_13fcd52146934b7781d06d4a",
|
||||
"type": "function",
|
||||
"function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.3 HiCache (Hierarchical KV Caching)
|
||||
|
||||
For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern.
|
||||
|
||||
### 3.4 Claude Code Integration
|
||||
|
||||
GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then:
|
||||
|
||||
```bash Command
|
||||
export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
|
||||
export ANTHROPIC_AUTH_TOKEN="dummy"
|
||||
export API_TIMEOUT_MS="3000000"
|
||||
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
|
||||
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
|
||||
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
|
||||
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"
|
||||
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"
|
||||
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"
|
||||
claude
|
||||
```
|
||||
|
||||
Two of these matter specifically for GLM-5.2:
|
||||
|
||||
- **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
|
||||
- **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side.
|
||||
|
||||
For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api).
|
||||
|
||||
### 3.5 Context Parallelism
|
||||
|
||||
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments:
|
||||
```bash
|
||||
--attn-cp-size 8 \
|
||||
--enable-prefill-cp \
|
||||
--cp-strategy interleave \
|
||||
```
|
||||
which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill.
|
||||
|
||||
When deploying with PD Disaggregation, the prefill node can choose to enable [LayerSplit](https://z.ai/blog/scaling-pain) technique with
|
||||
```bash
|
||||
--enable-dsa-cache-layer-split \
|
||||
--attn-cp-size 8 \
|
||||
--cp-strategy interleave \
|
||||
```
|
||||
With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side.
|
||||
@@ -0,0 +1,675 @@
|
||||
---
|
||||
title: GLM-5
|
||||
metatags:
|
||||
description: "Deploy GLM-5 with SGLang on NVIDIA H100/H200/B200 and AMD MI300X/MI325X/MI355X — state-of-the-art reasoning, enhanced coding, and robust tool calling capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity.
|
||||
|
||||
With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks
|
||||
- **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2)
|
||||
- **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs.
|
||||
|
||||
import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.jsx'
|
||||
|
||||
<GLM5Deployment />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.
|
||||
</Warning>
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
|
||||
- **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
|
||||
- The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
|
||||
- BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H100</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=16</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=32</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>H200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>tp=8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=16</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI300X/MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>tp=8</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
- **B200 (FP8)**: Use `--ep 1 --attention-backend dsa --dsa-decode-backend trtllm --dsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized DSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization.
|
||||
|
||||
- **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.
|
||||
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
|
||||
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
|
||||
|
||||
<Warning>
|
||||
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off.
|
||||
</Warning>
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy GLM-5 with the following command (FP8 on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5-FP8 \
|
||||
--tp 8 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--enable-flashinfer-allreduce-fusion \
|
||||
--mem-fraction-static 0.85 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command
|
||||
|
||||
The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path zai-org/GLM-5 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--dsa-prefill-backend tilelang \
|
||||
--dsa-decode-backend tilelang \
|
||||
--chunked-prefill-size 131072 \
|
||||
--mem-fraction-static 0.80 \
|
||||
--watchdog-timeout 1200 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Reasoning Parser
|
||||
|
||||
GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:
|
||||
|
||||
- **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
|
||||
- **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.
|
||||
|
||||
**Example 1: Thinking Mode (Default)**
|
||||
|
||||
Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Thinking mode is enabled by default, no extra parameters needed
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user wants me to solve a math problem: "What is 15% of 240?".
|
||||
|
||||
Step 1: Understand the problem. I need to calculate a percentage of a number.
|
||||
Formula: Percentage × Number = Result.
|
||||
|
||||
Step 2: Convert the percentage to a decimal or fraction.
|
||||
15% = 15/100 or 0.15.
|
||||
|
||||
Step 3: Perform the multiplication.
|
||||
Method A: Decimal multiplication.
|
||||
0.15 × 240.
|
||||
Break it down:
|
||||
10% of 240 = 24.
|
||||
5% is half of 10%, so 12.
|
||||
15% = 10% + 5% = 24 + 12 = 36.
|
||||
|
||||
Method B: Fraction multiplication.
|
||||
15/100 × 240.
|
||||
Simplify 240/100 = 2.4.
|
||||
15 × 2.4.
|
||||
10 × 2.4 = 24.
|
||||
5 × 2.4 = 12.
|
||||
24 + 12 = 36.
|
||||
|
||||
Method C: Direct multiplication.
|
||||
240 × 0.15.
|
||||
240 × 0.10 = 24.
|
||||
240 × 0.05 = 12.
|
||||
24 + 12 = 36.
|
||||
|
||||
Step 4: Final Verification.
|
||||
Is 36 reasonable?
|
||||
10% is 24. 20% is 48.
|
||||
15% is halfway between 10% and 20%.
|
||||
Halfway between 24 and 48 is 36.
|
||||
The result is correct.
|
||||
|
||||
Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%).
|
||||
=============== Content =================
|
||||
Here is the step-by-step solution:
|
||||
|
||||
**Step 1: Convert the percentage to a decimal.**
|
||||
To convert 15% to a decimal, divide by 100.
|
||||
$$15\% = \frac{15}{100} = 0.15$$
|
||||
|
||||
**Step 2: Multiply the decimal by the number.**
|
||||
Now, multiply 0.15 by 240.
|
||||
$$0.15 \times 240$$
|
||||
|
||||
**Step 3: Perform the calculation.**
|
||||
You can break this down to make it easier:
|
||||
$$0.15 = 0.10 + 0.05$$
|
||||
|
||||
* First, find 10% of 240:
|
||||
$$0.10 \times 240 = 24$$
|
||||
* Next, find 5% (which is half of 10%):
|
||||
$$\frac{24}{2} = 12$$
|
||||
* Add the two results together:
|
||||
$$24 + 12 = 36$$
|
||||
|
||||
**Answer:**
|
||||
15% of 240 is **36**.
|
||||
```
|
||||
|
||||
**Example 2: Instruct Mode (Thinking Off)**
|
||||
|
||||
To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Disable thinking mode via chat_template_kwargs
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What is 15% of 240?"}
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# In Instruct mode, the model responds directly without reasoning_content
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
To find **15% of 240**, follow these steps:
|
||||
|
||||
### Step 1: Convert the Percentage to a Decimal
|
||||
First, convert the percentage to a decimal by dividing by 100.
|
||||
|
||||
\[
|
||||
15\% = \frac{15}{100} = 0.15
|
||||
\]
|
||||
|
||||
### Step 2: Multiply by the Number
|
||||
Next, multiply the decimal by the number you want to find the percentage of.
|
||||
|
||||
\[
|
||||
0.15 \times 240
|
||||
\]
|
||||
|
||||
### Step 3: Perform the Multiplication
|
||||
Calculate the multiplication:
|
||||
|
||||
\[
|
||||
0.15 \times 240 = 36
|
||||
\]
|
||||
|
||||
### Final Answer
|
||||
\[
|
||||
\boxed{36}
|
||||
\]
|
||||
```
|
||||
|
||||
#### 4.3.2 Tool Calling
|
||||
|
||||
GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-5-FP8",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
if tool_call.function:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required:
|
||||
|
||||
- location: required, should be "Beijing"
|
||||
- unit: optional (not in required array), can be "celsius" or "fahrenheit"
|
||||
|
||||
Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you.
|
||||
=============== Content =================
|
||||
Tool Call: get_weather
|
||||
Arguments:
|
||||
Tool Call: None
|
||||
Arguments: {
|
||||
Tool Call: None
|
||||
Arguments: "location": "Be
|
||||
Tool Call: None
|
||||
Arguments: ijing"
|
||||
Tool Call: None
|
||||
Arguments: }
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: H200 (8x)
|
||||
- Model: GLM-5-FP8
|
||||
- Tensor Parallelism: 8
|
||||
- SGLang Version: commit 947927bdb
|
||||
|
||||
#### 5.1.1 Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 35.78
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4213
|
||||
Request throughput (req/s): 0.28
|
||||
Input token throughput (tok/s): 170.54
|
||||
Output token throughput (tok/s): 117.96
|
||||
Peak output token throughput (tok/s): 148.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 288.50
|
||||
Concurrency: 1.00
|
||||
Accept length: 3.48
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3576.31
|
||||
Median E2E Latency (ms): 2935.97
|
||||
P90 E2E Latency (ms): 5908.97
|
||||
P99 E2E Latency (ms): 8588.08
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 290.88
|
||||
Median TTFT (ms): 282.34
|
||||
P99 TTFT (ms): 332.27
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.54
|
||||
Median TPOT (ms): 6.97
|
||||
P99 TPOT (ms): 9.04
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.80
|
||||
Median ITL (ms): 6.81
|
||||
P95 ITL (ms): 13.51
|
||||
P99 ITL (ms): 26.99
|
||||
Max ITL (ms): 29.50
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/GLM-5-FP8 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 1000 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 1000
|
||||
Benchmark duration (s): 411.74
|
||||
Total input tokens: 502493
|
||||
Total input text tokens: 502493
|
||||
Total generated tokens: 500251
|
||||
Total generated tokens (retokenized): 499614
|
||||
Request throughput (req/s): 2.43
|
||||
Input token throughput (tok/s): 1220.41
|
||||
Output token throughput (tok/s): 1214.97
|
||||
Peak output token throughput (tok/s): 2648.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 2435.38
|
||||
Concurrency: 96.30
|
||||
Accept length: 3.50
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 39648.76
|
||||
Median E2E Latency (ms): 39058.12
|
||||
P90 E2E Latency (ms): 57009.82
|
||||
P99 E2E Latency (ms): 68880.33
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 20613.80
|
||||
Median TTFT (ms): 21429.21
|
||||
P99 TTFT (ms): 29543.17
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 38.73
|
||||
Median TPOT (ms): 36.52
|
||||
P99 TPOT (ms): 67.09
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 38.13
|
||||
Median ITL (ms): 16.57
|
||||
P95 ITL (ms): 86.01
|
||||
P99 ITL (ms): 164.88
|
||||
Max ITL (ms): 1307.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
<Note>
|
||||
The accuracy benchmark results below are shared with GLM-5.1, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate GLM-5.1 benchmark run is planned.
|
||||
</Note>
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.000
|
||||
Latency: 32.470 s
|
||||
Output throughput: 642.044 token/s
|
||||
```
|
||||
|
||||
#### 5.2.2 MMLU Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/mmlu/bench_sglang.py --port 30000
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
subject: abstract_algebra, #q:100, acc: 0.860
|
||||
subject: anatomy, #q:135, acc: 0.874
|
||||
subject: astronomy, #q:152, acc: 0.941
|
||||
subject: business_ethics, #q:100, acc: 0.880
|
||||
subject: clinical_knowledge, #q:265, acc: 0.932
|
||||
subject: college_biology, #q:144, acc: 0.972
|
||||
subject: college_chemistry, #q:100, acc: 0.640
|
||||
subject: college_computer_science, #q:100, acc: 0.900
|
||||
subject: college_mathematics, #q:100, acc: 0.810
|
||||
subject: college_medicine, #q:173, acc: 0.873
|
||||
subject: college_physics, #q:102, acc: 0.912
|
||||
subject: computer_security, #q:100, acc: 0.880
|
||||
subject: conceptual_physics, #q:235, acc: 0.928
|
||||
subject: econometrics, #q:114, acc: 0.807
|
||||
subject: electrical_engineering, #q:145, acc: 0.897
|
||||
subject: elementary_mathematics, #q:378, acc: 0.937
|
||||
subject: formal_logic, #q:126, acc: 0.778
|
||||
subject: global_facts, #q:100, acc: 0.710
|
||||
subject: high_school_biology, #q:310, acc: 0.961
|
||||
subject: high_school_chemistry, #q:203, acc: 0.847
|
||||
subject: high_school_computer_science, #q:100, acc: 0.960
|
||||
subject: high_school_european_history, #q:165, acc: 0.891
|
||||
subject: high_school_geography, #q:198, acc: 0.960
|
||||
subject: high_school_government_and_politics, #q:193, acc: 0.984
|
||||
subject: high_school_macroeconomics, #q:390, acc: 0.923
|
||||
subject: high_school_mathematics, #q:270, acc: 0.696
|
||||
subject: high_school_microeconomics, #q:238, acc: 0.962
|
||||
subject: high_school_physics, #q:151, acc: 0.821
|
||||
subject: high_school_psychology, #q:545, acc: 0.956
|
||||
subject: high_school_statistics, #q:216, acc: 0.889
|
||||
subject: high_school_us_history, #q:204, acc: 0.941
|
||||
subject: high_school_world_history, #q:237, acc: 0.945
|
||||
subject: human_aging, #q:223, acc: 0.857
|
||||
subject: human_sexuality, #q:131, acc: 0.908
|
||||
subject: international_law, #q:121, acc: 0.934
|
||||
subject: jurisprudence, #q:108, acc: 0.907
|
||||
subject: logical_fallacies, #q:163, acc: 0.933
|
||||
subject: machine_learning, #q:112, acc: 0.830
|
||||
subject: management, #q:103, acc: 0.942
|
||||
subject: marketing, #q:234, acc: 0.940
|
||||
subject: medical_genetics, #q:100, acc: 0.990
|
||||
subject: miscellaneous, #q:783, acc: 0.959
|
||||
subject: moral_disputes, #q:346, acc: 0.873
|
||||
subject: moral_scenarios, #q:895, acc: 0.837
|
||||
subject: nutrition, #q:306, acc: 0.922
|
||||
subject: philosophy, #q:311, acc: 0.897
|
||||
subject: prehistory, #q:324, acc: 0.929
|
||||
subject: professional_accounting, #q:282, acc: 0.844
|
||||
subject: professional_law, #q:1534, acc: 0.714
|
||||
subject: professional_medicine, #q:272, acc: 0.941
|
||||
subject: professional_psychology, #q:612, acc: 0.913
|
||||
subject: public_relations, #q:110, acc: 0.791
|
||||
subject: security_studies, #q:245, acc: 0.878
|
||||
subject: sociology, #q:201, acc: 0.940
|
||||
subject: us_foreign_policy, #q:100, acc: 0.920
|
||||
subject: virology, #q:166, acc: 0.596
|
||||
subject: world_religions, #q:171, acc: 0.936
|
||||
Total latency: 165.275
|
||||
Average accuracy: 0.877
|
||||
```
|
||||
|
||||
### 5.3 AMD GPU Benchmarks
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark (MI325/MI35x)
|
||||
|
||||
- MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang DSA backends)
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --num-questions 200
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.970
|
||||
Invalid: 0.000
|
||||
```
|
||||
|
||||
Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).
|
||||
@@ -0,0 +1,829 @@
|
||||
---
|
||||
title: GLM Glyph
|
||||
metatags:
|
||||
description: "Deploy GLM-Glyph with SGLang - community contribution guide for Zhipu AI's GLM Glyph model deployment."
|
||||
---
|
||||
|
||||
import { GLMGlyphDeployment } from '/src/snippets/autoregressive/glm-glyph-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
|
||||
- **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
|
||||
- **High Performance**: Optimized for both throughput and latency scenarios
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph)
|
||||
- **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8)
|
||||
|
||||
**License:**
|
||||
|
||||
Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options.
|
||||
|
||||
<GLMGlyphDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count. See the [GLM-4.5 cookbook page](/cookbook/autoregressive/GLM/GLM-4.5) for the full Thinking Budget usage example.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Thinking Mode
|
||||
|
||||
Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/Glyph \
|
||||
--reasoning-parser glm45 \
|
||||
--tp 4
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.
|
||||
|
||||
**Disable Thinking Mode:**
|
||||
|
||||
To disable thinking mode for a specific request:
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[{"role": "user", "content": "What is the capital of France?"}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}}
|
||||
)
|
||||
```
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
Glyph supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path zai-org/Glyph \
|
||||
--reasoning-parser glm45 \
|
||||
--tool-call-parser glm45 \
|
||||
--tp 4
|
||||
```
|
||||
|
||||
**Python Example (with Thinking Process):**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
|
||||
I should call the function with location="Beijing".
|
||||
=============== Content =================
|
||||
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
**Note:**
|
||||
|
||||
- The reasoning parser shows how the model decides to use a tool
|
||||
- Tool calls are clearly marked with the function name and arguments
|
||||
- You can then execute the function and send the result back to continue the conversation
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="zai-org/Glyph",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The weather in Beijing is currently 22°C and sunny."
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Model: Glyph
|
||||
- SGLang Version: 0.5.6.post1
|
||||
|
||||
**Benchmark Methodology:**
|
||||
|
||||
We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model zai-org/Glyph \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 17.03
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.59
|
||||
Input token throughput (tok/s): 358.17
|
||||
Output token throughput (tok/s): 247.74
|
||||
Peak output token throughput (tok/s): 251.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 605.91
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1702.14
|
||||
Median E2E Latency (ms): 1361.72
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 22.35
|
||||
Median TTFT (ms): 22.61
|
||||
P99 TTFT (ms): 23.76
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.99
|
||||
Median TPOT (ms): 3.99
|
||||
P99 TPOT (ms): 4.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.99
|
||||
Median ITL (ms): 3.99
|
||||
P95 ITL (ms): 4.03
|
||||
P99 ITL (ms): 4.12
|
||||
Max ITL (ms): 7.46
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 16.27
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40804
|
||||
Request throughput (req/s): 4.92
|
||||
Input token throughput (tok/s): 2438.06
|
||||
Output token throughput (tok/s): 2507.94
|
||||
Peak output token throughput (tok/s): 3069.00
|
||||
Peak concurrent requests: 26
|
||||
Total token throughput (tok/s): 4946.00
|
||||
Concurrency: 13.44
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2733.43
|
||||
Median E2E Latency (ms): 2892.98
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 33.10
|
||||
Median TTFT (ms): 27.73
|
||||
P99 TTFT (ms): 49.34
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.33
|
||||
Median TPOT (ms): 5.39
|
||||
P99 TPOT (ms): 5.86
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.30
|
||||
Median ITL (ms): 4.89
|
||||
P95 ITL (ms): 5.54
|
||||
P99 ITL (ms): 21.17
|
||||
Max ITL (ms): 25.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 25.67
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 252657
|
||||
Request throughput (req/s): 19.48
|
||||
Input token throughput (tok/s): 9733.69
|
||||
Output token throughput (tok/s): 9843.99
|
||||
Peak output token throughput (tok/s): 13398.00
|
||||
Peak concurrent requests: 127
|
||||
Total token throughput (tok/s): 19577.68
|
||||
Concurrency: 89.49
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 4593.75
|
||||
Median E2E Latency (ms): 4431.03
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 48.66
|
||||
Median TTFT (ms): 35.88
|
||||
P99 TTFT (ms): 120.61
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 9.10
|
||||
Median TPOT (ms): 9.55
|
||||
P99 TPOT (ms): 11.00
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.01
|
||||
Median ITL (ms): 6.51
|
||||
P95 ITL (ms): 23.19
|
||||
P99 ITL (ms): 25.54
|
||||
Max ITL (ms): 52.93
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Reasoning Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 201.53
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 44462
|
||||
Total generated tokens (retokenized): 44455
|
||||
Request throughput (req/s): 0.05
|
||||
Input token throughput (tok/s): 30.27
|
||||
Output token throughput (tok/s): 220.63
|
||||
Peak output token throughput (tok/s): 251.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 250.90
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20151.45
|
||||
Median E2E Latency (ms): 21576.31
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 2362.23
|
||||
Median TTFT (ms): 23.03
|
||||
P99 TTFT (ms): 21310.14
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.00
|
||||
Median TPOT (ms): 4.00
|
||||
P99 TPOT (ms): 4.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.00
|
||||
Median ITL (ms): 4.00
|
||||
P95 ITL (ms): 4.05
|
||||
P99 ITL (ms): 4.08
|
||||
Max ITL (ms): 5.67
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 118.67
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 318306
|
||||
Total generated tokens (retokenized): 318270
|
||||
Request throughput (req/s): 0.67
|
||||
Input token throughput (tok/s): 334.27
|
||||
Output token throughput (tok/s): 2682.26
|
||||
Peak output token throughput (tok/s): 3264.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3016.53
|
||||
Concurrency: 13.74
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 20387.23
|
||||
Median E2E Latency (ms): 20466.09
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 132.47
|
||||
Median TTFT (ms): 27.19
|
||||
P99 TTFT (ms): 583.15
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.09
|
||||
Median TPOT (ms): 5.13
|
||||
P99 TPOT (ms): 5.19
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.09
|
||||
Median ITL (ms): 5.08
|
||||
P95 ITL (ms): 5.18
|
||||
P99 ITL (ms): 5.57
|
||||
Max ITL (ms): 522.26
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 8000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 150.00
|
||||
Total input tokens: 158939
|
||||
Total input text tokens: 158939
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 1301025
|
||||
Total generated tokens (retokenized): 1300901
|
||||
Request throughput (req/s): 2.13
|
||||
Input token throughput (tok/s): 1059.59
|
||||
Output token throughput (tok/s): 8673.49
|
||||
Peak output token throughput (tok/s): 11899.00
|
||||
Peak concurrent requests: 71
|
||||
Total token throughput (tok/s): 9733.09
|
||||
Concurrency: 54.71
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 25645.42
|
||||
Median E2E Latency (ms): 26913.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 163.75
|
||||
Median TTFT (ms): 93.67
|
||||
P99 TTFT (ms): 426.19
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 6.27
|
||||
Median TPOT (ms): 6.39
|
||||
P99 TPOT (ms): 6.59
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 6.27
|
||||
Median ITL (ms): 0.17
|
||||
P95 ITL (ms): 32.94
|
||||
P99 ITL (ms): 67.89
|
||||
Max ITL (ms): 136.00
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.3 Summarization Scenario Benchmark
|
||||
|
||||
#### 5.1.3.1 Low Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 17.44
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.57
|
||||
Input token throughput (tok/s): 2405.19
|
||||
Output token throughput (tok/s): 242.00
|
||||
Peak output token throughput (tok/s): 250.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 2647.19
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1742.54
|
||||
Median E2E Latency (ms): 1412.47
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 53.48
|
||||
Median TTFT (ms): 45.05
|
||||
P99 TTFT (ms): 98.57
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.01
|
||||
Median TPOT (ms): 4.01
|
||||
P99 TPOT (ms): 4.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 4.01
|
||||
Median ITL (ms): 4.01
|
||||
P95 ITL (ms): 4.06
|
||||
P99 ITL (ms): 4.09
|
||||
Max ITL (ms): 4.95
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.2 Medium Concurrency
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 16.90
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41668
|
||||
Request throughput (req/s): 4.73
|
||||
Input token throughput (tok/s): 17753.58
|
||||
Output token throughput (tok/s): 2465.75
|
||||
Peak output token throughput (tok/s): 3005.00
|
||||
Peak concurrent requests: 25
|
||||
Total token throughput (tok/s): 20219.33
|
||||
Concurrency: 13.68
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 2890.33
|
||||
Median E2E Latency (ms): 3069.55
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 41.46
|
||||
Median TTFT (ms): 31.75
|
||||
P99 TTFT (ms): 93.18
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.52
|
||||
Median TPOT (ms): 5.58
|
||||
P99 TPOT (ms): 6.14
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.48
|
||||
Median ITL (ms): 5.13
|
||||
P95 ITL (ms): 5.93
|
||||
P99 ITL (ms): 20.76
|
||||
Max ITL (ms): 36.01
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.3.3 High Concurrency
|
||||
|
||||
- **Benchmark Command**:
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model zai-org/Glyph \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64 \
|
||||
--request-rate inf
|
||||
```
|
||||
- **Test Results**:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 35.54
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169994
|
||||
Request throughput (req/s): 9.01
|
||||
Input token throughput (tok/s): 35848.57
|
||||
Output token throughput (tok/s): 4783.96
|
||||
Peak output token throughput (tok/s): 8396.00
|
||||
Peak concurrent requests: 80
|
||||
Total token throughput (tok/s): 40632.53
|
||||
Concurrency: 59.26
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6580.96
|
||||
Median E2E Latency (ms): 6248.74
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 345.27
|
||||
Median TTFT (ms): 96.06
|
||||
P99 TTFT (ms): 2823.92
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 12.26
|
||||
Median TPOT (ms): 12.53
|
||||
P99 TPOT (ms): 23.58
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.76
|
||||
Median ITL (ms): 6.57
|
||||
P95 ITL (ms): 27.66
|
||||
P99 ITL (ms): 91.24
|
||||
Max ITL (ms): 2609.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.890
|
||||
Invalid: 0.000
|
||||
Latency: 3.718 s
|
||||
Output throughput: 5245.606 token/s
|
||||
```
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
title: GLM-OCR
|
||||
metatags:
|
||||
description: "Deploy GLM-OCR with SGLang - state-of-the-art OCR performance for complex document understanding."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization.
|
||||
|
||||
The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.
|
||||
|
||||
**Hardware Support:** NVIDIA B200/H100/H200
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction.
|
||||
- **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts.
|
||||
- **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments.
|
||||
- **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems.
|
||||
|
||||
For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding.
|
||||
|
||||
import { GLMOCRDeployment } from '/src/snippets/autoregressive/glm-ocr-deployment.jsx'
|
||||
|
||||
<GLMOCRDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT.
|
||||
- **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency.
|
||||
- **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 OCR Image Processing
|
||||
|
||||
GLM-OCR supports OCR tasks on various document types. Here's a basic example:
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Please extract all text from this image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-OCR",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
**Example Output:**
|
||||
|
||||
```text Output
|
||||
Response costs: 2.29s
|
||||
Generated text: CINNAMON SUGAR
|
||||
1 x 17,000 17,000
|
||||
|
||||
SUB TOTAL 17,000
|
||||
|
||||
GRAND TOTAL 17,000
|
||||
|
||||
CASH IDR 20,000
|
||||
|
||||
CHANGE DUE 3,000
|
||||
|
||||
```
|
||||
|
||||
#### 4.2.2 Complex Document Processing
|
||||
|
||||
GLM-OCR excels at processing complex documents including:
|
||||
|
||||
- **Tables**: Accurate extraction of tabular data with structure preservation
|
||||
- **Formulas**: Mathematical formula recognition
|
||||
- **Code Documents**: Source code extraction from screenshots
|
||||
- **Seals and Stamps**: Recognition of seals and stamps in documents
|
||||
- **Multi-layout Documents**: Mixed content with text, images, and tables
|
||||
|
||||
```python Example
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:30000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
# Example: Processing a document with tables
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "YOUR_DOCUMENT_IMAGE_URL"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Please extract the table content from this document and format it as markdown."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="zai-org/GLM-OCR",
|
||||
messages=messages,
|
||||
max_tokens=4096
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
Document model accuracy on standard benchmarks:
|
||||
|
||||
#### 5.1.1 OCRBench Benchmark
|
||||
|
||||
- Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python3 -m lmms_eval \
|
||||
--model openai_compatible \
|
||||
--model_args "model_version=zai-org/GLM-OCR" \
|
||||
--tasks ocrbench \
|
||||
--batch_size 128 \
|
||||
--log_samples \
|
||||
--log_samples_suffix "openai_compatible" \
|
||||
--output_path ./logs
|
||||
```
|
||||
|
||||
- Test Result
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
<col style={{width: "12.5%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Stderr</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yaml</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>ocrbench_accuracy</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.806</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>N/A</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 OmniDocBench V1.5
|
||||
|
||||
GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks.
|
||||
@@ -0,0 +1,306 @@
|
||||
---
|
||||
title: DiffusionGemma
|
||||
metatags:
|
||||
description: "Deploy DiffusionGemma with SGLang - Google's uniform-state renoising block-diffusion language model (26B-A4B MoE) served with the Gemma4Renoise sampler."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
DiffusionGemma is a uniform-state (renoising) block-diffusion language model from Google. An encoder builds causal context, and a decoder denoises a fixed-length bidirectional canvas of `canvas_length` tokens. The `Gemma4Renoise` sampler runs `max_denoising_steps` reverse steps over the canvas, feeding the previous step's logits back as self-conditioning and emitting the greedy argmax of the processed logits.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Uniform-State Renoising**: The canvas starts from random tokens and is refined each step by accepting confident positions and re-noising the rest, with no mask token.
|
||||
- **Encoder / Decoder Canvas**: The encoder produces causal context KV, the decoder attends bidirectionally over the canvas.
|
||||
- **Self-Conditioning**: Each step conditions on the previous step's logits.
|
||||
- **EntropyBound Acceptance**: Each step accepts the lowest-entropy canvas positions within an entropy budget and re-noises the rest.
|
||||
- **StableAndConfident Stopping**: A canvas stops early once it is stable and confident.
|
||||
- **MoE Architecture**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference.
|
||||
- **Multimodal Input**: Accepts text and image inputs (via a ~550M vision encoder) and generates text output.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "40.0%"}} />
|
||||
<col style={{width: "30.0%"}} />
|
||||
<col style={{width: "30.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameters</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>[google/diffusiongemma-26B-A4B-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MoE, uniform-state diffusion (text + image)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>25.2B total / 3.8B active</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Architecture Specifications:**
|
||||
|
||||
| Spec | Value |
|
||||
| --- | --- |
|
||||
| Total Parameters | 25.2B |
|
||||
| Active Parameters | 3.8B |
|
||||
| Layers | 30 |
|
||||
| Sliding Window | 1024 tokens |
|
||||
| Context Length | Up to 256K tokens |
|
||||
| Canvas Length | 256 |
|
||||
| Vocabulary Size | 262K |
|
||||
| Experts | 8 active / 128 total + 1 shared |
|
||||
| Supported Modalities | Text, Image |
|
||||
| Vision Encoder | ~550M parameters |
|
||||
|
||||
**License:**
|
||||
|
||||
Refer to the model card for license details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
The checkpoint ships its own modeling code, so `--trust-remote-code` is required when serving.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The required runtime settings are applied automatically for `Gemma4Renoise` (the Triton attention backend, eager mode, and unchunked prefill, needed because the full-attention head_dim is 512 and the canvas uses bidirectional attention), so a default launch works:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path google/diffusiongemma-26B-A4B-it \
|
||||
--dllm-algorithm Gemma4Renoise \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**dLLM-Specific Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diffusion decoding algorithm</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`Gemma4Renoise`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required to load the checkpoint's modeling code</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm-config`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional YAML overriding the renoise schedule</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Checkpoint defaults</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The attention backend, eager mode, and unchunked prefill are selected automatically for `Gemma4Renoise`, so they do not need to be passed on the command line.
|
||||
|
||||
Sampling is governed by the renoise schedule. Request-level `logprobs`, penalties, `logit_bias`, and grammar / structured output (`json_schema` / `regex` / `ebnf` / `structural_tag`) are not applied and are rejected with a 400. Core sampling controls (`temperature`, `top_k`, `top_p`) are accepted but have no effect. Streaming is block-level: one fully-denoised canvas per chunk.
|
||||
|
||||
**Gemma4Renoise Config** (defaults follow the checkpoint's `generation_config.json`):
|
||||
|
||||
```yaml Config
|
||||
# Number of reverse denoising steps per canvas.
|
||||
max_denoising_steps: 48
|
||||
# Optional. Makes the renoise sampling reproducible (also shared across TP ranks).
|
||||
seed: 1234
|
||||
sampler_config:
|
||||
# Entropy budget. Accept the lowest-entropy canvas positions within this bound each step (the rest are re-noised).
|
||||
entropy_bound: 0.1
|
||||
# Linear temperature schedule applied over the denoising steps.
|
||||
temperature_schedule:
|
||||
t_min: 0.4
|
||||
t_max: 0.8
|
||||
# Stop early once the canvas is stable and confident.
|
||||
stopping_config:
|
||||
confidence_threshold: 0.005
|
||||
stability_threshold: 1
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Deployment
|
||||
|
||||
Start the server with the command from [Section 3.1](#3-1-basic-configuration).
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="google/diffusiongemma-26B-A4B-it",
|
||||
messages=[
|
||||
{"role": "user", "content": "What are the key differences between TCP and UDP?"}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 4.3 Streaming
|
||||
|
||||
Streaming emits one fully-denoised canvas per chunk.
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="google/diffusiongemma-26B-A4B-it",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
Not benchmarked for speed.
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
Full test splits, every item scored (no failed-request exclusions). Text MCQ benchmarks use greedy generate-and-parse, MATH uses boxed-answer extraction plus sympy equivalence. MMLU, ARC-Challenge, and MATH-500 are the mean of two independent server launches.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50.0%"}} />
|
||||
<col style={{width: "50.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Benchmark</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Score</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GSM8K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>95.4%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ARC-Challenge</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>91.6%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HumanEval</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.7% pass@1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMLU</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>76.2%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMLU-Pro</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>73.7%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GSM-Symbolic</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.2%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MATH-500</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>72.1%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AIME-2026</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>10.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HMMT-Feb-2025</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>10.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>GPQA-main</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>59.2%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
Multimodal, full standard split per task (MMMU / MMMU-Pro / MMStar / AI2D as multiple-choice, MathVista testmini, DocVQA by ANLS, ChartQA by relaxed accuracy):
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50.0%"}} />
|
||||
<col style={{width: "50.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Multimodal benchmark</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Score</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMMU (val, MC)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>64.9%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMMU-Pro (standard 10-opt, MC)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>57.3%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MathVista (testmini)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>68.4%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DocVQA (val)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>85.9%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ChartQA (test)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>61.7%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI2D (test)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>78.7%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MMStar (val)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>65.9%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: EmbeddingGemma
|
||||
description: Serve Google's EmbeddingGemma text embedding model with SGLang.
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
[EmbeddingGemma](https://huggingface.co/google/embeddinggemma-300m) is Google's 300M-parameter text embedding model. SGLang detects its bidirectional Gemma 3 encoder, applies normalized mean pooling, and serves embeddings through the OpenAI-compatible `/v1/embeddings` endpoint.
|
||||
|
||||
On NVIDIA CUDA, SGLang uses breakable CUDA graph (BCG) for its complete prefill by default. It also disables prefix caching and chunked prefill, which are incompatible with this bidirectional encoder.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- NVIDIA CUDA GPU.
|
||||
- A Hugging Face account that has accepted the [EmbeddingGemma license](https://huggingface.co/google/embeddinggemma-300m).
|
||||
- A Hugging Face access token. Export it before starting the server so it can download the gated checkpoint:
|
||||
|
||||
```bash
|
||||
export HF_TOKEN=<your-hugging-face-token>
|
||||
```
|
||||
|
||||
Install an SGLang build that includes EmbeddingGemma support:
|
||||
|
||||
```bash
|
||||
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
```
|
||||
|
||||
## Start the server
|
||||
|
||||
The standard configuration detects EmbeddingGemma and enables embedding mode,
|
||||
BCG, and the checkpoint's BF16 dtype automatically:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path google/embeddinggemma-300m \
|
||||
--host 0.0.0.0
|
||||
```
|
||||
|
||||
### Hopper performance defaults
|
||||
|
||||
On H100 and H200, SGLang automatically selects FA3 and captures BCG through
|
||||
16,384 tokens, covering eight 2K embedding requests in one replay. No extra
|
||||
performance flags are required for this workload.
|
||||
|
||||
To capture larger aggregate prefills, raise the BCG tier explicitly:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--model-path google/embeddinggemma-300m \
|
||||
--cuda-graph-max-bs-prefill 32768 \
|
||||
--host 0.0.0.0
|
||||
```
|
||||
|
||||
EmbeddingGemma automatically enables batch tokenization for list-valued
|
||||
embedding requests, so do not add a separate tokenizer batching flag.
|
||||
|
||||
## Create embeddings
|
||||
|
||||
Send one string or a batch of strings to the OpenAI-compatible endpoint:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:30000/v1/embeddings \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"model": "google/embeddinggemma-300m",
|
||||
"input": [
|
||||
"A short guide to serving text embeddings.",
|
||||
"Vector search retrieves semantically similar documents."
|
||||
],
|
||||
"encoding_format": "float"
|
||||
}'
|
||||
```
|
||||
|
||||
See [OpenAI-compatible embedding APIs](/docs/basic_usage/openai_api_embeddings) for Python and OpenAI client examples.
|
||||
|
||||
## Deployment behavior
|
||||
|
||||
EmbeddingGemma performs bidirectional attention over the complete input, so reusing a prefix KV cache or splitting the input into chunked prefills would produce incorrect attention states. SGLang applies the required settings automatically:
|
||||
|
||||
- disables RadixAttention prefix caching;
|
||||
- disables chunked prefill;
|
||||
- disables the decode CUDA graph because this is an embedding-only model;
|
||||
- uses BCG for CUDA prefill;
|
||||
- uses the FlashAttention raw-K/V path when the prefill backend is FA3 or FA4 on supported Hopper and Blackwell CUDA GPUs.
|
||||
|
||||
No prefill CUDA-graph override is required for this recipe. Keep BCG enabled to use the optimized EmbeddingGemma path.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,700 @@
|
||||
---
|
||||
title: LLaDA 2.1
|
||||
metatags:
|
||||
description: "Deploy LLaDA 2.1 with SGLang - large-scale discrete diffusion language model with parallel token generation, iterative denoising, MoE architecture, and reinforcement learning for reasoning."
|
||||
---
|
||||
|
||||
import { LLaDA21Deployment } from '/src/snippets/autoregressive/llada-21-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight
|
||||
- **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores
|
||||
- **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling
|
||||
- **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following
|
||||
- **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
<col style={{width: "20.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameters</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFace</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-mini**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE (20 layers, 16 attention heads)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**LLaDA2.1-flash**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>32,768 tokens</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**License:**
|
||||
|
||||
Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs.
|
||||
|
||||
<LLaDA21Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**dLLM-Specific Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dllm-algorithm`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Diffusion decoding algorithm</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`JointThreshold`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for LLaDA model loading</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.8`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--max-running-requests`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Maximum concurrent requests</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1` (for best quality)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention computation backend</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Decoding Mode Comparison:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Threshold</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Speed</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Quality</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Best For</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Quality Mode (Q)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Conservative</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Moderate</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Higher benchmark scores</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Accuracy-critical tasks</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Speed Mode (S)**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Aggressive</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Very fast, relies on T2T editing</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Slightly lower</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Throughput-critical tasks</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Hardware Requirements:**
|
||||
|
||||
- **LLaDA2.1-mini (16B)**: ~47 GB VRAM, runs on a single GPU (TP=1)
|
||||
- **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200)
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Deployment
|
||||
|
||||
Start the server using the command generated above, for example:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-mini \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 1 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### 4.2 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
**Simple Completion Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Explain what a diffusion language model is in simple terms."}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
Sure! Let's break it down in simple terms.
|
||||
|
||||
A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text.
|
||||
|
||||
Here’s how it works, using a simple real-life analogy:
|
||||
|
||||
Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together.
|
||||
|
||||
Now, think of the process like this:
|
||||
|
||||
1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper).
|
||||
2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book.
|
||||
3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start.
|
||||
4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word.
|
||||
|
||||
So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story.
|
||||
|
||||
In short:
|
||||
A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned.
|
||||
```
|
||||
|
||||
### 4.3 Advanced Usage
|
||||
|
||||
#### 4.3.1 Streaming
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
Here are several ways to implement the Fibonacci sequence in Python:
|
||||
|
||||
## 1. Recursive Approach (Simple but Inefficient)
|
||||
|
||||
```python
|
||||
def fibonacci_recursive(n):
|
||||
"""
|
||||
Compute the nth Fibonacci number using recursion.
|
||||
|
||||
Args:
|
||||
n (int): The position in the Fibonacci sequence (0-indexed)
|
||||
|
||||
Returns:
|
||||
int: The nth Fibonacci number
|
||||
|
||||
Raises:
|
||||
ValueError: If n is negative
|
||||
"""
|
||||
if n < 0:
|
||||
raise ValueError("n must be non-negative")
|
||||
|
||||
if n <= 1:
|
||||
return n
|
||||
|
||||
return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)
|
||||
|
||||
# Example usage
|
||||
print(fibonacci_recursive(10)) # Output: 55
|
||||
```
|
||||
|
||||
## 2. Iterative Approach (Efficient)
|
||||
...
|
||||
````
|
||||
|
||||
#### 4.3.2 Code Generation
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/LLaDA2.1-mini",
|
||||
messages=[
|
||||
{"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."}
|
||||
],
|
||||
max_tokens=2048
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
```python
|
||||
def is_palindrome(s):
|
||||
"""
|
||||
Check if a string is a palindrome.
|
||||
|
||||
A palindrome is a word, phrase, or sequence that reads the same backward as forward.
|
||||
This function ignores case, spaces, punctuation, and non characters characters.
|
||||
|
||||
Args:
|
||||
s (str): The string to check
|
||||
|
||||
Returns:
|
||||
bool: True if the string is a palindrome, False otherwise
|
||||
|
||||
Examples:
|
||||
>>> is_palindrome("racecar")
|
||||
True
|
||||
>>> is_palindrome("A man a plan a canal Panama")
|
||||
True
|
||||
>>> is_palindrome("race a car")
|
||||
False
|
||||
>>> is_palindrome("")
|
||||
True
|
||||
>>> is_palindrome("a")
|
||||
True
|
||||
"""
|
||||
# Remove non-alphanumeric characters and convert to lowercase
|
||||
cleaned = ''.join(char.lower() for char in s if char.isalnum())
|
||||
|
||||
# Check if the cleaned string reads the same forwards and backwards
|
||||
return cleaned == cleaned[::-1]
|
||||
|
||||
# Test cases
|
||||
def test_is_palindrome():
|
||||
"""Test the is_palindrome function with various inputs."""
|
||||
|
||||
# Test basic palindromes
|
||||
assert is_palindrome("racecar") == True
|
||||
assert is_palindrome("level") == True
|
||||
assert is_palindrome("madam") == True
|
||||
assert is_palindrome("radar") == True
|
||||
|
||||
# Test palindromes with spaces and punctuation
|
||||
assert is_palindrome("A man a plan a canal Panama") == True
|
||||
assert is_palindrome("race a car") == False
|
||||
assert is_palindrome("Was it a car or a cat I saw?") == True
|
||||
assert is_palindrome("Madam, I'm Adam") == True
|
||||
|
||||
# Test edge cases
|
||||
assert is_palindrome("") == True
|
||||
assert is_palindrome("a") == True
|
||||
assert is_palindrome("A") == True
|
||||
assert is_palindrome("Aa") == True
|
||||
|
||||
# Test non-palindromes
|
||||
assert is_palindrome("hello") == False
|
||||
assert is_palindrome("world") == False
|
||||
assert is_palindrome("python") == False
|
||||
|
||||
# Test single characters
|
||||
assert is_palindrome("1") == True
|
||||
assert is_palindrome("1") == True
|
||||
|
||||
print("All tests passed!")
|
||||
|
||||
# Run the tests
|
||||
if __name__ == "__main__":
|
||||
# Example usage
|
||||
print("Testing isalindrome function:")
|
||||
print(f"'racecar' {is_palindrome('racecar')}")
|
||||
print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}")
|
||||
print(f"'race a car': {is_palindrome('race a car')}")
|
||||
print(f"'hello': {is_palindrome('hello')}")
|
||||
|
||||
# Run tests
|
||||
test_is_palindrome()
|
||||
```
|
||||
|
||||
This implementation includes:
|
||||
|
||||
1. **Comprehensive function** `is_palindrome()` that:
|
||||
- Ignores case by converting to lowercase
|
||||
- Removes all non-alphanumeric characters (spaces, punctuation, etc.)
|
||||
- Uses string slicing (`[::-1]`) to reverse the string
|
||||
|
||||
2. **Detailed docstring** explaining:
|
||||
- What the function does
|
||||
- How it works
|
||||
- Return value
|
||||
- Examples of usage
|
||||
|
||||
3. **Extensive test cases** covering:
|
||||
- Basic palindromes
|
||||
- Palindromes with spaces and punctuation
|
||||
- Edge cases (empty string, single character)
|
||||
- Non-palindromes
|
||||
- Mixed case scenarios
|
||||
|
||||
4. **Test function** that uses assertions to verify the function works correctly
|
||||
|
||||
The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama".
|
||||
````
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA B200 (4x)
|
||||
- SGLang Version: 0.5.8+
|
||||
|
||||
#### 5.1.1 LLaDA2.1-mini
|
||||
|
||||
**Model Deployment:**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-mini \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 1 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer
|
||||
```
|
||||
|
||||
- Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-mini \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Latency Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 9.90
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3433
|
||||
Request throughput (req/s): 1.01
|
||||
Input token throughput (tok/s): 616.26
|
||||
Output token throughput (tok/s): 426.26
|
||||
Peak output token throughput (tok/s): 1010.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 1042.53
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 988.87
|
||||
Median E2E Latency (ms): 655.27
|
||||
P90 E2E Latency (ms): 1952.50
|
||||
P99 E2E Latency (ms): 2932.19
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 152.74
|
||||
Median TTFT (ms): 150.37
|
||||
P99 TTFT (ms): 229.78
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 2.16
|
||||
Median TPOT (ms): 2.08
|
||||
P99 TPOT (ms): 3.72
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 2.10
|
||||
Median ITL (ms): 1.99
|
||||
P95 ITL (ms): 4.03
|
||||
P99 ITL (ms): 6.34
|
||||
Max ITL (ms): 26.59
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-mini \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Throughput Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 467.74
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 189717
|
||||
Request throughput (req/s): 1.07
|
||||
Input token throughput (tok/s): 534.12
|
||||
Output token throughput (tok/s): 540.17
|
||||
Peak output token throughput (tok/s): 1753.00
|
||||
Peak concurrent requests: 105
|
||||
Total token throughput (tok/s): 1074.30
|
||||
Concurrency: 90.77
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 84912.27
|
||||
Median E2E Latency (ms): 86564.26
|
||||
P90 E2E Latency (ms): 110567.26
|
||||
P99 E2E Latency (ms): 114303.38
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 83920.39
|
||||
Median TTFT (ms): 85669.54
|
||||
P99 TTFT (ms): 112969.91
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 2.67
|
||||
Median TPOT (ms): 1.65
|
||||
P99 TPOT (ms): 4.43
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 1.69
|
||||
Median ITL (ms): 1.46
|
||||
P95 ITL (ms): 3.96
|
||||
P99 ITL (ms): 4.84
|
||||
Max ITL (ms): 92.08
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 LLaDA2.1-flash
|
||||
|
||||
**Model Deployment:**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path inclusionAI/LLaDA2.1-flash \
|
||||
--dllm-algorithm JointThreshold \
|
||||
--tp 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 1 \
|
||||
--attention-backend flashinfer
|
||||
```
|
||||
|
||||
- Latency Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Latency Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 14.46
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 3276
|
||||
Request throughput (req/s): 0.69
|
||||
Input token throughput (tok/s): 421.79
|
||||
Output token throughput (tok/s): 291.75
|
||||
Peak output token throughput (tok/s): 676.00
|
||||
Peak concurrent requests: 3
|
||||
Total token throughput (tok/s): 713.53
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 1445.16
|
||||
Median E2E Latency (ms): 968.06
|
||||
P90 E2E Latency (ms): 3101.86
|
||||
P99 E2E Latency (ms): 4208.49
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 231.63
|
||||
Median TTFT (ms): 242.67
|
||||
P99 TTFT (ms): 341.33
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 3.04
|
||||
Median TPOT (ms): 2.79
|
||||
P99 TPOT (ms): 5.33
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 3.05
|
||||
Median ITL (ms): 2.41
|
||||
P95 ITL (ms): 7.25
|
||||
P99 ITL (ms): 8.27
|
||||
Max ITL (ms): 29.27
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Throughput Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model inclusionAI/LLaDA2.1-flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- **Throughput Result**:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 671.85
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 177961
|
||||
Request throughput (req/s): 0.74
|
||||
Input token throughput (tok/s): 371.85
|
||||
Output token throughput (tok/s): 376.07
|
||||
Peak output token throughput (tok/s): 1521.00
|
||||
Peak concurrent requests: 103
|
||||
Total token throughput (tok/s): 747.92
|
||||
Concurrency: 91.28
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 122658.36
|
||||
Median E2E Latency (ms): 125265.55
|
||||
P90 E2E Latency (ms): 159554.07
|
||||
P99 E2E Latency (ms): 165174.88
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 121009.17
|
||||
Median TTFT (ms): 124437.80
|
||||
P99 TTFT (ms): 163579.29
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 4.73
|
||||
Median TPOT (ms): 2.16
|
||||
P99 TPOT (ms): 7.13
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 2.38
|
||||
Median ITL (ms): 1.40
|
||||
P95 ITL (ms): 6.89
|
||||
P99 ITL (ms): 8.60
|
||||
Max ITL (ms): 176.78
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
```bash Command
|
||||
python -m sglang.test.few_shot_gsm8k \
|
||||
--num-questions 200 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**Results:**
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.895
|
||||
Invalid: 0.000
|
||||
Latency: 100.552 s
|
||||
Output throughput: 262.094 token/s
|
||||
```
|
||||
@@ -0,0 +1,217 @@
|
||||
---
|
||||
title: Ling-2.5-1T
|
||||
metatags:
|
||||
description: "Deploy Ling-2.5-1T with SGLang - 1T parameter MoE model with 63B active parameters, trillion-scale context length up to 1M tokens, and agentic tool calling capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens.
|
||||
- **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume ~4x the output tokens.
|
||||
- **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following.
|
||||
- **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4.
|
||||
- **Context Length**: 256K -> 1M (YaRN)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ling-2.5-1T runs on the standard SGLang Docker image:
|
||||
|
||||
```bash Command
|
||||
# NVIDIA (H200 / B200 / GB200 / GB300)
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/install) guide for setup instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform.
|
||||
|
||||
import { Ling251TDeployment } from '/src/snippets/autoregressive/ling-25-1t-deployment.jsx'
|
||||
|
||||
<Ling251TDeployment />
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
|
||||
- `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes.
|
||||
- Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading.
|
||||
- On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For example, launch the server on 2 H200 nodes:
|
||||
|
||||
```bash Command
|
||||
export MASTER_IP=10.10.0.1 # The IP of Node 0
|
||||
export PORT=30000
|
||||
export DIST_PORT=50000
|
||||
|
||||
# Node 0:
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path inclusionAI/Ling-2.5-1T \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--pp-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
|
||||
--mem-frac 0.95
|
||||
|
||||
|
||||
# Node 1:
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path inclusionAI/Ling-2.5-1T \
|
||||
--trust-remote-code \
|
||||
--tp-size 8 \
|
||||
--pp-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
|
||||
--mem-frac 0.95
|
||||
```
|
||||
|
||||
Once the server is running, send requests to the master node:
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
|
||||
```
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "e82af153da844ee6aed7a27a3187f2f4",
|
||||
"object": "chat.completion",
|
||||
"created": 1771216764,
|
||||
"model": "auto",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "The capital of France is **Paris**.\n\n**Additional details:**\n* It is the largest city in France.\n* It is located in the north-central part of the country along the Seine River.\n* Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).",
|
||||
"reasoning_content": null,
|
||||
"tool_calls": null
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "stop",
|
||||
"matched_stop": 156895
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 25,
|
||||
"total_tokens": 93,
|
||||
"completion_tokens": 68,
|
||||
"prompt_tokens_details": null,
|
||||
"reasoning_tokens": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For more API usage examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Tool Calling Example
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "inclusionAI/Ling-2.5-1T",
|
||||
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"description": "Search for information on the internet",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {"type": "string", "description": "The search query"}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto"
|
||||
}'
|
||||
```
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "b968e45c7d414f7482c8ffc0f9c6b688",
|
||||
"object": "chat.completion",
|
||||
"created": 1771216520,
|
||||
"model": "inclusionAI/Ling-2.5-1T",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": null,
|
||||
"reasoning_content": null,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_e75f711d8ad840ed9d382c9e",
|
||||
"index": 0,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"arguments": "{\"query\": \"latest news about AI\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "tool_calls",
|
||||
"matched_stop": null
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 173,
|
||||
"total_tokens": 196,
|
||||
"completion_tokens": 23,
|
||||
"prompt_tokens_details": null,
|
||||
"reasoning_tokens": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.960
|
||||
Invalid: 0.000
|
||||
Latency: 45.410 s
|
||||
Output throughput: 560.642 token/s
|
||||
```
|
||||
@@ -0,0 +1,223 @@
|
||||
---
|
||||
title: Ling-2.6
|
||||
metatags:
|
||||
description: "Deploy the Ling-2.6 family with SGLang - Ling-2.6-flash (104B total / 7.4B active BF16 MoE) and Ling-2.6-1T (~1T FP8 MoE) with hybrid linear attention and agentic tool calling."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
The **Ling-2.6** family from inclusionAI is the next iteration of the Ling instant-model series. Continuing the architectural direction set by Ling-2.5, Ling-2.6 doubles down on **inference efficiency**, **token efficiency**, and **agent performance** — staying competitive with frontier instant models while being faster, leaner, and better suited for production agent workloads.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid Linear Attention**: A `1:7 MLA + Lightning Linear` hybrid built on top of a highly sparse MoE backbone. Compared with same-class SOTA models, Ling-2.6-flash shows up to ~4× higher prefill and decode throughput in long-context scenarios; Ling-2.6-1T is shipped in FP8 so it fits a single GB300 node with `--tp 4`.
|
||||
- **Token Efficiency**: Trained with explicit token-efficiency objectives. On the full Artificial Analysis suite, Ling-2.6-flash uses only ~15M output tokens while remaining competitive — a meaningfully stronger intelligence-per-token profile than long-reasoning peers.
|
||||
- **Agentic Capabilities**: Refined for tool use, multi-step planning, and long-horizon execution. Reaches SOTA-class results on **BFCL-V4**, **TAU2-bench**, **SWE-bench Verified**, **Claw-Eval**, and **PinchBench**, and is validated against Claude Code, Kilo Code, Qwen Code, Hermes Agent, and OpenClaw.
|
||||
- **Long Context**: Native 128K, extendable to **256K (Ling-2.6-flash)** and **256K → 1M (Ling-2.6-1T via YaRN)**.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **BF16**: [inclusionAI/Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) — 104B total / 7.4B active
|
||||
- **FP8 (E4M3)**: [inclusionAI/Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) — ~1T total
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Ling-2.6-flash
|
||||
|
||||
Ling-2.6-flash is a 104B/7.4B-active MoE that runs comfortably on a single 4-GPU node. Use the selector below to generate the launch command for your hardware.
|
||||
|
||||
import { Ling26FlashDeployment } from '/src/snippets/autoregressive/ling-26-flash-deployment.jsx'
|
||||
|
||||
<Ling26FlashDeployment />
|
||||
|
||||
#### Configuration Tips
|
||||
|
||||
- `--trust-remote-code` is required (custom `BailingMoeV2_5ForCausalLM` modeling code).
|
||||
- `--tp-size 4` is the reference layout. On 4× H20-3e the model reaches ~340 tokens/s decode at TP=4, batch 32.
|
||||
- Native context is 128K. Enable YaRN (`--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 2.0, ...}}'`) to extend to 256K — the snippet does this for you.
|
||||
- `--tool-call-parser qwen25` matches the model's `<tool_call>...</tool_call>` schema.
|
||||
- The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6 is a controllable-reasoning model whose chat template defaults to `detailed thinking off`; the SGLang `qwen3` reasoning parser, in contrast, assumes default-thinking semantics and would mis-route normal output into `reasoning_content`. Only enable it if you specifically want `<think>...</think>` blocks split out — see [§4.3 Thinking Mode](#4-3-thinking-mode).
|
||||
- **MTP (multi-token prediction)** is supported. Add `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-radix-cache-strategy extra_buffer` to enable it — see the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash#run-inference) for the full example.
|
||||
|
||||
### 3.2 Ling-2.6-1T
|
||||
|
||||
Ling-2.6-1T ships in **FP8 (E4M3)**, so unlike Ling-2.5-1T it fits a **single GB300 node with `--tp 4`**. On smaller GPUs (H200/B200), a 2-node deployment with `--pp-size 2` is required.
|
||||
|
||||
import { Ling261TDeployment } from '/src/snippets/autoregressive/ling-26-1t-deployment.jsx'
|
||||
|
||||
<Ling261TDeployment />
|
||||
|
||||
#### Configuration Tips
|
||||
|
||||
- `--trust-remote-code` is required for the custom modeling code.
|
||||
- `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` significantly speeds up the multi-shard FP8 weight load (26 safetensors shards + an MTP layer).
|
||||
- Use `--tool-call-parser qwen` for tool calling.
|
||||
- The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6's chat template defaults to `detailed thinking off`, while SGLang's `qwen3` reasoning parser assumes default-thinking semantics — combining the two requires a per-request workaround for tool calls (see [§4.3 Thinking Mode](#4-3-thinking-mode)). Only enable `--reasoning-parser qwen3` if you specifically want `<think>...</think>` blocks split into `reasoning_content`.
|
||||
- For 2-node deployments, set `MASTER_IP`, `PORT`, and `DIST_PORT` consistently across both nodes.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
For example, launch a Ling-2.6-1T server on a single GB300 node:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ling-2.6-1T \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--tool-call-parser qwen \
|
||||
--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
|
||||
```
|
||||
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"id": "...",
|
||||
"object": "chat.completion",
|
||||
"model": "auto",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "The capital of France is **Paris**.",
|
||||
"reasoning_content": null,
|
||||
"tool_calls": null
|
||||
},
|
||||
"finish_reason": "stop"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 Tool Calling Example
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"description": "Search for information on the internet",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {"type": "string", "description": "The search query"}
|
||||
},
|
||||
"required": ["query"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto"
|
||||
}'
|
||||
```
|
||||
|
||||
Output:
|
||||
```json Config
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": null,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_...",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "search",
|
||||
"arguments": "{\"query\": \"latest news about AI\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"finish_reason": "tool_calls"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 Thinking Mode
|
||||
|
||||
Both Ling-2.6-flash and Ling-2.6-1T are **controllable-reasoning** models. Their chat template uses textual directives in the system message — `detailed thinking on` or `detailed thinking off` — to toggle thinking. The template **defaults to `detailed thinking off`** when neither phrase is present, and it does **not** read the Qwen3-style `enable_thinking` template variable.
|
||||
|
||||
#### Enabling thinking
|
||||
|
||||
Include `detailed thinking on` in the first system message:
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [
|
||||
{"role": "system", "content": "detailed thinking on"},
|
||||
{"role": "user", "content": "If a box has 12 red balls and 8 blue balls, then 5 red balls are removed, how many balls remain?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
If you already have a system prompt, append the directive on its own line:
|
||||
|
||||
```json
|
||||
{"role": "system", "content": "You are a helpful assistant.\ndetailed thinking on"}
|
||||
```
|
||||
|
||||
When thinking is on, the model emits `<think>...</think>` blocks before its final answer. To get those split into `message.reasoning_content` automatically, also launch the server with `--reasoning-parser qwen3`.
|
||||
|
||||
#### Caveat: `--reasoning-parser qwen3` + tool calling
|
||||
|
||||
The SGLang `qwen3` reasoning parser was written for Qwen3, where models are **default-thinking** and clients opt out via `chat_template_kwargs.enable_thinking=false`. Ling-2.6 is the opposite — default-non-thinking, with toggling done in the system message. As a result, when the server is launched with **both** `--tool-call-parser qwen` and `--reasoning-parser qwen3`, every tool-call request must include `chat_template_kwargs.enable_thinking=false`, otherwise the parser routes the `<tool_call>...</tool_call>` block into `reasoning_content` instead of `message.tool_calls`:
|
||||
|
||||
```bash Command
|
||||
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Search for the latest news about AI"}],
|
||||
"tools": [...],
|
||||
"tool_choice": "auto",
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
`enable_thinking` here is consumed by the SGLang reasoning parser, **not** by the chat template — Ling-2.6's template ignores it. For the simplest configuration, just omit `--reasoning-parser qwen3` and toggle thinking via the system message.
|
||||
|
||||
For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request).
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K (Ling-2.6-1T, GB300 × 4)
|
||||
|
||||
Reference run on a single GB300 node with `--tp 4`:
|
||||
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py
|
||||
```
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.9621 (1269 / 1319)
|
||||
```
|
||||
|
||||
For Ling-2.6-flash, see the official numbers on the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash) (BFCL-V4, TAU2-bench, SWE-bench Verified, Claw-Eval, PinchBench, Artificial Analysis).
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
title: Ring-2.5-1T
|
||||
metatags:
|
||||
description: "Deploy Ring-2.5-1T with SGLang - world's first open-source 1T parameter reasoning model with hybrid linear attention, deep reasoning, and agentic tool calling capabilities."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Model**: ~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention)
|
||||
- **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens
|
||||
- **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback
|
||||
- **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training
|
||||
- **Tool Calling**: Supports function calling with XML-style tool call format
|
||||
- **Context Length**: 128K -> 256K (YaRN)
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ring-2.5-1T runs on the standard SGLang Docker image:
|
||||
|
||||
```bash Command
|
||||
# NVIDIA (H200 / B200 / GB200 / GB300)
|
||||
docker pull lmsysorg/sglang:latest
|
||||
|
||||
# For MI300X/325X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x
|
||||
|
||||
# For MI355X
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
|
||||
|
||||
import { Ring251TDeployment } from '/src/snippets/autoregressive/ring-25-1t-deployment.jsx'
|
||||
|
||||
<Ring251TDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- The `--trust-remote-code` flag is required for this model due to custom modeling code.
|
||||
- The model uses FP8 quantization (compressed-tensors format).
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
Deploy Ring-2.5-1T with the following command (on H200, all features enabled):
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser deepseek-r1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ring-2.5-1T",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary>Output Example</summary>
|
||||
|
||||
````text Output
|
||||
We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution.
|
||||
|
||||
We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36.
|
||||
|
||||
But we need to present step by step. Also ensure it's clear.
|
||||
|
||||
We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36.
|
||||
|
||||
But any method is fine.
|
||||
|
||||
We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)."
|
||||
|
||||
We'll show:
|
||||
|
||||
15% = 15/100 = 0.15
|
||||
|
||||
Then 0.15 × 240 = 36.
|
||||
|
||||
Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36.
|
||||
|
||||
Finally, answer: 36.
|
||||
|
||||
We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36."
|
||||
|
||||
We'll present as a final answer: \boxed{36}.
|
||||
|
||||
However, we need to provide step-by-step solution as per instructions. We'll write a full explanation.
|
||||
|
||||
We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36.
|
||||
|
||||
Alr.
|
||||
|
||||
I think that's it.
|
||||
|
||||
|
||||
**Step 1:** Write 15% as a fraction or decimal.
|
||||
\[ 15\% = \frac{15}{100} = 0.15\]
|
||||
|
||||
**Step 2:** Multiply the number (240) by this fraction/decimal.
|
||||
\[ 240 \times 0.15 = 36\]
|
||||
|
||||
Alternatively, using the fraction:
|
||||
\[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\]
|
||||
|
||||
**Conclusion:** 15% of 240 is 36.
|
||||
|
||||
\[ \boxed{36} \]
|
||||
````
|
||||
|
||||
</details>
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
To enable tool calling, add `--tool-call-parser qwen` when launching the server.
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--tool-call-parser qwen \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="inclusionAI/Ring-2.5-1T",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
[ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### GSM8K
|
||||
|
||||
- Deployment Command
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.5-1T \
|
||||
--tp-size 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- Benchmark Command
|
||||
```bash Command
|
||||
python3 benchmark/gsm8k/bench_sglang.py --temperature 1.2 --top-p 0.8 --max-new-tokens 32768 --num-questions 200 --tokenizer-path inclusionAI/Ring-2.5-1T --enable-thinking
|
||||
```
|
||||
|
||||
- Test Result
|
||||
```text Output
|
||||
Accuracy: 0.955
|
||||
Invalid: 0.010
|
||||
Latency: 615.833 s
|
||||
Output throughput: 412.360 token/s
|
||||
```
|
||||
@@ -0,0 +1,374 @@
|
||||
---
|
||||
title: Ring-2.6-1T
|
||||
metatags:
|
||||
description: "Deploy Ring-2.6-1T with SGLang - a trillion-parameter InclusionAI reasoning model for agent workflows, high/xhigh reasoning effort, and tool use."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is InclusionAI's trillion-parameter flagship reasoning model for real-world complex task execution. It targets agent workflows, engineering development, scientific research analysis, enterprise automation, and other long-horizon settings where the model must plan, use tools, recover from intermediate errors, and keep context across multiple steps.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Trillion-Scale Reasoning Model**: `BailingMoeV2_5ForCausalLM` with a `bailing_hybrid` architecture, 80 hidden layers, 256 routed experts, 8 selected experts per token, and FP8 compressed-tensors weights.
|
||||
- **Agent Execution**: Designed for multi-step task decomposition, tool collaboration, context continuation, and long-horizon execution. The model card reports 87.60 on PinchBench, 63.82 on ClawEval, and 95.32 on Tau2-Bench Telecom for the `high` setting.
|
||||
- **Reasoning Effort**: The model card describes `high` and `xhigh` reasoning-effort modes. In SGLang's OpenAI-compatible chat API, use top-level `reasoning_effort: "high"` for production agent workflows. To request the model-card `xhigh` prompt path, pass it through `chat_template_kwargs.reasoning_effort`.
|
||||
- **Hybrid Attention**: Uses the Bailing hybrid stack with MLA plus Lightning linear attention kernels in SGLang.
|
||||
- **Context Length**: Native 128K in the released config. Configure YaRN separately if you need a 256K deployment.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
- **FP8 (E4M3 compressed-tensors)**: [inclusionAI/Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T)
|
||||
|
||||
**License:** MIT
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Ring-2.6-1T requires recent SGLang builds with Bailing hybrid model support. Start with the latest SGLang Docker image when validating this cookbook:
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
Use the selector below to generate a single-node command for the tested hardware targets.
|
||||
|
||||
import { Ring261TDeployment } from '/src/snippets/autoregressive/ring-26-1t-deployment.jsx'
|
||||
|
||||
<Ring261TDeployment />
|
||||
|
||||
### Configuration Tips
|
||||
|
||||
- `--trust-remote-code` is required for the model's custom Bailing hybrid implementation.
|
||||
- Use `--tp-size 4` on a single 4-GPU GB300 node.
|
||||
- Use `--tp-size 8` on a single 8-GPU B200 node.
|
||||
- Use `--tp-size 8` on a single 8-GPU H200 node.
|
||||
- Use `--mem-fraction-static 0.95` on GB300 x4. The model uses about 238.5GB/GPU after loading, so lower values can fail during KV-pool initialization.
|
||||
- Use `--mem-fraction-static 0.8` on B200 x8.
|
||||
- Use `--mem-fraction-static 0.95` on H200 x8.
|
||||
- `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` is recommended because the model has 175 large safetensors shards.
|
||||
- Keep `--tool-call-parser glm` enabled by default for OpenAI-compatible tool calls. Ring's template emits XML `<arg_key>/<arg_value>` tool calls, which the `qwen` parser does not convert into `message.tool_calls`.
|
||||
- Keep `--reasoning-parser deepseek-r1` enabled by default so `<think>...</think>` content is split into `message.reasoning_content`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For example, launch the server on a single 4-GPU GB300 node:
|
||||
|
||||
```bash Command
|
||||
export PORT=30000
|
||||
|
||||
sglang serve \
|
||||
--model-path inclusionAI/Ring-2.6-1T \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--mem-fraction-static 0.95 \
|
||||
--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}' \
|
||||
--tool-call-parser glm \
|
||||
--reasoning-parser deepseek-r1
|
||||
```
|
||||
|
||||
Send a basic chat request:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
||||
"max_tokens": 128
|
||||
}'
|
||||
```
|
||||
|
||||
### 4.2 Reasoning Effort
|
||||
|
||||
Ring-2.6-1T exposes two reasoning-effort levels in the model card: `high` and `xhigh`. In SGLang's OpenAI-compatible chat API, start with top-level `reasoning_effort: "high"` for agent and production workflows:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}],
|
||||
"reasoning_effort": "high",
|
||||
"max_tokens": 512
|
||||
}'
|
||||
```
|
||||
|
||||
For the model-card `xhigh` path, pass the template value explicitly:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}],
|
||||
"chat_template_kwargs": {"reasoning_effort": "xhigh"},
|
||||
"max_tokens": 512
|
||||
}'
|
||||
```
|
||||
|
||||
With the default deployment command, thinking text is separated into `message.reasoning_content` when the model emits `<think>...</think>` blocks.
|
||||
|
||||
### 4.3 Tool Calling Example
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "auto",
|
||||
"messages": [{"role": "user", "content": "What is the weather in Beijing?"}],
|
||||
"tools": [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}],
|
||||
"tool_choice": "auto",
|
||||
"max_tokens": 512
|
||||
}'
|
||||
```
|
||||
|
||||
For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request).
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
- Hardware: NVIDIA B200 GPU (8x), NVIDIA H200 GPU (8x), and NVIDIA GB300 GPU (4x)
|
||||
- Model: `inclusionAI/Ring-2.6-1T`
|
||||
- Docker image: `lmsysorg/sglang:latest`
|
||||
- SGLang version tested: `0.5.11`
|
||||
- Tensor Parallelism: 8 on B200 x8 and H200 x8, 4 on GB300 x4
|
||||
|
||||
Use the deployment command from [Section 3](#3-model-deployment), then confirm that the server is healthy before running benchmarks:
|
||||
|
||||
```bash Command
|
||||
curl -s http://localhost:${PORT}/health
|
||||
curl -s http://localhost:${PORT}/v1/models
|
||||
```
|
||||
|
||||
#### 5.1.1 Latency-Sensitive Benchmark
|
||||
|
||||
- Test Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port ${PORT} \
|
||||
--model inclusionAI/Ring-2.6-1T \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results (B200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 207.18
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.05
|
||||
Input token throughput (tok/s): 29.45
|
||||
Output token throughput (tok/s): 20.37
|
||||
Total token throughput (tok/s): 49.82
|
||||
Mean E2E Latency (ms): 20715.16
|
||||
Mean TTFT (ms): 187.86
|
||||
Mean TPOT (ms): 44.65
|
||||
Mean ITL (ms): 48.76
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (GB300 x4):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 62.21
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.16
|
||||
Input token throughput (tok/s): 98.07
|
||||
Output token throughput (tok/s): 67.83
|
||||
Total token throughput (tok/s): 165.91
|
||||
Mean E2E Latency (ms): 6218.57
|
||||
Mean TTFT (ms): 233.04
|
||||
Mean TPOT (ms): 14.21
|
||||
Mean ITL (ms): 14.22
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (H200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 57.10
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.18
|
||||
Input token throughput (tok/s): 106.85
|
||||
Output token throughput (tok/s): 73.91
|
||||
Total token throughput (tok/s): 180.76
|
||||
Mean E2E Latency (ms): 5707.72
|
||||
Mean TTFT (ms): 163.35
|
||||
Mean TPOT (ms): 13.17
|
||||
Mean ITL (ms): 13.17
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Throughput-Sensitive Benchmark
|
||||
|
||||
- Test Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port ${PORT} \
|
||||
--model inclusionAI/Ring-2.6-1T \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 100 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results (B200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 46.30
|
||||
Total input tokens: 50561
|
||||
Total generated tokens: 52444
|
||||
Request throughput (req/s): 2.16
|
||||
Input token throughput (tok/s): 1092.10
|
||||
Output token throughput (tok/s): 1132.77
|
||||
Total token throughput (tok/s): 2224.86
|
||||
Mean E2E Latency (ms): 27581.74
|
||||
Mean TTFT (ms): 1710.53
|
||||
Mean TPOT (ms): 51.27
|
||||
Mean ITL (ms): 49.43
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (GB300 x4):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 55.80
|
||||
Total input tokens: 50561
|
||||
Total generated tokens: 52444
|
||||
Request throughput (req/s): 1.79
|
||||
Input token throughput (tok/s): 906.10
|
||||
Output token throughput (tok/s): 939.84
|
||||
Total token throughput (tok/s): 1845.94
|
||||
Mean E2E Latency (ms): 33736.85
|
||||
Mean TTFT (ms): 2156.40
|
||||
Mean TPOT (ms): 63.09
|
||||
Mean ITL (ms): 60.33
|
||||
==================================================
|
||||
```
|
||||
|
||||
- Test Results (H200 x8):
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 100
|
||||
Benchmark duration (s): 44.51
|
||||
Total input tokens: 50561
|
||||
Total generated tokens: 52444
|
||||
Request throughput (req/s): 2.25
|
||||
Input token throughput (tok/s): 1135.88
|
||||
Output token throughput (tok/s): 1178.18
|
||||
Total token throughput (tok/s): 2314.06
|
||||
Mean E2E Latency (ms): 27177.14
|
||||
Mean TTFT (ms): 2173.08
|
||||
Mean TPOT (ms): 51.11
|
||||
Mean ITL (ms): 47.77
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.test.run_eval \
|
||||
--eval-name gsm8k \
|
||||
--host 127.0.0.1 \
|
||||
--port ${PORT} \
|
||||
--model auto \
|
||||
--num-examples 200 \
|
||||
--num-threads 64 \
|
||||
--max-tokens 2048 \
|
||||
--reasoning-effort high
|
||||
```
|
||||
|
||||
- Test Results (B200 x8):
|
||||
|
||||
```text Output
|
||||
Total latency: 100.378 s
|
||||
Score: 0.990
|
||||
Output throughput: 627.401 token/s
|
||||
```
|
||||
|
||||
- Test Results (GB300 x4):
|
||||
|
||||
```text Output
|
||||
Total latency: 98.386 s
|
||||
Score: 0.990
|
||||
Output throughput: 621.469 token/s
|
||||
```
|
||||
|
||||
- Test Results (H200 x8):
|
||||
|
||||
```text Output
|
||||
Total latency: 76.849 s
|
||||
Score: 0.990
|
||||
Output throughput: 793.125 token/s
|
||||
```
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: Intern-S1
|
||||
metatags:
|
||||
description: "Deploy Intern-S1 with SGLang - community contribution guide for InternLM's Intern-S1 model deployment."
|
||||
---
|
||||
|
||||
import { InternS1Deployment } from '/src/snippets/autoregressive/intern-s1-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
Intern-S1 includes the large **Intern-S1** MoE model and the smaller **Intern-S1-mini** dense model. The command generator below covers BF16 and FP8 serving on NVIDIA H100/H200/B200/B300 platforms.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install from source:
|
||||
|
||||
```bash Command
|
||||
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
<InternS1Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- FP8 checkpoints use the matching BF16 checkpoint as tokenizer path.
|
||||
- B300 deployments use `--attention-backend flashinfer`.
|
||||
- Enable `--reasoning-parser interns1` and `--tool-call-parser interns1` when your workload needs structured reasoning or tool-call parsing.
|
||||
@@ -0,0 +1,214 @@
|
||||
---
|
||||
title: Intern-S2-Preview
|
||||
metatags:
|
||||
description: "Deploy Intern-S2-Preview with SGLang"
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**Intern-S2-Preview** is an efficient 35B scientific multimodal foundation model. Beyond conventional parameter and data scaling, Intern-S2-Preview explores task scaling: increasing the difficulty, diversity, and coverage of scientific tasks to further unlock model capabilities.
|
||||
|
||||
**Resources:**
|
||||
|
||||
- HuggingFace: [internLM/Intern-S2-Preview](https://huggingface.co/internLM/Intern-S2-Preview)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
Install SGLang from source or use an NVIDIA Docker image:
|
||||
|
||||
```bash Command
|
||||
# Install from source
|
||||
uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
|
||||
|
||||
# Or use Docker for NVIDIA GPUs
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
For how to actually launch a docker image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the [command generator](#3-model-deployment) below produces):
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware and parser configuration.
|
||||
|
||||
import { InternS2PreviewDeployment } from "/src/snippets/autoregressive/intern-s2-preview-deployment.jsx";
|
||||
|
||||
<InternS2PreviewDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- Use `tp>=2` for the NVIDIA deployment commands.
|
||||
- Use `--reasoning-parser qwen3` to separate reasoning content from final content in streaming responses.
|
||||
- Use `--tool-call-parser qwen3_coder` when serving tool-calling workloads.
|
||||
- Add `--mamba-radix-cache-strategy extra_buffer` with `--speculative-algo 'NEXTN'` to enable MTP.
|
||||
- If weight loading is slow, add `--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 64}'`.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, see:
|
||||
|
||||
- [Basic API Usage](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Vision Input
|
||||
|
||||
Intern-S2-Preview supports image inputs. Here is an example with an image:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="internLM/Intern-S2-Preview",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Describe this image in detail.",
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
#### 4.2.2 Reasoning Parser
|
||||
|
||||
Enable streaming to read reasoning content separately from the final answer:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="internLM/Intern-S2-Preview",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, "reasoning_content") and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
if delta.content:
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
#### 4.2.3 Tool Calling
|
||||
|
||||
Serve with `--tool-call-parser qwen3_coder` enabled, then send OpenAI-compatible tool requests:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name",
|
||||
}
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="internLM/Intern-S2-Preview",
|
||||
messages=[{"role": "user", "content": "What is the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
max_tokens=1024,
|
||||
)
|
||||
|
||||
print(response.choices[0].message)
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
---
|
||||
title: InternVL3.5
|
||||
metatags:
|
||||
description: "Deploy InternVL3.5 vision-language model with SGLang - community contribution guide for OpenGVLab's multimodal model."
|
||||
---
|
||||
|
||||
|
||||
## 📝 Community Contribution Welcome
|
||||
|
||||
This guide is currently under development. We welcome community contributions!
|
||||
|
||||
If you have experience deploying **InternVL3.5** with SGLang, please help us complete this documentation.
|
||||
|
||||
## 🚀 How to Contribute
|
||||
|
||||
```shell Command
|
||||
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
|
||||
cd sglang-cookbook
|
||||
git checkout -b add-internvl3-5-guide
|
||||
# Edit this file and submit a PR
|
||||
```
|
||||
|
||||
## 📚 Reference
|
||||
|
||||
- [GLM-4.6V](../GLM/GLM-4.6V)
|
||||
|
||||
---
|
||||
|
||||
**Let's build this together!** 🌟
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
title: Jina-reranker-m0
|
||||
metatags:
|
||||
description: "Deploy Jina-reranker-m0 with SGLang - community contribution guide for Jina AI's reranker model deployment."
|
||||
---
|
||||
|
||||
## 📝 Community Contribution Welcome
|
||||
|
||||
This guide is currently under development. We welcome community contributions!
|
||||
|
||||
If you have experience deploying **Jina-reranker-m0** with SGLang, please help us complete this documentation.
|
||||
|
||||
## 🚀 How to Contribute
|
||||
|
||||
```shell Command
|
||||
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
|
||||
cd sglang-cookbook
|
||||
git checkout -b add-jina-reranker-m0-guide
|
||||
# Edit this file and submit a PR
|
||||
```
|
||||
|
||||
## 📚 Reference
|
||||
|
||||
- [DeepSeek-V3.2](../DeepSeek/DeepSeek-V3_2.md)
|
||||
|
||||
---
|
||||
|
||||
**Let's build this together!** 🌟
|
||||
@@ -0,0 +1,359 @@
|
||||
---
|
||||
title: LFM2.5
|
||||
description: "Deploy Liquid AI's LFM2.5 with SGLang — hybrid gated short conv + GQA models from 350M to the 8B-A1B MoE, plus LFM2.5-VL vision, with reasoning and Pythonic tool calling."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install sglang
|
||||
```
|
||||
|
||||
<Note>
|
||||
LFM2.5 support — the dense / MoE / VL model classes and the `lfm2` tool-call parser — ships on SGLang `main`. If your installed release predates it, install from source or use the Docker dev image.
|
||||
</Note>
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
LFM2.5 support ships in the pinned SGLang dev image:
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:dev-cu13
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:dev-cu13 \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Every LFM2.5 model runs on a **single GPU (TP=1)** — pick your hardware + model variant to generate the launch command. One recipe covers all operating points per variant; the commands differ only by the parsers a model needs and, on Blackwell, the attention backend. The `lfm2` tool-call parser and each reasoning model's `--reasoning-parser` are already part of the verified command.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/LiquidAI/lfm2.5.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/LiquidAI/lfm2.5-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> (top of the command box):</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>Python / Docker</strong> — bare <code>sglang serve …</code> for an existing SGLang env, or a <code>docker run … sglang serve …</code> wrap against the dev image from the <a href="#install">Install SGLang</a> panel above.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⧉ Copy</strong> — copies the current command (with whichever framing is active) to your clipboard.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>$ cURL</strong> — a sample request against <code>localhost:30000</code> to confirm the server is up.</li>
|
||||
<li style={{marginBottom: "0.2rem"}}><strong>⚙ Env</strong> — edits the placeholders (<code>HOST_IP</code>, <code>PORT</code>, <code>HF_TOKEN</code>) the command and cURL share. Persists in localStorage across cookbooks.</li>
|
||||
<li><strong>Verified / Not Verified</strong> badge — green when the <code>(hw, variant, quant, strategy, nodes)</code> combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.
|
||||
|
||||
For LFM2.5 the exposed knob is the **TP override** (every variant is verified at TP=1; TP=2 is available for experimentation on the larger checkpoints). The reasoning and tool-call parsers are not playground toggles here — they are variant-intrinsic and already baked into each verified command.
|
||||
|
||||
Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
<div style={{fontSize: "0.85em", lineHeight: "1.55", color: "#6b7280", margin: "0.5rem 0 1rem 0"}}>
|
||||
<p style={{margin: "0 0 0.3rem 0"}}><strong>Panel controls</strong> reuse <strong>Python / Docker</strong> · <strong>⧉ Copy</strong> · <strong>$ cURL</strong> · <strong>⚙ Env</strong> from the Deploy panel, plus one extra:</p>
|
||||
<ul style={{margin: 0, paddingLeft: "1.25rem"}}>
|
||||
<li><strong>Submit ↗</strong> — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says <strong>Not Verified</strong>; click it once you've actually run the command on your hardware and confirmed it works.</li>
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
LFM2.5 is [Liquid AI](https://www.liquid.ai/)'s family of hybrid models for on-device deployment, released under the [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE). It builds on the LFM2 architecture with extended pre-training — 10T → 28T tokens for the dense models, 12T → 38T for the 8B-A1B MoE — and large-scale reinforcement learning.
|
||||
|
||||
The backbone interleaves **gated short convolution blocks** with a small minority of **grouped query attention (GQA) blocks**. Each convolution block applies input-dependent multiplicative gating around a depthwise short convolution, giving fast local mixing at low compute and memory cost. The GQA blocks handle global context and long-range retrieval.
|
||||
|
||||
This minimal hybrid layout was selected by a hardware-in-the-loop architecture search under edge latency and memory budgets. On CPUs it delivers up to 2× faster prefill and decode than similarly sized models (see the [LFM2 Technical Report](https://arxiv.org/abs/2511.23404)).
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Hybrid gated short conv + GQA layout**: the 1.2B / 350M dense models are 16 layers (10 conv + 6 GQA); the 8B-A1B MoE is 24 layers (18 conv + 6 GQA). With only 6 attention layers per model, the KV cache stays small even at long context.
|
||||
- **Block details**: depthwise convolutions with kernel size 3; GQA with 8 KV groups and head size 64, plus RoPE and QK-Norm; pre-norm RMSNorm and SwiGLU MLPs throughout.
|
||||
- **Sparse MoE (8B-A1B)**: 8.3B total / 1.5B active parameters. Every layer except the first two replaces its dense MLP with a 32-expert MoE block; each token is routed to the top-4 SwiGLU experts by a normalized sigmoid router with adaptive bias load balancing.
|
||||
- **New in 2.5 (8B-A1B)**: the blocks are unchanged from LFM2-8B-A1B, but the context window grows from 32K to 128K (a RoPE base-θ increase plus long-context midtraining) and the vocabulary doubles from 65,536 to 128,000 tokens for more efficient non-Latin tokenization.
|
||||
- **Pythonic tool calling**: function calls are emitted as a Python list between `<|tool_call_start|>` and `<|tool_call_end|>` tokens. The `lfm2` tool-call parser surfaces these as standard `message.tool_calls`.
|
||||
- **Reasoning variants**: the 8B-A1B and 1.2B-Thinking checkpoints are reasoning-only models that always emit an explicit `<think>...</think>` chain-of-thought before the answer. The MoE's 1.5B active parameters keep those reasoning tokens cheap.
|
||||
- **Multilingual**: every model except the JP checkpoints covers at least English, Arabic, Chinese, French, German, Japanese, Korean, and Spanish (some variants add more). The dedicated JP chat checkpoints focus on Japanese (Japanese + English only).
|
||||
- **Vision**: LFM2.5-VL-1.6B pairs the 1.2B language backbone with a SigLIP2 So400M NaFlex encoder for OCR, document understanding, and multilingual vision. LFM2.5-VL-450M pairs the 350M backbone with a SigLIP2 Base-86M encoder for captioning and object detection at edge sizes; bounding-box grounding and function calling are new in the 2.5 release.
|
||||
|
||||
**Available Models:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "26%"}} />
|
||||
<col style={{width: "18%"}} />
|
||||
<col style={{width: "12%"}} />
|
||||
<col style={{width: "44%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Parameters</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Context</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Role</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-8B-A1B">LFM2.5-8B-A1B</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8.3B total / 1.5B active (MoE)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>128K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning-tuned, agentic / tool use</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct">LFM2.5-1.2B-Instruct</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>General instruct, RAG, data extraction</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking">LFM2.5-1.2B-Thinking</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning (always-on chain-of-thought)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-350M">LFM2.5-350M</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>350M (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Compact instruct, structured output</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-230M">LFM2.5-230M</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>230M (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Most compact; data extraction, structured output</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606">LFM2.5-1.2B-JP-202606</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Japanese chat (latest)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP">LFM2.5-1.2B-JP</a></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Japanese chat (original)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B">LFM2.5-VL-1.6B</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.2B LM + SigLIP2 400M</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Vision-language (OCR, docs, multi-image)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><a href="https://huggingface.co/LiquidAI/LFM2.5-VL-450M">LFM2.5-VL-450M</a></strong></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>350M LM + SigLIP2 86M</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Compact vision-language (captioning, object detection)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base">LFM2.5-1.2B-Base</a></td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.17B (dense)</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>32K</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Pre-trained base (no post-training)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The Deploy panel above covers the eight serving variants; **LFM2.5-1.2B-JP** (original — launch without `--tool-call-parser`) and the **Base** repos (pre-trained only, no post-training — see [§3.5](#3-5-base-checkpoints)) launch the same way with the model path swapped.
|
||||
|
||||
**Choosing a variant:**
|
||||
|
||||
- **8B-A1B** — flagship for agentic and tool-calling workloads; the only 128K-context option.
|
||||
- **1.2B-Thinking** — reasoning-heavy tasks: math, tool use, programming.
|
||||
- **1.2B-Instruct** — the recommended pick for chat and creative writing.
|
||||
- **350M** — tool use, data extraction, and structured output; not recommended for math, code, or creative writing.
|
||||
- **230M** — the most compact checkpoint; same use as the 350M, not for math, code, or creative writing.
|
||||
|
||||
**License:** [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE).
|
||||
|
||||
**Resources:** [LFM2.5 announcement](https://www.liquid.ai/blog/introducing-lfm2-5-the-next-generation-of-on-device-ai), [LFM2.5-8B-A1B blog](https://www.liquid.ai/blog/lfm2-5-8b-a1b), [LFM docs](https://docs.liquid.ai/lfm/getting-started/welcome), [LFM2 Technical Report (arXiv:2511.23404)](https://arxiv.org/abs/2511.23404).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Reasoning parser**: LFM2.5 reasoning models wrap their chain-of-thought in `<think>...</think>` tags. The command generator passes `--reasoning-parser qwen3` for **8B-A1B** (it emits an explicit opening `<think>`) and `--reasoning-parser qwen3-thinking` for **1.2B-Thinking** (always-on reasoning). This splits the thinking process into `reasoning_content`; without it the chain-of-thought stays inline in `content`.
|
||||
- **Tool calling**: `--tool-call-parser lfm2` surfaces LFM2.5's Pythonic `<|tool_call_start|>[...]<|tool_call_end|>` calls as standard `message.tool_calls`. The original **1.2B-JP** does not expose tool calling; **Base** has no post-training (see [§3.5](#3-5-base-checkpoints)).
|
||||
- **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for the dense text models. The **8B-A1B** uses a mamba-style state cache that runs on a page-size-1 backend, so the generator picks `--attention-backend flashinfer` for it. The **VL** language model also uses that state cache and offers two backends: `--attention-backend flashinfer` (keeps prefix/radix caching — what the generator emits), or `--attention-backend trtllm_mha --disable-radix-cache` to run the language model on Blackwell `trtllm_mha` attention (`--disable-radix-cache` lifts the page-size-1 requirement, at the cost of prefix caching). Pair either with `--mm-attention-backend fa4` for the vision tower.
|
||||
- **VL vision tower (`--mm-attention-backend`)**: on sm100 the `trtllm_mha` default is fastest for text but applies *causal* attention to image tokens. For the VL model, pass `--mm-attention-backend fa4` on B200/B300 (or `fa3` on H100/H200) to restore bidirectional image-token attention and full vision quality.
|
||||
- **VL multimodal feature transport**: the generator launches the VL models with `SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1`. The first moves the processor→scheduler image-feature handoff onto CUDA IPC instead of serializing tensors between processes; the second ships the pool handle so the scheduler opens it once and caches it, instead of opening a per-item handle on every request. On the image serving workload (1 image @ 720p, measured on VL-1.6B on H100 and B200) this pair is worth roughly 30–50% higher image throughput and 30–40% lower image TTFT vs running without them (measured on VL-1.6B, H100 and B200); decode speed (TPOT) is unaffected.
|
||||
- **VL-450M memory headroom (`--mem-fraction-static 0.8`)**: with the default memory fraction, the 450M's small weights make SGLang size its static KV/mamba pools to nearly the whole GPU, leaving no headroom for image-feature tensors — under sustained concurrent image load the scheduler can crash with a CUDA OOM in the radix-cache free path. The generator caps `--mem-fraction-static 0.8` for VL-450M; the pool is still far larger than this model ever needs.
|
||||
- **Mamba scheduling**: LFM2.5 runs on the default `no_buffer` mamba scheduler strategy — no `--mamba-radix-cache-strategy` flag is needed. The `extra_buffer` strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use `mamba_chunk_size=1`.
|
||||
- **Hardware requirements**: all LFM2.5 models run on a single GPU (TP=1) on either Hopper or Blackwell. The 1.2B / 350M dense models fit in a few GB; the 8B-A1B MoE needs roughly 16 GB for bf16 weights plus KV cache. Multi-GPU tensor parallelism is not required for any variant.
|
||||
|
||||
**Recommended sampling parameters** — pass these explicitly on every request. Some LFM2.5 checkpoints do not ship sampling defaults in `generation_config.json`, so the server will not apply them for you. `top_k`, `min_p`, and `repetition_penalty` are not standard OpenAI `chat.completions` fields — pass them through **`extra_body`** and SGLang forwards them to its sampler. Do not set `max_tokens` unless you intend to cap output, as it can truncate a response (or a reasoning model's chain-of-thought) mid-stream.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>temperature</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>extra_body (sampler)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-8B-A1B</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.2</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 80, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-Instruct</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-Thinking</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.05</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-350M</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-230M</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-JP-202606</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"top_k": 50, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-JP</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.3</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-VL-1.6B (text)</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-VL-450M (text)</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.1</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
<tr><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>LFM2.5-1.2B-Base</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0.3</td><td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>{`{"min_p": 0.15, "repetition_penalty": 1.05}`}</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Basic Usage
|
||||
|
||||
A single client with the recommended sampling presets applied per model (the examples in the following sections reuse this `chat` helper):
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
# Non-OpenAI fields (top_k / min_p / repetition_penalty) ride in extra_body.
|
||||
SAMPLING = {
|
||||
"LiquidAI/LFM2.5-8B-A1B": dict(temperature=0.2, extra_body={"top_k": 80, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-1.2B-Instruct": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-1.2B-Thinking": dict(temperature=0.05, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-350M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-230M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-1.2B-JP-202606": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-VL-1.6B": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
|
||||
"LiquidAI/LFM2.5-VL-450M": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
|
||||
}
|
||||
|
||||
def chat(model, messages, **overrides):
|
||||
cfg = SAMPLING[model]
|
||||
body = cfg["extra_body"] | overrides.pop("extra_body", {})
|
||||
return client.chat.completions.create(
|
||||
model=model, messages=messages,
|
||||
temperature=cfg["temperature"], extra_body=body, **overrides,
|
||||
)
|
||||
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
[{"role": "user", "content": "What is C. elegans? Answer in one sentence."}],
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 3.2 Reasoning
|
||||
|
||||
The 8B-A1B and 1.2B-Thinking checkpoints emit chain-of-thought as a built-in behavior. The Deploy panel launches them with the matching `--reasoning-parser`, which separates the thinking process into `reasoning_content`:
|
||||
|
||||
```python Example
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-8B-A1B",
|
||||
[{"role": "user", "content": "If a train travels 60 km/h for 2.5 hours, how far does it go?"}],
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
print("Reasoning:", msg.reasoning_content)
|
||||
print("Answer:", msg.content)
|
||||
```
|
||||
|
||||
### 3.3 Tool Calling
|
||||
|
||||
LFM2.5 writes Pythonic tool calls. With `--tool-call-parser lfm2` (already part of the launch command) they are surfaced as standard `message.tool_calls`:
|
||||
|
||||
```python Example
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=[{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"location": {"type": "string"}},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}],
|
||||
)
|
||||
for call in resp.choices[0].message.tool_calls or []:
|
||||
print(call.function.name, call.function.arguments)
|
||||
```
|
||||
|
||||
Tool calling is supported on 8B-A1B, 1.2B-Thinking, 1.2B-Instruct, 350M, 230M, 1.2B-JP-202606, VL-1.6B, and VL-450M. For the **VL** models it is text-turn-only — do not combine an image and tools in the same turn.
|
||||
|
||||
### 3.4 Vision Input
|
||||
|
||||
The VL models (VL-1.6B and VL-450M) accept images via standard OpenAI multimodal content blocks. Base64 data URIs (`data:image/jpeg;base64,...`) work in place of a URL:
|
||||
|
||||
```python Example
|
||||
resp = chat(
|
||||
"LiquidAI/LFM2.5-VL-1.6B",
|
||||
[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {
|
||||
"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
|
||||
{"type": "text", "text": "What is in this image?"},
|
||||
],
|
||||
}],
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### 3.5 Base Checkpoints
|
||||
|
||||
Each size ships a pre-trained Base repo — [LFM2.5-230M-Base](https://huggingface.co/LiquidAI/LFM2.5-230M-Base), [LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base), [LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base), and [LFM2.5-8B-A1B-Base](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-Base) — intended for fine-tuning and continued pre-training.
|
||||
|
||||
The repos ship a ChatML-style chat template, so `chat.completions` requests format normally. The checkpoints have no post-training, though — don't expect instruction following. For raw text continuation:
|
||||
|
||||
```python Example
|
||||
comp = client.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-Base",
|
||||
prompt="The capital of France is",
|
||||
temperature=0.3,
|
||||
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(comp.choices[0].text)
|
||||
```
|
||||
@@ -0,0 +1,650 @@
|
||||
---
|
||||
title: Llama-3.1
|
||||
metatags:
|
||||
description: "Deploy Llama 3.1 (8B/70B/405B) with SGLang - 128K context, tool use, multilingual support, and speculative decoding optimization."
|
||||
---
|
||||
## 1. Model Introduction
|
||||
|
||||
Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time.
|
||||
|
||||
These models bring open intelligence to all, with several new features and improvements:
|
||||
|
||||
- **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities.
|
||||
- **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning.
|
||||
- **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools.
|
||||
- **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai.
|
||||
|
||||
For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models.
|
||||
|
||||
import { Llama31Deployment } from "/src/snippets/autoregressive/llama31-deployment.jsx";
|
||||
|
||||
<Llama31Deployment />
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Speculative Decoding (NVIDIA GPUs):**
|
||||
|
||||
- Using Speculative Decoding for latency-sensitive scenarios:
|
||||
- `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
|
||||
- `--speculative-num-steps 3`: Number of speculative verification rounds
|
||||
- `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
|
||||
- `--speculative-num-draft-tokens 4`: Number of draft tokens per step
|
||||
- `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B).
|
||||
|
||||
**AMD GPU Deployment:**
|
||||
|
||||
- **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB)
|
||||
- **Verified TP Configurations**:
|
||||
- MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1)
|
||||
- MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1)
|
||||
- **FP8 Model Variants**:
|
||||
- 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8`
|
||||
- 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV`
|
||||
- **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models
|
||||
|
||||
**Xeon CPU Deployment:**
|
||||
|
||||
- Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
SGLang exposes an OpenAI-compatible endpoint. First, start the server
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="Meta-Llama/Llama-3.1-405B-Instruct",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful coding assistant."},
|
||||
{"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
|
||||
],
|
||||
temperature=0.2,
|
||||
max_tokens=512,
|
||||
)
|
||||
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
````text Output
|
||||
**Exponential Backoff Retry Function in Python**
|
||||
=====================================================
|
||||
|
||||
Below is a Python function that uses the `requests` library to retry a request with exponential backoff.
|
||||
|
||||
```python
|
||||
import requests
|
||||
import time
|
||||
import random
|
||||
|
||||
def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60):
|
||||
"""
|
||||
Retry a request with exponential backoff.
|
||||
|
||||
Args:
|
||||
url (str): The URL to make the request to.
|
||||
method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.).
|
||||
retries (int): The number of retries to attempt. Defaults to 3.
|
||||
backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1.
|
||||
max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60.
|
||||
|
||||
Returns:
|
||||
The response object from the successful request.
|
||||
"""
|
||||
|
||||
delay = 1
|
||||
for attempt in range(retries + 1):
|
||||
try:
|
||||
response = requests.request(method, url)
|
||||
response.raise_for_status() # Raise an exception for HTTP errors
|
||||
return response
|
||||
except requests.RequestException as e:
|
||||
if attempt < retries:
|
||||
# Calculate the delay for this retry
|
||||
delay = min(delay * backoff_factor, max_delay)
|
||||
# Add a random jitter to the delay to prevent thundering herd problem
|
||||
delay += random.uniform(0, delay * 0.1)
|
||||
# Wait for the calculated delay before retrying
|
||||
time.sleep(delay)
|
||||
else:
|
||||
# If all retries have failed, raise the exception
|
||||
raise e
|
||||
...
|
||||
````
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Tool Calling
|
||||
|
||||
Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-405B-Instruct \
|
||||
--tool-call-parser llama3 \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
**Python Example**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the weather in a given location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string",
|
||||
"description": "The city to find the weather for, e.g. 'San Francisco'",
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"description": "The unit to fetch the temperature in",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
},
|
||||
},
|
||||
"required": ["city", "unit"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.1-405B-Instruct",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "What's the weather like in Boston today?",
|
||||
}
|
||||
],
|
||||
temperature=0.7,
|
||||
stream=True,
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
|
||||
arguments = []
|
||||
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#openai-compatible-api)
|
||||
|
||||
**Output Example**
|
||||
|
||||
```text Output
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"city": "Boston", "unit": "fahrenheit"}
|
||||
```
|
||||
|
||||
**Handling Tool Call Results**
|
||||
After getting the tool call, you can execute the function:
|
||||
|
||||
```python Example
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather like in Boston today?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Boston", "unit": "fahrenheit"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Boston", "fahrenheit")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="Meta-Llama/Llama-3.1-405B-Instruct",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside"
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
### 5.1 Speed Benchmark
|
||||
|
||||
**Test Environment:**
|
||||
|
||||
- Hardware: NVIDIA A100 GPU (8x)
|
||||
- Model: Meta-Llama/Llama-3.1-70B
|
||||
- Tensor Parallelism: 8
|
||||
- sglang version: 0.5.6
|
||||
|
||||
We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.
|
||||
|
||||
#### 5.1.1 Standard Scenario Benchmark
|
||||
|
||||
- Model Deployment Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
##### 5.1.1.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 79.81
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4208
|
||||
Request throughput (req/s): 0.13
|
||||
Input token throughput (tok/s): 76.44
|
||||
Output token throughput (tok/s): 52.88
|
||||
Peak output token throughput (tok/s): 54.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 129.32
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7977.81
|
||||
Median E2E Latency (ms): 6373.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 131.61
|
||||
Median TTFT (ms): 131.77
|
||||
P99 TTFT (ms): 163.88
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.63
|
||||
Median TPOT (ms): 18.63
|
||||
P99 TPOT (ms): 18.65
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.64
|
||||
Median ITL (ms): 18.64
|
||||
P95 ITL (ms): 18.69
|
||||
P99 ITL (ms): 18.74
|
||||
Max ITL (ms): 21.95
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.2 Medium Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 79.47
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 38450
|
||||
Request throughput (req/s): 1.01
|
||||
Input token throughput (tok/s): 499.17
|
||||
Output token throughput (tok/s): 513.48
|
||||
Peak output token throughput (tok/s): 674.00
|
||||
Peak concurrent requests: 20
|
||||
Total token throughput (tok/s): 1012.65
|
||||
Concurrency: 13.47
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13376.67
|
||||
Median E2E Latency (ms): 14130.48
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 264.84
|
||||
Median TTFT (ms): 147.02
|
||||
P99 TTFT (ms): 791.93
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 26.09
|
||||
Median TPOT (ms): 26.08
|
||||
P99 TPOT (ms): 34.65
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 25.76
|
||||
Median ITL (ms): 23.95
|
||||
P95 ITL (ms): 24.72
|
||||
P99 ITL (ms): 98.32
|
||||
Max ITL (ms): 478.92
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.1.3 High Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 131.64
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 243641
|
||||
Request throughput (req/s): 3.80
|
||||
Input token throughput (tok/s): 1897.87
|
||||
Output token throughput (tok/s): 1919.38
|
||||
Peak output token throughput (tok/s): 3100.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3817.25
|
||||
Concurrency: 89.70
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 23616.71
|
||||
Median E2E Latency (ms): 22770.44
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 245.98
|
||||
Median TTFT (ms): 184.22
|
||||
P99 TTFT (ms): 1251.67
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 47.19
|
||||
Median TPOT (ms): 48.67
|
||||
P99 TPOT (ms): 56.37
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 46.34
|
||||
Median ITL (ms): 33.46
|
||||
P95 ITL (ms): 108.61
|
||||
P99 ITL (ms): 166.11
|
||||
Max ITL (ms): 1107.09
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.1.2 Summarization Scenario Benchmark
|
||||
|
||||
##### 5.1.2.1 Low Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B\
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 83.25
|
||||
Total input tokens: 41941
|
||||
Total input text tokens: 41941
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.12
|
||||
Input token throughput (tok/s): 503.77
|
||||
Output token throughput (tok/s): 50.69
|
||||
Peak output token throughput (tok/s): 54.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 554.46
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 8322.45
|
||||
Median E2E Latency (ms): 6873.36
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 395.25
|
||||
Median TTFT (ms): 318.02
|
||||
P99 TTFT (ms): 850.80
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 18.80
|
||||
Median TPOT (ms): 18.81
|
||||
P99 TPOT (ms): 19.03
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 18.83
|
||||
Median ITL (ms): 18.81
|
||||
P95 ITL (ms): 19.06
|
||||
P99 ITL (ms): 19.08
|
||||
Max ITL (ms): 23.08
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.2 Medium Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 107.12
|
||||
Total input tokens: 300020
|
||||
Total input text tokens: 300020
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 41669
|
||||
Total generated tokens (retokenized): 41603
|
||||
Request throughput (req/s): 0.75
|
||||
Input token throughput (tok/s): 2800.81
|
||||
Output token throughput (tok/s): 389.00
|
||||
Peak output token throughput (tok/s): 624.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 3189.81
|
||||
Concurrency: 14.18
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18988.30
|
||||
Median E2E Latency (ms): 20290.66
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 603.42
|
||||
Median TTFT (ms): 531.82
|
||||
P99 TTFT (ms): 2607.95
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 36.94
|
||||
Median TPOT (ms): 36.73
|
||||
P99 TPOT (ms): 79.19
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 35.36
|
||||
Median ITL (ms): 25.72
|
||||
P95 ITL (ms): 27.07
|
||||
P99 ITL (ms): 439.74
|
||||
Max ITL (ms): 2529.51
|
||||
==================================================
|
||||
```
|
||||
|
||||
##### 5.1.2.3 High Concurrency
|
||||
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--backend sglang \
|
||||
--model-path Meta-Llama/Llama-3.1-70B \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 320 \
|
||||
--max-concurrency 64
|
||||
```
|
||||
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 64
|
||||
Successful requests: 320
|
||||
Benchmark duration (s): 215.66
|
||||
Total input tokens: 1273893
|
||||
Total input text tokens: 1273893
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 170000
|
||||
Total generated tokens (retokenized): 169035
|
||||
Request throughput (req/s): 1.48
|
||||
Input token throughput (tok/s): 5906.92
|
||||
Output token throughput (tok/s): 788.27
|
||||
Peak output token throughput (tok/s): 1920.00
|
||||
Peak concurrent requests: 69
|
||||
Total token throughput (tok/s): 6695.19
|
||||
Concurrency: 60.01
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 40443.85
|
||||
Median E2E Latency (ms): 39813.12
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 633.32
|
||||
Median TTFT (ms): 616.38
|
||||
P99 TTFT (ms): 1912.97
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 74.95
|
||||
Median TPOT (ms): 82.85
|
||||
P99 TPOT (ms): 118.46
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 75.08
|
||||
Median ITL (ms): 34.12
|
||||
P95 ITL (ms): 261.18
|
||||
P99 ITL (ms): 828.12
|
||||
Max ITL (ms): 1970.03
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
- **Results**:
|
||||
|
||||
```text Output
|
||||
Accuracy: 0.830
|
||||
Invalid: 0.000
|
||||
Latency: 11.794 s
|
||||
Output throughput: 1406.961 token/s
|
||||
```
|
||||
@@ -0,0 +1,235 @@
|
||||
---
|
||||
title: Llama-3.3-70B
|
||||
metatags:
|
||||
description: "Deploy Llama-3.3-70B-Instruct with SGLang on AMD GPUs - 128K context, enhanced reasoning, tool calling, and multilingual support."
|
||||
---
|
||||
## 1. Model Introduction
|
||||
|
||||
[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment.
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1
|
||||
- **Tool Calling**: Native support for function calling and tool use scenarios
|
||||
- **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai)
|
||||
- **Extended Context**: 128K token context window for processing long documents and complex tasks
|
||||
- **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X
|
||||
|
||||
**License:**
|
||||
Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details.
|
||||
|
||||
For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs.
|
||||
|
||||
### 3.1 Interactive Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup.
|
||||
|
||||
import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx";
|
||||
|
||||
<Llama33Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**AMD GPU Deployment:**
|
||||
|
||||
- All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants
|
||||
- **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV`
|
||||
- **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support
|
||||
- **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput
|
||||
|
||||
**Xeon CPU Deployment:**
|
||||
|
||||
Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Tool Calling
|
||||
|
||||
Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment:
|
||||
|
||||
```shell Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.3-70B-Instruct \
|
||||
--tool-call-parser llama3 \
|
||||
--tp 1 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Tokyo?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
# Check for tool calls
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
tool_call = message.tool_calls[0]
|
||||
print(f"Function: {tool_call.function.name}")
|
||||
print(f"Arguments: {tool_call.function.arguments}")
|
||||
```
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After executing the function, send the result back
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Build conversation with tool result
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Tokyo?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Tokyo", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Tokyo", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=messages,
|
||||
temperature=0.7
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
# Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!"
|
||||
```
|
||||
|
||||
#### 4.2.2 Long Context Processing
|
||||
|
||||
Leverage the 128K context window for processing long documents:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Example with long document
|
||||
long_document = "..." * 10000 # Your long document here
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": f"Summarize this document:\n\n{long_document}"}
|
||||
],
|
||||
temperature=0.7,
|
||||
max_tokens=1000
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## 5. Benchmarking
|
||||
|
||||
Use the SGLang benchmarking suite to test model performance with different workload patterns:
|
||||
|
||||
### 5.1 Basic Benchmark Command
|
||||
|
||||
```bash Command
|
||||
python -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 1000 \
|
||||
--random-input 1024 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 16
|
||||
```
|
||||
|
||||
### 5.2 Adjusting Benchmark Parameters
|
||||
|
||||
**Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns:
|
||||
|
||||
- Short conversations: `--random-input 1024 --random-output 1024`
|
||||
- Long outputs: `--random-input 1024 --random-output 8192`
|
||||
- Long inputs: `--random-input 8192 --random-output 1024`
|
||||
|
||||
**Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios:
|
||||
|
||||
- Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100`
|
||||
- Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000`
|
||||
- High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000`
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [Meta Llama Models Repository](https://github.com/meta-llama/llama-models)
|
||||
- [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
|
||||
- [SGLang Documentation](/)
|
||||
- [AMD ROCm Documentation](https://rocm.docs.amd.com/)
|
||||
@@ -0,0 +1,572 @@
|
||||
---
|
||||
title: Llama 4
|
||||
metatags:
|
||||
description: "Deploy Llama 4 Scout and Maverick with SGLang - Meta's latest generation open-source LLMs with industry-leading performance."
|
||||
---
|
||||
|
||||
import { Llama4ScoutDeployment } from '/src/snippets/autoregressive/llama4-scout-deployment.jsx';
|
||||
import { Llama4MaverickDeployment } from '/src/snippets/autoregressive/llama4-maverick-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance.
|
||||
|
||||
SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).
|
||||
|
||||
Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
The highly capable Llama 4 Maverick with 17B active parameters out of ~400B total, with 128 experts.
|
||||
The efficient Llama 4 Scout also has 17B active parameters out of ~109B total, using just 16 experts.
|
||||
Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi).
|
||||
|
||||
For more details, please refer to the official llama4 Repository:https://www.llama.com/models/llama-4/
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<Llama4ScoutDeployment />
|
||||
|
||||
<Llama4MaverickDeployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
- **OOM Mitigation:** Reduce `--context-length` to avoid GPU out-of-memory. Recommended: Scout up to 1M on 8×H100, up to 2.5M on 8×H200; Maverick doesn't need context-length set on 8×H200. With hybrid KV cache enabled, Scout can reach 5M on 8×H100 and 10M on 8×H200.
|
||||
- **Attention Backend Auto-Selection:** SGLang automatically picks the optimal backend. Manual override with `--attention-backend`:
|
||||
- Blackwell (B200/GB200): `trtllm_mha`
|
||||
- Hopper (H100/H200): `fa3`
|
||||
- AMD GPUs: `aiter`
|
||||
- Intel XPU: `intel_xpu`
|
||||
- Other: `triton`
|
||||
- **Chat Template:** Add `--chat-template llama-4` for chat completion tasks.
|
||||
- **Multi-Modal:** Add `--enable-multimodal` to enable image input support.
|
||||
- **Hybrid KV Cache:** Set `--swa-full-tokens-ratio` to control the ratio of SWA (local attention) KV tokens to full-attention KV tokens (default: 0.8, range: 0–1).
|
||||
- **EAGLE Speculative Decoding:** Supported for Llama 4 Scout and Maverick via EAGLE3. Enable with the interactive command generator above.
|
||||
- **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
- [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Launch the docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Llama4 \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 4.2.2 Launch the server
|
||||
|
||||
### Llama-4-Scout
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### Llama-4-Maverick
|
||||
8-GPU deployment command:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--tp 8 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
#### 4.2.3 EAGLE Speculative Decoding
|
||||
|
||||
SGLang supports Llama 4 Maverick (400B) with [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding). Enable with the EAGLE3 algorithm and the SGLang EAGLE3 draft model:
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1 \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--trust-remote-code \
|
||||
--tp 8
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
### 5.1 Speed Benchmark (Scout)
|
||||
Test Environment:
|
||||
|
||||
Hardware: AMD MI300x GPU
|
||||
|
||||
Model: Llama-4-Scout
|
||||
|
||||
Tensor Parallelism: 8
|
||||
|
||||
sglang version: 0.5.9
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 5.1.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 74.62
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4211
|
||||
Request throughput (req/s): 0.14
|
||||
Input token throughput (tok/s): 82.88
|
||||
Output token throughput (tok/s): 57.42
|
||||
Peak output token throughput (tok/s): 146.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 140.20
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 7459.48
|
||||
Median E2E Latency (ms): 4489.77
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4246.98
|
||||
Median TTFT (ms): 68.57
|
||||
P99 TTFT (ms): 48091.05
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 7.49
|
||||
Median TPOT (ms): 7.40
|
||||
P99 TPOT (ms): 7.40
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 7.49
|
||||
Median ITL (ms): 7.49
|
||||
P95 ITL (ms): 7.47
|
||||
P99 ITL (ms): 7.52
|
||||
Max ITL (ms): 10.44
|
||||
==================================================
|
||||
```
|
||||
### 5.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 45.41
|
||||
Total input tokens: 49668
|
||||
Total input text tokens: 49668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40516
|
||||
Request throughput (req/s): 2.26
|
||||
Input token throughput (tok/s): 1120.46
|
||||
Output token throughput (tok/s): 1152.47
|
||||
Peak output token throughput (tok/s): 1520.00
|
||||
Peak concurrent requests: 21
|
||||
Total token throughput (tok/s): 2272.84
|
||||
Concurrency: 14.76
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6089.22
|
||||
Median E2E Latency (ms): 6568.80
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 124.44
|
||||
Median TTFT (ms): 87.42
|
||||
P99 TTFT (ms): 268.72
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 11.88
|
||||
Median TPOT (ms): 12.00
|
||||
P99 TPOT (ms): 15.49
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 11.72
|
||||
Median ITL (ms): 10.54
|
||||
P95 ITL (ms): 11.22
|
||||
P99 ITL (ms): 67.88
|
||||
Max ITL (ms): 74.05
|
||||
==================================================
|
||||
```
|
||||
### 5.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 85.84
|
||||
Total input tokens: 249841
|
||||
Total input text tokens: 249841
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250498
|
||||
Request throughput (req/s): 5.84
|
||||
Input token throughput (tok/s): 2910.84
|
||||
Output token throughput (tok/s): 2944.82
|
||||
Peak output token throughput (tok/s): 4100.00
|
||||
Peak concurrent requests: 110
|
||||
Total token throughput (tok/s): 5854.65
|
||||
Concurrency: 92.24
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 15844.00
|
||||
Median E2E Latency (ms): 15262.56
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 204.46
|
||||
Median TTFT (ms): 129.96
|
||||
P99 TTFT (ms): 528.54
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 41.56
|
||||
Median TPOT (ms): 42.90
|
||||
P99 TPOT (ms): 47.48
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 40.99
|
||||
Median ITL (ms): 24.46
|
||||
P95 ITL (ms): 84.46
|
||||
P99 ITL (ms): 87.64
|
||||
Max ITL (ms): 226.06
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark (Maverick)
|
||||
Test Environment:
|
||||
|
||||
Hardware: AMD MI300x GPU
|
||||
|
||||
Model: Llama-4-Maverick
|
||||
|
||||
Tensor Parallelism: 8
|
||||
|
||||
sglang version: 0.5.9
|
||||
|
||||
- **Model Deployment**
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--tp 8 \
|
||||
--context-length 1000000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
### 5.2.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 68.08
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4202
|
||||
Request throughput (req/s): 0.15
|
||||
Input token throughput (tok/s): 89.62
|
||||
Output token throughput (tok/s): 61.99
|
||||
Peak output token throughput (tok/s): 168.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 151.61
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 6805.62
|
||||
Median E2E Latency (ms): 2733.91
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4296.56
|
||||
Median TTFT (ms): 57.45
|
||||
P99 TTFT (ms): 38633.95
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 5.95
|
||||
Median TPOT (ms): 5.96
|
||||
P99 TPOT (ms): 5.97
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 5.96
|
||||
Median ITL (ms): 5.96
|
||||
P95 ITL (ms): 6.02
|
||||
P99 ITL (ms): 6.08
|
||||
Max ITL (ms): 7.02
|
||||
==================================================
|
||||
```
|
||||
### 5.2.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 30.72
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40923
|
||||
Request throughput (req/s): 2.60
|
||||
Input token throughput (tok/s): 1291.39
|
||||
Output token throughput (tok/s): 1328.41
|
||||
Peak output token throughput (tok/s): 1760.00
|
||||
Peak concurrent requests: 22
|
||||
Total token throughput (tok/s): 2619.80
|
||||
Concurrency: 13.92
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 5345.15
|
||||
Median E2E Latency (ms): 5679.73
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 259.30
|
||||
Median TTFT (ms): 72.60
|
||||
P99 TTFT (ms): 1063.45
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 10.53
|
||||
Median TPOT (ms): 10.22
|
||||
P99 TPOT (ms): 20.27
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 9.99
|
||||
Median ITL (ms): 9.10
|
||||
P95 ITL (ms): 9.87
|
||||
P99 ITL (ms): 55.62
|
||||
Max ITL (ms): 868.54
|
||||
==================================================
|
||||
```
|
||||
### 5.2.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 90.95
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 251625
|
||||
Request throughput (req/s): 5.50
|
||||
Input token throughput (tok/s): 2746.77
|
||||
Output token throughput (tok/s): 2777.90
|
||||
Peak output token throughput (tok/s): 3700.00
|
||||
Peak concurrent requests: 109
|
||||
Total token throughput (tok/s): 5524.67
|
||||
Concurrency: 93.04
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 16924.17
|
||||
Median E2E Latency (ms): 16294.85
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 188.19
|
||||
Median TTFT (ms): 128.96
|
||||
P99 TTFT (ms): 534.81
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 33.63
|
||||
Median TPOT (ms): 35.37
|
||||
P99 TPOT (ms): 38.26
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 33.19
|
||||
Median ITL (ms): 27.66
|
||||
P95 ITL (ms): 76.91
|
||||
P99 ITL (ms): 78.82
|
||||
Max ITL (ms): 268.17
|
||||
==================================================
|
||||
```
|
||||
### 5.3 Accuracy Benchmark
|
||||
|
||||
#### 5.3.1 GSM8K Benchmark
|
||||
|
||||
- **Benchmark Command:**
|
||||
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
- Llama-4-Scout-17B-16E-Instruct
|
||||
```text Output
|
||||
Accuracy: 0.945
|
||||
Invalid: 0.000
|
||||
Latency: 12.731 s
|
||||
Output throughput: 1595.418 token/s
|
||||
```
|
||||
- Llama-4-Maverick-17B-128E-Instruct
|
||||
```text Output
|
||||
Accuracy: 0.895
|
||||
Invalid: 0.000
|
||||
Latency: 9.739 s
|
||||
Output throughput: 2405.505 token/s
|
||||
```
|
||||
|
||||
#### 5.3.2 MMLU Pro with lm-eval
|
||||
|
||||
Accuracy on MMLU Pro matches [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) on 8×H100 (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)):
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Official</th>
|
||||
<th style={{textAlign: "right", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Scout-17B-16E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>74.3</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>75.2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-4-Maverick-17B-128E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.05)"}}>80.5</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "right", backgroundColor: "rgba(255,255,255,0.02)"}}>80.7</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Scout:**
|
||||
|
||||
```bash Command
|
||||
# Start the server
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
|
||||
# Run lm_eval
|
||||
lm_eval --model local-chat-completions \
|
||||
--model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
|
||||
--tasks mmlu_pro \
|
||||
--batch_size 128 \
|
||||
--apply_chat_template \
|
||||
--num_fewshot 0
|
||||
```
|
||||
|
||||
**Maverick:**
|
||||
|
||||
```bash Command
|
||||
# Start the server
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
|
||||
--port 30000 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 65536
|
||||
|
||||
# Run lm_eval
|
||||
lm_eval --model local-chat-completions \
|
||||
--model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
|
||||
--tasks mmlu_pro \
|
||||
--batch_size 128 \
|
||||
--apply_chat_template \
|
||||
--num_fewshot 0
|
||||
```
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
title: LongCat-2.0
|
||||
description: "Deploy LongCat-2.0-FP8 with SGLang - config-driven recipes for Meituan's 1.6T sparse MoE model on B300, B200, H200, and H20 GPUs."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). LongCat-2.0 support is on SGLang `main`; use a nightly wheel or rolling nightly Docker image until the next tagged release includes it. The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
|
||||
# Choose the nightly wheel index for your CUDA runtime.
|
||||
SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu130 # B300 / CUDA 13
|
||||
# SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu129 # CUDA 12.9
|
||||
|
||||
uv pip install --prerelease=allow --extra-index-url "${SGLANG_WHL_INDEX}" "sglang[all]"
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# Choose the rolling nightly image for your hardware.
|
||||
SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev-cu13 # B300 / CUDA 13
|
||||
# SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev # Other supported hardware
|
||||
|
||||
docker pull "${SGLANG_DOCKER_IMAGE}"
|
||||
```
|
||||
|
||||
For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. LongCat-2.0 currently exposes one model-card-aligned serving strategy:
|
||||
|
||||
- **Balanced** - the validated B300 recipe and the 2-node H200/B200/H20 topology use TP/EP parallelism with LongCat sparse attention prefill.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/meituan-longcat/longcat-2.0.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/meituan-longcat/longcat-2.0-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the LongCat sparse-attention indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on LongCat-2.0.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
The B300 single-node recipe was validated end-to-end with CUDA graph capture enabled. H200, B200, and H20 are shown as 2-node recipes because LongCat-2.0-FP8 needs 16 ranks for those GPU memory profiles.
|
||||
</Note>
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) is the FP8 checkpoint of Meituan LongCat-2.0, a large sparse Mixture-of-Experts language model with 1.6T total parameters and about 48B activated parameters per token. It combines LongCat Sparse Attention (LSA), expert parallel MoE layers, and an n-gram/token-table embedding path for serving long-context workloads efficiently.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Architecture</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Serving precision</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><strong><a href="https://huggingface.co/meituan-longcat/LongCat-2.0-FP8">LongCat-2.0-FP8</a></strong></td>
|
||||
<td style={{padding: "9px 12px"}}>Sparse MoE · LongCat Sparse Attention · n-gram embedding</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 weights, BF16 KV cache</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Resources:** [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8).
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
- **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint.
|
||||
- **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you.
|
||||
- **LongCat sparse attention.** Keep `--nsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path.
|
||||
- **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster.
|
||||
- **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time.
|
||||
- **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout.
|
||||
- **Host, port, and ranks.** Use the command panel environment fields for `HOST_IP`, `PORT`, `NODE0_IP`, and `NODE_RANK` instead of hardcoding them in the recipe.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Test the deployment
|
||||
|
||||
<Accordion title="Chat completion example (cURL)">
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "meituan-longcat/LongCat-2.0-FP8",
|
||||
"messages": [
|
||||
{"role": "user", "content": "A shop has 17 apples and sells 8. Then it buys 6 more. How many apples are there? Answer with only the final number."}
|
||||
],
|
||||
"max_tokens": 32,
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Expected output">
|
||||
|
||||
```text Output
|
||||
15
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="OpenAI-compatible client (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="meituan-longcat/LongCat-2.0-FP8",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Solve: A shop has 17 apples and sells 8, then buys 6 more. Answer with only the final number.",
|
||||
}
|
||||
],
|
||||
max_tokens=32,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example output">
|
||||
|
||||
```text Output
|
||||
15
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
## 4. Validation
|
||||
|
||||
The B300 recipe was validated with `meituan-longcat/LongCat-2.0-FP8` on 8x B300 using the command generated above.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Evaluation</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Examples</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Accuracy</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>GSM8K</td>
|
||||
<td style={{padding: "9px 12px"}}>200</td>
|
||||
<td style={{padding: "9px 12px"}}>98.0%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>GSM8K</td>
|
||||
<td style={{padding: "9px 12px"}}>1314</td>
|
||||
<td style={{padding: "9px 12px"}}>95.8904109589041%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
CUDA graph was enabled, and decode CUDA graph capture completed successfully during serving validation.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,730 @@
|
||||
---
|
||||
title: MiniMax-M2.7
|
||||
metatags:
|
||||
description: "Deploy MiniMax-M2.7 with SGLang on NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs — model self-evolution, professional software engineering, and native agent teams."
|
||||
---
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search.
|
||||
|
||||
Key highlights:
|
||||
|
||||
- **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**.
|
||||
- **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems.
|
||||
- **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier).
|
||||
- **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making.
|
||||
|
||||
For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en).
|
||||
|
||||
**License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE)
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
**Docker Images by Hardware Platform:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Hardware Platform</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Docker Image</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA A100 / H100 / H200 / B200</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NVIDIA B300 / GB300</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-cu130`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI300X / MI325X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AMD MI355X</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities.
|
||||
|
||||
import { MiniMaxM27Deployment } from '/src/snippets/autoregressive/minimax-m27-deployment.jsx'
|
||||
|
||||
<MiniMaxM27Deployment />
|
||||
|
||||
### 3.2 Configuration Tips
|
||||
|
||||
**Key Parameters:**
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Recommended Value</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tool-call-parser`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tool call parser for function calling support</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-m2`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--reasoning-parser`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Reasoning parser for thinking mode</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`minimax-append-think`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--trust-remote-code`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Required for MiniMax model loading</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Always enabled</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--mem-fraction-static`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Static memory fraction for KV cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`0.85`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--tp`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tensor parallelism size</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`2` / `4` / `8` depending on hardware</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--ep`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Expert parallelism size</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`8` (NVIDIA 8-GPU) or EP=TP (AMD)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--kv-cache-dtype`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>KV cache data type (AMD only)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`fp8_e4m3`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-backend`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Attention backend (AMD only)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
**Hardware Requirements: NVIDIA**
|
||||
|
||||
- **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4
|
||||
- **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8
|
||||
|
||||
**Hardware Requirements: NVIDIA GB300**
|
||||
|
||||
- **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2
|
||||
- **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput
|
||||
|
||||
**Hardware Requirements: AMD**
|
||||
|
||||
- **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2
|
||||
- **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
|
||||
- **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8
|
||||
|
||||
**Hardware Requirements: Intel Xeon CPU**
|
||||
|
||||
- It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server.
|
||||
|
||||
- For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
**Deployment Command:**
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2.7 \
|
||||
--tp 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**Testing Deployment:**
|
||||
|
||||
After startup, you can test the SGLang OpenAI-compatible API with the following command:
|
||||
|
||||
```bash Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "MiniMaxAI/MiniMax-M2.7",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
|
||||
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
**Simple Completion Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "Who won the world series in 2020?"}
|
||||
],
|
||||
max_tokens=1024
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Example Output**:
|
||||
```text Output
|
||||
<think>The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly.
|
||||
|
||||
We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context.
|
||||
|
||||
We should answer concisely.
|
||||
|
||||
Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas.
|
||||
|
||||
We must avoid disallowed content, no issue.
|
||||
|
||||
Thus final.
|
||||
</think>
|
||||
|
||||
The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic.
|
||||
```
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
|
||||
MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2.7 \
|
||||
--tp 4 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**Streaming with Thinking Process**
|
||||
|
||||
With `minimax-append-think`, the thinking content is wrapped in `<think>...</think>` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections:
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream, separating <think>...</think> from content
|
||||
in_think = False
|
||||
think_printed_header = False
|
||||
content_printed_header = False
|
||||
buffer = ""
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
if delta.content:
|
||||
buffer += delta.content
|
||||
|
||||
while buffer:
|
||||
if in_think:
|
||||
# Look for closing </think> tag
|
||||
end_idx = buffer.find("</think>")
|
||||
if end_idx != -1:
|
||||
print(buffer[:end_idx], end="", flush=True)
|
||||
buffer = buffer[end_idx + len("</think>"):]
|
||||
in_think = False
|
||||
else:
|
||||
# Still in thinking, print what we have
|
||||
print(buffer, end="", flush=True)
|
||||
buffer = ""
|
||||
else:
|
||||
# Look for opening <think> tag
|
||||
start_idx = buffer.find("<think>")
|
||||
if start_idx != -1:
|
||||
# Print any content before <think>
|
||||
before = buffer[:start_idx]
|
||||
if before:
|
||||
if not content_printed_header:
|
||||
print("=============== Content =================", flush=True)
|
||||
content_printed_header = True
|
||||
print(before, end="", flush=True)
|
||||
buffer = buffer[start_idx + len("<think>"):]
|
||||
in_think = True
|
||||
if not think_printed_header:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
think_printed_header = True
|
||||
else:
|
||||
# No <think> tag, print as content
|
||||
if not content_printed_header and think_printed_header:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
content_printed_header = True
|
||||
print(buffer, end="", flush=True)
|
||||
buffer = ""
|
||||
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
=============== Thinking =================
|
||||
The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation.
|
||||
|
||||
But also ensure we follow any policy? No issues. Just straightforward.
|
||||
|
||||
I'll provide a step-by-step solution.
|
||||
|
||||
Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.
|
||||
|
||||
Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36.
|
||||
|
||||
All good.
|
||||
|
||||
=============== Content =================
|
||||
|
||||
**Step‑by‑step solution**
|
||||
|
||||
1. **Convert the percent to a decimal (or a fraction).**
|
||||
|
||||
15% = 15/100 = 0.15 = 3/20
|
||||
|
||||
2. **Multiply the original number (240) by this decimal/fraction.**
|
||||
|
||||
Using the decimal:
|
||||
240 × 0.15 = 36
|
||||
|
||||
Or using the fraction:
|
||||
240 × 3/20 = (240/20) × 3 = 12 × 3 = 36
|
||||
|
||||
3. **Result:**
|
||||
|
||||
15% of 240 = **36**
|
||||
|
||||
*Check:*
|
||||
- 10% of 240 = 24
|
||||
- 5% of 240 = 12
|
||||
- Adding them: 24 + 12 = 36, which matches the calculation.
|
||||
```
|
||||
|
||||
**Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `<think>...</think>` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately.
|
||||
|
||||
#### 4.2.2 Tool Calling
|
||||
|
||||
MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2.7 \
|
||||
--tp 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**Python Example:**
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Non-streaming request
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
|
||||
# Check for tool calls
|
||||
if message.tool_calls:
|
||||
for tool_call in message.tool_calls:
|
||||
print(f"Tool Call: {tool_call.function.name}")
|
||||
print(f" Arguments: {tool_call.function.arguments}")
|
||||
else:
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
**Output Example**:
|
||||
```text Output
|
||||
Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
**Handling Tool Call Results:**
|
||||
|
||||
```python Example
|
||||
# After getting the tool call, execute the function
|
||||
def get_weather(location, unit="celsius"):
|
||||
# Your actual weather API call here
|
||||
return f"The weather in {location} is 22°{unit[0].upper()} and sunny."
|
||||
|
||||
# Send tool result back to the model
|
||||
messages = [
|
||||
{"role": "user", "content": "What's the weather in Beijing?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{
|
||||
"id": "call_123",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": '{"location": "Beijing", "unit": "celsius"}'
|
||||
}
|
||||
}]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_123",
|
||||
"content": get_weather("Beijing", "celsius")
|
||||
}
|
||||
]
|
||||
|
||||
final_response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2.7",
|
||||
messages=messages
|
||||
)
|
||||
|
||||
print(final_response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
```text Output
|
||||
The weather in Beijing is currently 22°C and sunny.
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
|
||||
This section uses **industry-standard configurations** for comparable benchmark results.
|
||||
|
||||
**Test Environment**:
|
||||
|
||||
- Hardware: 2× NVIDIA GB300 (275GB per die)
|
||||
- Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130`
|
||||
- Model: MiniMax-M2.7 (FP8)
|
||||
- Tensor Parallelism: 2
|
||||
- SGLang version: 0.5.10.post1
|
||||
|
||||
### 5.1 Accuracy Benchmark
|
||||
|
||||
**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills)
|
||||
|
||||
**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True`
|
||||
|
||||
#### 5.1.1 GPQA Diamond
|
||||
|
||||
- Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
|
||||
- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking))
|
||||
- Evaluation command:
|
||||
```bash Command
|
||||
ns prepare_data gpqa
|
||||
|
||||
ns eval \
|
||||
--cluster=local \
|
||||
--server_type=openai \
|
||||
--model=MiniMaxAI/MiniMax-M2.7 \
|
||||
--server_address=http://localhost:30000/v1 \
|
||||
--output_dir=./m2.7-eval/ \
|
||||
--benchmarks=gpqa:8 \
|
||||
++prompt_config=eval/aai/mcq-4choices \
|
||||
++inference.tokens_to_generate=120000 \
|
||||
++inference.temperature=0.6 \
|
||||
++inference.top_p=0.95 \
|
||||
++parse_reasoning=True
|
||||
```
|
||||
- Test Results:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>84.91%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>3.54%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**88.89%**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>96.46%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.2 AIME 2025
|
||||
|
||||
- Dataset: AIME 2025 (30 problems)
|
||||
- Prompt: `generic/math` (boxed answer format)
|
||||
- Evaluation command:
|
||||
```bash Command
|
||||
ns prepare_data aime25
|
||||
|
||||
ns eval \
|
||||
--cluster=local \
|
||||
--server_type=openai \
|
||||
--model=MiniMaxAI/MiniMax-M2.7 \
|
||||
--server_address=http://localhost:30000/v1 \
|
||||
--output_dir=./m2.7-eval/ \
|
||||
--benchmarks=aime25:8 \
|
||||
++inference.tokens_to_generate=120000 \
|
||||
++inference.temperature=0.6 \
|
||||
++inference.top_p=0.95 \
|
||||
++parse_reasoning=True
|
||||
```
|
||||
- Test Results:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (avg-of-8)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>92.50% ± 5.56%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>2.92%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**majority@8**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>**97.08%**</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>100.00%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.00%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
#### 5.1.3 MMLU-Pro
|
||||
|
||||
- Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
|
||||
- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice)
|
||||
- Evaluation command:
|
||||
```bash Command
|
||||
ns prepare_data mmlu-pro
|
||||
|
||||
ns eval \
|
||||
--cluster=local \
|
||||
--server_type=openai \
|
||||
--model=MiniMaxAI/MiniMax-M2.7 \
|
||||
--server_address=http://localhost:30000/v1 \
|
||||
--output_dir=./m2.7-eval/ \
|
||||
--benchmarks=mmlu-pro \
|
||||
++prompt_config=eval/aai/mcq-10choices \
|
||||
++inference.tokens_to_generate=32768 \
|
||||
++inference.temperature=0.0 \
|
||||
++parse_reasoning=True
|
||||
```
|
||||
- Test Results:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Evaluation Mode</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Accuracy</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>No Answer</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>pass@1 (greedy)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>69.41%</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>18.75%</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly.
|
||||
|
||||
#### 5.1.4 GSM8K Benchmark
|
||||
- Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API
|
||||
- Test Results:
|
||||
```text Output
|
||||
GSM8K Results (8-shot CoT)
|
||||
Model: MiniMaxAI/MiniMax-M2.7
|
||||
Total: 1319
|
||||
Correct: 1218
|
||||
Accuracy: 92.34%
|
||||
```
|
||||
|
||||
### 5.2 Speed Benchmark
|
||||
|
||||
#### 5.2.1 Low Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 34.33
|
||||
Total input tokens: 6101
|
||||
Total generated tokens: 4220
|
||||
Request throughput (req/s): 0.29
|
||||
Input token throughput (tok/s): 177.71
|
||||
Output token throughput (tok/s): 122.92
|
||||
Total token throughput (tok/s): 300.63
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 3431.21
|
||||
Median E2E Latency (ms): 2742.57
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 50.28
|
||||
Median TTFT (ms): 53.85
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 8.02
|
||||
Median TPOT (ms): 8.01
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 8.03
|
||||
Median ITL (ms): 8.02
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### 5.2.2 High Concurrency
|
||||
|
||||
- Benchmark Command:
|
||||
```shell Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2.7 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 100.20
|
||||
Total input tokens: 249831
|
||||
Total generated tokens: 252662
|
||||
Request throughput (req/s): 4.99
|
||||
Input token throughput (tok/s): 2493.41
|
||||
Output token throughput (tok/s): 2521.66
|
||||
Total token throughput (tok/s): 5015.07
|
||||
Concurrency: 90.19
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 18072.69
|
||||
Median E2E Latency (ms): 17761.84
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 247.94
|
||||
Median TTFT (ms): 92.05
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 35.75
|
||||
Median TPOT (ms): 36.67
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 35.34
|
||||
Median ITL (ms): 30.55
|
||||
==================================================
|
||||
```
|
||||
@@ -0,0 +1,613 @@
|
||||
---
|
||||
title: MiniMax-M2
|
||||
metatags:
|
||||
description: "Deploy MiniMax-M2 with SGLang - community contribution guide for MiniMax M2 model deployment."
|
||||
---
|
||||
|
||||
import { MiniMaxM2Deployment } from '/src/snippets/autoregressive/minimax-m2-deployment.jsx';
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence.
|
||||
|
||||
This generation delivers comprehensive upgrades across the board:
|
||||
|
||||
- **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally.
|
||||
|
||||
- **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages.
|
||||
|
||||
- **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps.
|
||||
|
||||
- **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks.
|
||||
|
||||
For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI).
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The AMD environment is currently available in SGLang via Docker image install.
|
||||
|
||||
### 2.1 AMD Docker
|
||||
#### 2.1.1 Launch docker
|
||||
```shell Command
|
||||
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
|
||||
```
|
||||
```shell Command
|
||||
docker run -d -it --ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN \
|
||||
--device=/dev/kfd --device=/dev/dri --device=/dev/mem \
|
||||
--group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v /:/work \
|
||||
-e SHELL=/bin/bash \
|
||||
--name Minimax \
|
||||
lmsysorg/sglang:v0.5.9-rocm720-mi30x \
|
||||
/bin/bash
|
||||
```
|
||||
|
||||
#### 2.1.2 Make modifications inside the docker
|
||||
|
||||
```shell Command
|
||||
mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \
|
||||
/sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py
|
||||
```
|
||||
|
||||
#### 2.1.3 Fix torch compile
|
||||
Comment out the following line: @torch.compile(dynamic=True, backend=get_compiler_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax_m2.py
|
||||
```shell Command
|
||||
#@torch.compile(dynamic=True, backend=get_compiler_backend())
|
||||
```
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.
|
||||
|
||||
<MiniMaxM2Deployment />
|
||||
|
||||
#### 3.1.1 NVIDIA GPU Deployment
|
||||
|
||||
The interactive command generator above covers AMD deployments. For NVIDIA GPUs (H100/H200/B200), use these explicit commands:
|
||||
|
||||
**4-GPU deployment (up to 400K context):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 30000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
**8-GPU deployment (up to 3M context):**
|
||||
|
||||
```bash Command
|
||||
python -m sglang.launch_server \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--host 0.0.0.0 \
|
||||
--trust-remote-code \
|
||||
--port 30000 \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
### 3.2 System Requirements
|
||||
|
||||
Recommended configurations — actual requirements depend on workload:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.02)"}}>GPUs</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, backgroundColor: "rgba(255,255,255,0.05)"}}>Context Length Support</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>4× 96 GB GPUs</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Up to 400K tokens</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>8× 144 GB GPUs</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Up to 3M tokens</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
### 3.3 Testing Deployment
|
||||
|
||||
After the server starts, verify with:
|
||||
|
||||
```shell Command
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "MiniMaxAI/MiniMax-M2",
|
||||
"messages": [
|
||||
{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
|
||||
{"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## 4. Model Invocation
|
||||
|
||||
### 4.1 Basic Usage
|
||||
|
||||
For basic API usage and request examples, please refer to:
|
||||
|
||||
- [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
|
||||
|
||||
### 4.2 Advanced Usage
|
||||
|
||||
#### 4.2.1 Reasoning Parser
|
||||
Server Command:
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--reasoning-parser minimax-append-think \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
Test Code:
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Enable streaming to see the thinking process in real-time
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2",
|
||||
messages=[
|
||||
{"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
|
||||
],
|
||||
temperature=0.6,
|
||||
max_tokens=2048,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process the stream
|
||||
has_thinking = False
|
||||
has_answer = False
|
||||
thinking_started = False
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Print answer content
|
||||
if delta.content:
|
||||
# Close thinking section and add content header
|
||||
if has_thinking and not has_answer:
|
||||
print("\n=============== Content =================", flush=True)
|
||||
has_answer = True
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
print()
|
||||
```
|
||||
Output Example:
|
||||
```text Output
|
||||
<think>First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction.
|
||||
|
||||
The problem is: What is 15% of 240?
|
||||
|
||||
To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15.
|
||||
|
||||
Therefore, 15% of 240 is 240 times 0.15.
|
||||
|
||||
Let me calculate that: 240 × 0.15.
|
||||
|
||||
I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100.
|
||||
|
||||
Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600.
|
||||
|
||||
Then, divide by 100: 3600 / 100 = 36.
|
||||
|
||||
So, 15% of 240 is 36.
|
||||
|
||||
I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer.
|
||||
|
||||
Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36.
|
||||
|
||||
All methods confirm it's 36.
|
||||
|
||||
The user said "solve this problem step by step," so I should present the steps clearly.
|
||||
|
||||
Step 1: Understand that "15% of 240" means 15 per hundred of 240.
|
||||
|
||||
Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15.
|
||||
|
||||
Step 3: Multiply the number by the decimal: 240 × 0.15.
|
||||
|
||||
Step 4: Calculate the multiplication: 240 × 0.15 = 36.
|
||||
|
||||
Step 5: Therefore, 15% of 240 is 36.
|
||||
|
||||
I should also mention that percentage means per hundred, so it's straightforward.
|
||||
|
||||
Finally, I need to box the answer as per the instruction.
|
||||
|
||||
So, the final answer is 36.
|
||||
</think>
|
||||
|
||||
To find 15% of 240, follow these steps:
|
||||
|
||||
1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100.
|
||||
2. **Convert to a decimal**: 15/100 = 0.15.
|
||||
3. **Multiply by the number**: 240 × 0.15.
|
||||
4. **Calculate the result**:
|
||||
- 240 × 0.15 = 36.
|
||||
|
||||
Alternatively, you can break it down:
|
||||
- 10% of 240 is 24 (since 240 ÷ 10 = 24).
|
||||
- 5% of 240 is half of 10%, which is 12.
|
||||
- Therefore, 15% is 10% + 5% = 24 + 12 = 36.
|
||||
|
||||
Both methods confirm the result.
|
||||
|
||||
**Answer**: 36
|
||||
```
|
||||
|
||||
### 4.2.2 Tool Calling
|
||||
|
||||
Server Command:
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--tool-call-parser minimax-m2 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
Test Code:
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY"
|
||||
)
|
||||
|
||||
# Define available tools
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {
|
||||
"type": "string",
|
||||
"description": "The city name"
|
||||
},
|
||||
"unit": {
|
||||
"type": "string",
|
||||
"enum": ["celsius", "fahrenheit"],
|
||||
"description": "Temperature unit"
|
||||
}
|
||||
},
|
||||
"required": ["location"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
# Make request with streaming to see thinking process
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M2",
|
||||
messages=[
|
||||
{"role": "user", "content": "What's the weather in Beijing?"}
|
||||
],
|
||||
tools=tools,
|
||||
temperature=0.7,
|
||||
stream=True
|
||||
)
|
||||
|
||||
# Process streaming response
|
||||
thinking_started = False
|
||||
has_thinking = False
|
||||
tool_calls_accumulator = {}
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
|
||||
# Print thinking process
|
||||
if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
|
||||
if not thinking_started:
|
||||
print("=============== Thinking =================", flush=True)
|
||||
thinking_started = True
|
||||
has_thinking = True
|
||||
print(delta.reasoning_content, end="", flush=True)
|
||||
|
||||
# Accumulate tool calls
|
||||
if hasattr(delta, 'tool_calls') and delta.tool_calls:
|
||||
# Close thinking section if needed
|
||||
if has_thinking and thinking_started:
|
||||
print("\n=============== Content =================\n", flush=True)
|
||||
thinking_started = False
|
||||
|
||||
for tool_call in delta.tool_calls:
|
||||
index = tool_call.index
|
||||
if index not in tool_calls_accumulator:
|
||||
tool_calls_accumulator[index] = {
|
||||
'name': None,
|
||||
'arguments': ''
|
||||
}
|
||||
|
||||
if tool_call.function:
|
||||
if tool_call.function.name:
|
||||
tool_calls_accumulator[index]['name'] = tool_call.function.name
|
||||
if tool_call.function.arguments:
|
||||
tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments
|
||||
|
||||
# Print content
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True)
|
||||
|
||||
# Print accumulated tool calls
|
||||
for index, tool_call in sorted(tool_calls_accumulator.items()):
|
||||
print(f"🔧 Tool Call: {tool_call['name']}")
|
||||
print(f" Arguments: {tool_call['arguments']}")
|
||||
|
||||
print()
|
||||
```
|
||||
Output Example:
|
||||
```text Output
|
||||
Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me.
|
||||
|
||||
Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me.
|
||||
|
||||
Looking at the tool's parameters, I can see it requires:
|
||||
1. location - which is required and should be a string representing the city name
|
||||
2. unit - which is optional and can be either "celsius" or "fahrenheit"
|
||||
|
||||
For the location parameter, I'll use "Beijing" since that's what the user asked about.
|
||||
|
||||
For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information.
|
||||
|
||||
So I need to make a tool call to get_weather with the following parameters:
|
||||
- location: "Beijing"
|
||||
- unit: "celsius"
|
||||
|
||||
This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions.
|
||||
</think>
|
||||
|
||||
🔧 Tool Call: get_weather
|
||||
Arguments: {"location": "Beijing", "unit": "celsius"}
|
||||
```
|
||||
|
||||
## 5. Benchmark
|
||||
### 5.1 Speed Benchmark
|
||||
**Test Environment**:
|
||||
|
||||
- Hardware: AMD MI300X GPU(4x)
|
||||
|
||||
- Model: MiniMax-M2
|
||||
|
||||
- Tensor Parallelism: 4
|
||||
|
||||
- sglang version: 0.5.7
|
||||
|
||||
**Model Deployment**:
|
||||
|
||||
```bash Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
### 5.1.1 Low Concurrency (Latency-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 10 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate inf
|
||||
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 1
|
||||
Successful requests: 10
|
||||
Benchmark duration (s): 138.91
|
||||
Total input tokens: 6101
|
||||
Total input text tokens: 6101
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 4220
|
||||
Total generated tokens (retokenized): 4220
|
||||
Request throughput (req/s): 0.07
|
||||
Input token throughput (tok/s): 43.92
|
||||
Output token throughput (tok/s): 30.38
|
||||
Peak output token throughput (tok/s): 46.00
|
||||
Peak concurrent requests: 2
|
||||
Total token throughput (tok/s): 74.30
|
||||
Concurrency: 1.00
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13887.62
|
||||
Median E2E Latency (ms): 10377.26
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 4528.94
|
||||
Median TTFT (ms): 385.23
|
||||
P99 TTFT (ms): 38338.51
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 22.21
|
||||
Median TPOT (ms): 22.24
|
||||
P99 TPOT (ms): 22.25
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 22.23
|
||||
Median ITL (ms): 22.24
|
||||
P95 ITL (ms): 22.35
|
||||
P99 ITL (ms): 22.41
|
||||
Max ITL (ms): 23.64
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.1.2 Medium Concurrency (Balanced)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 80 \
|
||||
--max-concurrency 16 \
|
||||
--request-rate inf
|
||||
|
||||
```
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 16
|
||||
Successful requests: 80
|
||||
Benchmark duration (s): 81.07
|
||||
Total input tokens: 39668
|
||||
Total input text tokens: 39668
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 40805
|
||||
Total generated tokens (retokenized): 40803
|
||||
Request throughput (req/s): 0.99
|
||||
Input token throughput (tok/s): 489.29
|
||||
Output token throughput (tok/s): 503.32
|
||||
Peak output token throughput (tok/s): 704.00
|
||||
Peak concurrent requests: 19
|
||||
Total token throughput (tok/s): 992.61
|
||||
Concurrency: 13.74
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 13925.95
|
||||
Median E2E Latency (ms): 14348.75
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 532.32
|
||||
Median TTFT (ms): 147.69
|
||||
P99 TTFT (ms): 1978.48
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 27.49
|
||||
Median TPOT (ms): 26.56
|
||||
P99 TPOT (ms): 46.52
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 26.31
|
||||
Median ITL (ms): 23.47
|
||||
P95 ITL (ms): 24.37
|
||||
P99 ITL (ms): 125.10
|
||||
Max ITL (ms): 1192.51
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.1.3 High Concurrency (Throughput-Optimized)
|
||||
- Benchmark Command:
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--model MiniMaxAI/MiniMax-M2 \
|
||||
--dataset-name random \
|
||||
--random-input-len 1000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 500 \
|
||||
--max-concurrency 100 \
|
||||
--request-rate inf
|
||||
```
|
||||
|
||||
- Test Results:
|
||||
```text Output
|
||||
============ Serving Benchmark Result ============
|
||||
Backend: sglang
|
||||
Traffic request rate: inf
|
||||
Max request concurrency: 100
|
||||
Successful requests: 500
|
||||
Benchmark duration (s): 153.71
|
||||
Total input tokens: 249831
|
||||
Total input text tokens: 249831
|
||||
Total input vision tokens: 0
|
||||
Total generated tokens: 252662
|
||||
Total generated tokens (retokenized): 250982
|
||||
Request throughput (req/s): 3.25
|
||||
Input token throughput (tok/s): 1625.33
|
||||
Output token throughput (tok/s): 1643.75
|
||||
Peak output token throughput (tok/s): 2597.00
|
||||
Peak concurrent requests: 107
|
||||
Total token throughput (tok/s): 3269.09
|
||||
Concurrency: 91.14
|
||||
----------------End-to-End Latency----------------
|
||||
Mean E2E Latency (ms): 28017.24
|
||||
Median E2E Latency (ms): 26865.28
|
||||
---------------Time to First Token----------------
|
||||
Mean TTFT (ms): 387.41
|
||||
Median TTFT (ms): 183.90
|
||||
P99 TTFT (ms): 1192.44
|
||||
-----Time per Output Token (excl. 1st token)------
|
||||
Mean TPOT (ms): 55.23
|
||||
Median TPOT (ms): 57.84
|
||||
P99 TPOT (ms): 70.23
|
||||
---------------Inter-Token Latency----------------
|
||||
Mean ITL (ms): 54.79
|
||||
Median ITL (ms): 39.01
|
||||
P95 ITL (ms): 143.10
|
||||
P99 ITL (ms): 150.46
|
||||
Max ITL (ms): 986.14
|
||||
==================================================
|
||||
```
|
||||
|
||||
### 5.2 Accuracy Benchmark
|
||||
#### 5.2.1 GSM8K Benchmark
|
||||
|
||||
- **Server Command**:
|
||||
```shell Command
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M2 \
|
||||
--tp-size 4 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.85
|
||||
```
|
||||
|
||||
- **Benchmark Command**:
|
||||
```shell Command
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
- **Result**:
|
||||
- MiniMax-M2
|
||||
```text Output
|
||||
Accuracy: 0.950
|
||||
Invalid: 0.000
|
||||
Latency: 15.120 s
|
||||
Output throughput: 1306.711 token/s
|
||||
```
|
||||
@@ -0,0 +1,506 @@
|
||||
---
|
||||
title: MiniMax-M3
|
||||
description: "Deploy MiniMax-M3 with SGLang — a ~428B-param (23B activated) multimodal Mixture-of-Experts reasoning model with MiniMax Sparse Attention and 1M context, MXFP8 on NVIDIA Blackwell & AMD Instinct, bf16 on Hopper."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
<a id="install" />
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Python (pip / uv)">
|
||||
|
||||
```bash Command
|
||||
pip install -U uv
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
|
||||
# MiniMax-M3 ships in SGLang PR #27944, not yet in a tagged release — install from
|
||||
# the PR head. The serving runtime is in the base dependencies, so no extra is needed:
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git fetch origin pull/27944/head && git checkout FETCH_HEAD
|
||||
uv pip install -e python
|
||||
```
|
||||
|
||||
Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — its image bundles the CUDA-13 runtime and the #27944 code. Once [PR #27944](https://github.com/sgl-project/sglang/pull/27944) is merged and released, `uv pip install sglang` will pull M3 support directly.
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
# Pull the M3 image the command panel selects for your platform, e.g.:
|
||||
docker pull lmsysorg/sglang:dev-cu13-minimax-m3
|
||||
```
|
||||
|
||||
The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces.
|
||||
|
||||
<Note>
|
||||
These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path.
|
||||
</Note>
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/MiniMaxAI/minimax-m3.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/MiniMaxAI/minimax-m3-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
|
||||
## Playground
|
||||
|
||||
The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.
|
||||
|
||||
import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
[MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) is MiniMax's native-multimodal Mixture-of-Experts reasoning model: **~428B total parameters with ~23B activated per token** (128 experts, 4 active per token), 60 layers, and a **1M-token context** over text, image, and video. Its defining feature is **MiniMax Sparse Attention (MSA)** — a block-sparse "lightning indexer" attention that keeps long-context cost low (MiniMax reports ~9× prefill / ~15× decode speedup over M2 at 1M context). This page serves the **MXFP8** variant (`MiniMaxAI/MiniMax-M3-MXFP8`, ~440 GB) on NVIDIA Blackwell and AMD Instinct; on NVIDIA Hopper (H200), use the full-precision **bfloat16** build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3) (§2.4). Released under the **MiniMax Community License**.
|
||||
|
||||
Key characteristics as served by SGLang:
|
||||
|
||||
- **Multimodal (vision + text)**: accepts interleaved text and images through the OpenAI-compatible chat API (loaded as `MiniMaxM3SparseForConditionalGeneration`). Image input via URL and base64 is validated; video input has not been tested here.
|
||||
- **Reasoning model**: emits its chain of thought wrapped in `<mm:think>...</mm:think>`. Always launch with **`--reasoning-parser auto`** — it auto-detects the right parser from the chat template, and SGLang then strips the tags and returns the trace separately in `message.reasoning_content`.
|
||||
- **Native tool calling**: a custom namespace-token XML format, parsed into standard OpenAI `tool_calls`. Always launch with **`--tool-call-parser auto`** — it auto-detects the right parser from the chat template. Single, parallel, and nested (object / array) arguments are supported.
|
||||
- **Sparse attention**: most layers use M3's "lightning indexer" block-sparse attention (top-k 128-token blocks), which keeps decode cost roughly flat in context length. On Blackwell, MiniMax's open-source [MSA kernel](https://github.com/MiniMax-AI/MSA) accelerates this path further (§2.1).
|
||||
- **MXFP8 quantization across vendors**: the MXFP8 MoE weights run natively on NVIDIA Blackwell (B200 / B300 / GB200 / GB300) and on AMD Instinct MI350X/MI355X (gfx950 / CDNA4), both of which have hardware MX-scaled matmul. On AMD MI300X/MI325X (gfx942 / CDNA3) — no hardware MX — SGLang converts the weights to block-fp8 `[128,128]` at load and serves them on the tuned ROCm kernels (§2.3). The vision tower stays unquantized.
|
||||
|
||||
**Recommended generation**: the model's `generation_config.json` sets `temperature` 1.0 / `top_p` 0.95, which SGLang applies automatically (the default `--sampling-defaults model`). The model card additionally suggests `top_k` 40, but that value is **not** in `generation_config.json`, so SGLang does not apply it by default. `top_k` is a per-request sampling parameter (not a launch flag) — set it per call if you want it, e.g. `extra_body={"top_k": 40}` with the OpenAI client.
|
||||
|
||||
**Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) · [MSA kernel](https://github.com/MiniMax-AI/MSA)
|
||||
|
||||
## 2. Configuration Tips
|
||||
|
||||
### 2.1 MSA sparse-attention fast path (recommended for Blackwell users)
|
||||
|
||||
[MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. **It ships pre-installed in the M3 dev image** (`lmsysorg/sglang:dev-minimax-m3`, also published under the `dev-cu13-minimax-m3` tag), so the Blackwell recipe above engages it automatically with no extra setup — `import fmha_sm100` works out of the box and the kernels JIT-compile on first use. It is otherwise purely additive: on a custom image, install it (below) and the recipe engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops ~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs.
|
||||
|
||||
**Requirements** (from the [MSA README](https://github.com/MiniMax-AI/MSA#requirements)):
|
||||
|
||||
- **GPU**: NVIDIA SM100 family — sm_100 (B200 / GB200) and sm_103 (B300 / GB300).
|
||||
- **Toolchain**: CUDA Toolkit with `nvcc` ≥ 12.x on `PATH` (or `CUDA_HOME` set) — the kernels are JIT-compiled at first import.
|
||||
- **Python**: ≥ 3.10; **OS**: Linux — works on both **x86_64 and aarch64 (Grace, e.g. GB200 / GB300)**; the aarch64 build needs no source edits.
|
||||
|
||||
<Accordion title="Install MSA (only on a custom image) & verify the gate (Python)">
|
||||
|
||||
The M3 Blackwell dev images above already bundle MSA, so you can skip straight to the gate check. The `git clone` / `pip install` steps are only needed on a custom image that doesn't have `fmha_sm100`.
|
||||
|
||||
```bash Command
|
||||
# Only on a custom image: --recursive pulls the CUTLASS submodule required for JIT compilation
|
||||
git clone --recursive https://github.com/MiniMax-AI/MSA.git msa
|
||||
cd msa && pip install .
|
||||
# Verify the SGLang gate (True -> MSA engaged on this device; False -> Triton fallback):
|
||||
python -c "from sglang.srt.layers.attention.minimax_sparse_ops.msa import msa_available; print(msa_available())"
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
The first import JIT-compiles the kernels, which can take 30 s to a few minutes on a cold `nvcc` cache — this is normal, not a hang. Subsequent server starts hit the JIT cache.
|
||||
</Note>
|
||||
|
||||
<Warning>
|
||||
**Warm the JIT cache before a multi-GPU launch.** On a *cold* cache, several tensor-parallel ranks racing to JIT-compile MSA's plan kernel can leave one rank loading a half-linked module (`AttributeError: Module has no function 'plan'` at CUDA-graph capture). Run the gate-check `python -c "..."` (or any single-process `fmha_sm100_plan` call) once before launching the server — that compiles the kernel single-process, and every rank then hits the warm cache.
|
||||
</Warning>
|
||||
|
||||
The gate requires `--attention-backend fa4` (MSA's sparse blocks are 128 tokens, so the page size must be 128). SGLang auto-forces `page_size` to 128 for the `fa4` backend — including the combined `--attention-backend fa4` the M3 recipe uses (#28976) — so `--page-size 128` is omitted from the Blackwell cells below. Force the Triton path at any time with the env var `SGLANG_DISABLE_MSA=1`. MSA is a Blackwell (SM100) kernel and does not apply to the AMD ROCm paths.
|
||||
|
||||
<Note>
|
||||
For multimodal (image) serving, keep the same text recipe above — `--attention-backend fa4` (MSA) is unchanged — and add `--mm-attention-backend flashinfer_cudnn` for the vision tower. The text and vision-tower attention backends are independent knobs; MSA only touches the language-model sparse attention, not image handling.
|
||||
</Note>
|
||||
|
||||
### 2.2 Memory and workload tuning
|
||||
|
||||
The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and **B300 / GB300 at `--tp 4`** (4-GPU is also the GB200 / GB300 single-node ceiling). GB200 (sm_100, aarch64) is inferred-supported — both of its axes are validated above (B200 is sm_100; GB300 is sm_103 aarch64) — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**.
|
||||
|
||||
- **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context.
|
||||
- **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated.
|
||||
- **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom.
|
||||
- **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy.
|
||||
- `--trust-remote-code` is required to load the MiniMax config / processor classes.
|
||||
|
||||
### 2.3 AMD Instinct (ROCm)
|
||||
|
||||
MiniMax-M3 runs on AMD Instinct GPUs through two code paths, by architecture — both selected automatically; you still pass `--quantization mxfp8` either way:
|
||||
|
||||
- **MI350X / MI355X (gfx950, CDNA4)** has hardware MX-scaled matmul, so the **MXFP8 weights are served natively**. SGLang auto-detects the checkpoint, selects the Triton MiniMax-M3 MoE path with the packaged tuned MXFP8 configs, and enables AITER fused all-reduce for single-node tensor parallelism. The launch command is the NVIDIA recipe minus the Blackwell-only backend flags.
|
||||
- **MI300X / MI325X (gfx942, CDNA3)** has **no** hardware MX matmul. SGLang transparently **converts the MXFP8 weights to block-fp8 `[128,128]` at load time**, then serves them with the tuned ROCm block-fp8 kernels (`--attention-backend aiter`, `--moe-runner-backend triton`; the `aiter` runner also works and scores marginally higher). On a cold start the first generation can JIT-compile AITER configs and exceed the default warmup/HTTP timeout, so the recipe adds `--watchdog-timeout 3600 --skip-server-warmup`. The block-fp8 step adds only a small relative error over MXFP8's native `1×32` scaling — negligible on GSM8K (see the benchmark card).
|
||||
|
||||
Select an MI300X/MI325X or MI350X/MI355X tile in the command panel above to get the exact launch command for each path.
|
||||
|
||||
<Note>
|
||||
The AMD recipes are validated end-to-end on **text** workloads — chat, reasoning separation, and tool calling. The vision tower was not exercised on ROCm; for image input on AMD, omit the Blackwell `--mm-attention-backend flashinfer_cudnn` flag and let the encoder use the ROCm default backend, and treat vision as unvalidated on that path.
|
||||
</Note>
|
||||
|
||||
### 2.4 Serving on Hopper (H200) with the bf16 build
|
||||
|
||||
The MXFP8 kernels are Blackwell-only, so Hopper (H200) serves the full-precision bfloat16 build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3). Select **H200 + BF16** in the Deploy panel above for the exact command — it runs at `--tp 8` (the bf16 weights need a full 8-GPU node). SGLang picks the right backends for Hopper automatically, so the recipe stays minimal:
|
||||
|
||||
- **MoE runner**: Triton, auto-selected for bf16 weights.
|
||||
- **Attention**: FlashAttention-3 with page size 1. MSA (§2.1) is a Blackwell kernel, so M3's sparse step runs on the built-in Triton path here.
|
||||
- **CUDA graph**: on, with full decode-graph capture.
|
||||
|
||||
**High-concurrency throughput (optional).** On Hopper the sparse prefill runs on the Triton path as a separate eager forward, which briefly stalls the in-flight decode batch under heavy concurrent load. Adding `--enable-mixed-chunk --chunked-prefill-size 2048` merges the running decodes into the prefill step instead of preempting them, which recovers roughly **+10% output throughput** and **~10% lower median TPOT** at high concurrency on 8×H200, with no change in accuracy. Leave it off for latency-sensitive low-concurrency serving.
|
||||
|
||||
Validated on 8×H200 — reasoning and tool-call auto-detection plus long-context generation. For prefill/decode disaggregation on Hopper, see §3.4.
|
||||
|
||||
## 3. Advanced Usage
|
||||
|
||||
### 3.1 Reasoning
|
||||
|
||||
Launch with `--reasoning-parser auto` (or toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)). The `<mm:think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.
|
||||
|
||||
<Accordion title="Reasoning Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
print("=============== Reasoning ===============")
|
||||
print(message.reasoning_content)
|
||||
print("=============== Answer ==================")
|
||||
print(message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
=============== Reasoning ===============
|
||||
15% of 240. 15% = 0.15. 240 * 0.15 = 36. Quick check: 10% is 24, 5% is 12, 24 + 12 = 36.
|
||||
=============== Answer ==================
|
||||
15% of 240 is **36**.
|
||||
(10% of 240 = 24, and 5% of 240 = 12; 24 + 12 = 36.)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
When streaming, the trace arrives on `delta.reasoning_content` and the answer on `delta.content`, so the two sections can be rendered separately in real time:
|
||||
|
||||
<Accordion title="Streaming Reasoning (Python)">
|
||||
|
||||
```python Example
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "Solve step by step: what is 15% of 240?"}],
|
||||
max_tokens=2048,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if not chunk.choices:
|
||||
continue
|
||||
delta = chunk.choices[0].delta
|
||||
if getattr(delta, "reasoning_content", None):
|
||||
print(delta.reasoning_content, end="", flush=True) # thinking stream
|
||||
if delta.content:
|
||||
print(delta.content, end="", flush=True) # answer stream
|
||||
print()
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
[delta.reasoning_content — thinking stream]
|
||||
Let me solve this step by step.
|
||||
|
||||
15% of 240
|
||||
= 0.15 × 240
|
||||
= 36
|
||||
|
||||
Let me verify: 10% of 240 = 24, 5% of 240 = 12, so 15% = 24 + 12 = 36. ✓
|
||||
|
||||
[delta.content — answer stream]
|
||||
# Solving 15% of 240
|
||||
## Step 1: Convert the percentage to a decimal
|
||||
15% = 15/100 = 0.15
|
||||
## Step 2: Multiply by 240
|
||||
0.15 × 240 = 36
|
||||
## Answer
|
||||
**15% of 240 = 36**
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.2 Tool Calling
|
||||
|
||||
Launch with `--tool-call-parser auto` (or toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) — it auto-detects M3's tool-call parser from the chat template. M3 emits tool calls in a custom namespace-token XML format:
|
||||
|
||||
```text Raw model output
|
||||
]<]minimax[>[<tool_call>
|
||||
]<]minimax[>[<invoke name="get_weather">]<]minimax[>[<location>Beijing]<]minimax[>[</location>]<]minimax[>[</invoke>
|
||||
]<]minimax[>[</tool_call>
|
||||
```
|
||||
|
||||
The parser converts that into the standard OpenAI `tool_calls` structure:
|
||||
|
||||
<Accordion title="Tool Calling Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"location": {"type": "string", "description": "The city name"},
|
||||
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
|
||||
},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
|
||||
tools=tools,
|
||||
)
|
||||
|
||||
message = response.choices[0].message
|
||||
if message.tool_calls:
|
||||
for call in message.tool_calls:
|
||||
print(f"Tool: {call.function.name}")
|
||||
print(f"Args: {call.function.arguments}")
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output">
|
||||
|
||||
```text Output
|
||||
Tool: get_weather
|
||||
Args: {"location": "Beijing"}
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
Beyond a single flat call, the parser also supports:
|
||||
|
||||
- **Parallel calls** — multiple `<invoke>` blocks inside the single `<tool_call>` wrapper, surfaced as multiple `message.tool_calls` entries.
|
||||
- **Nested object arguments** — an `object`-typed parameter is emitted as nested XML tags and reconstructed into a JSON object.
|
||||
- **Array arguments** — an `array`-typed parameter uses repeated `<item>` children and is reconstructed into a JSON list.
|
||||
|
||||
For example, a tool with object and array parameters round-trips cleanly:
|
||||
|
||||
```text Output
|
||||
create_event {"title": "Design sync", "attendees": ["alice", "bob"], "location": {"room": "R2", "floor": 3}}
|
||||
```
|
||||
|
||||
To return a tool result, append the assistant's `tool_calls` turn plus a matching `tool` message and ask the model to continue — the follow-up answer may place text in `reasoning_content` as well as `content`, so print both.
|
||||
|
||||
### 3.3 Multimodal (Vision) Input
|
||||
|
||||
Images go through the standard OpenAI `image_url` content type. The vision tower is always loaded; for image serving add `--mm-attention-backend flashinfer_cudnn` (the vision-tower backend) to the Blackwell deployment recipe — the text `--attention-backend` is unchanged (§2.1 note). On AMD, omit `--mm-attention-backend` and let the encoder use the ROCm default (vision is unvalidated on ROCm — §2.3).
|
||||
|
||||
<Accordion title="Vision Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
|
||||
},
|
||||
},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
],
|
||||
}
|
||||
],
|
||||
max_tokens=1024,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
This image captures a striking and unusual urban scene on what appears to be a busy New York City street.
|
||||
|
||||
**Main Subject:**
|
||||
A man stands on the rear bumper of a yellow taxi cab (an SUV-style cab, likely a Ford Escape hybrid), operating a full-sized ironing board set up across the back of the vehicle. He is wearing a bright yellow long-sleeved shirt and dark pants, and is actively ironing a blue garment, holding an iron in his right hand.
|
||||
|
||||
**Vehicles:**
|
||||
- The yellow SUV taxi on the right is stationary, its rear hatch serving as the ironing platform.
|
||||
- A second yellow taxi (a sedan) drives past on the left, captured with motion blur.
|
||||
|
||||
**Setting:**
|
||||
Tall city buildings with classic urban architecture, an American flag, and white lane markings — a bustling downtown area, possibly Midtown Manhattan.
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
Notes:
|
||||
|
||||
- If the server cannot fetch external URLs, embed the image as a base64 `data:image/png;base64,...` URI — SGLang decodes it server-side.
|
||||
- Multiple images per message are supported; add more `image_url` entries to the `content` list.
|
||||
- Reasoning and tool calling work the same way for multimodal requests — a vision prompt can still produce a `<mm:think>` trace and/or tool calls.
|
||||
|
||||
### 3.4 Prefill-Decode (PD) Disaggregation
|
||||
|
||||
[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. M3 needs one thing beyond a dense model: alongside the main KV cache, every sparse "lightning-indexer" layer keeps a **K-only index buffer**, and that buffer must reach the decode server too — otherwise sparse attention reads stale state. SGLang transfers it alongside the main KV — reusing the same page mapping — so M3 disaggregates correctly with no extra flags.
|
||||
|
||||
**Supported topology** (the released MiniMax-M3, whose sparse layers are all K-only):
|
||||
|
||||
- **Equal tensor parallelism** — the prefill and decode servers run the same `--tp`.
|
||||
- **Single pipeline stage** — PP = 1 (the default).
|
||||
- **mooncake or NIXL** transfer backend over RDMA / InfiniBand.
|
||||
|
||||
Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Pick your hardware:
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Blackwell · MXFP8">
|
||||
|
||||
On Blackwell the MXFP8 recipe — fa4, page size 128, deep_gemm MoE, and the MSA fast path (§2.1) — is auto-selected, so each role adds only the `--disaggregation-*` flags. This is the validated **2 × 4×B200** setup (TP4 prefill on node A, TP4 decode on node B); point `--disaggregation-ib-device` at your RDMA NIC(s).
|
||||
|
||||
```bash Prefill server (node A)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 4 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-ib-device mlx5_0 \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--disaggregation-bootstrap-port 8998
|
||||
```
|
||||
|
||||
```bash Decode server (node B)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 4 \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend nixl \
|
||||
--disaggregation-ib-device mlx5_0 \
|
||||
--host 0.0.0.0 --port 30001
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Hopper · bf16">
|
||||
|
||||
On Hopper (H200) M3 runs the bf16 build (§2.4) with Triton MoE and the built-in Triton sparse path, pinned to `--page-size 128` so both roles share the page layout the sparse-index transfer relies on. This is the validated **2 × 8×H200** setup (TP8 each).
|
||||
|
||||
```bash Prefill server (node A)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 8 \
|
||||
--attention-backend triton \
|
||||
--moe-runner-backend triton \
|
||||
--page-size 128 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30000 \
|
||||
--disaggregation-bootstrap-port 8998
|
||||
```
|
||||
|
||||
```bash Decode server (node B)
|
||||
sglang serve \
|
||||
--model-path MiniMaxAI/MiniMax-M3 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser auto \
|
||||
--tool-call-parser auto \
|
||||
--tp 8 \
|
||||
--attention-backend triton \
|
||||
--moe-runner-backend triton \
|
||||
--page-size 128 \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend mooncake \
|
||||
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
|
||||
--host 0.0.0.0 --port 30001
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint:
|
||||
|
||||
```bash PD router
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--prefill http://<prefill-host>:30000 8998 \
|
||||
--decode http://<decode-host>:30001 \
|
||||
--policy round_robin \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
Clients hit the router exactly like a single server — it splits each request across the two stages transparently:
|
||||
|
||||
<Accordion title="PD Client Example (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://<router-host>:8000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3-MXFP8",
|
||||
messages=[{"role": "user", "content": "What is 2 + 2?"}],
|
||||
max_tokens=64,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
**Output Example:**
|
||||
|
||||
```text Output
|
||||
2 + 2 = 4
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
**Validation.** PD disaggregation preserves output quality — the K-only sparse index transfers arrive intact and disaggregated output matches non-disaggregated serving. GSM8K is scored with the single sgl-eval harness used by the benchmark card above (full 1319-question split, chat with `--thinking`); see that card for per-platform single-node accuracy.
|
||||
|
||||
- **2 × 4×B200** (TP4+TP4, MXFP8, NIXL over InfiniBand) — output matches single-node serving. The 2-node PD serving benchmark (512-token input, 256-token output, 16 concurrent — a different workload from the card's single-node `random` isl=2048 / osl=256 / conc=64 row, so the throughput figures are not directly comparable) measured mean TTFT 1.1 s and TPOT 16.6 ms (≈ 60 tok/s per stream, ≈ 2.3k tok/s aggregate).
|
||||
- **2 × 8×H200** (TP8+TP8, bf16, mooncake) — output matches single-node serving.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user