minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Deps] Retire the CUDA 12 lane (#38404)

Browse Source 
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
This commit is contained in:
Mohammad Miadh Angkad
2026-09-10 16:58:09 -07:00
committed by GitHub
co-authored by Mohammad Angkad
parent fae8cd84cb
commit 52c191da52
38 changed files with 170 additions and 737 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx
+1 -1
View File
@@ -39,7 +39,7 @@ Then run the **Python** output of the command panel below in that environment. T
docker pull lmsysorg/sglang:dev-cu13-minimax-m3
```
The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces.
The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200; final build, the CUDA 12 lane is retired), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces.
<Note>
These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path.
docs/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx
+2 -1
View File
@@ -32,7 +32,8 @@ Pull the nightly Docker image (rolling tag, tracks `main`):
# CUDA 13 (Hopper / Blackwell, default)
docker pull lmsysorg/sglang:dev
# CUDA 12 (Ampere or older drivers)
# CUDA 12 (Ampere or older drivers). Final build: the CUDA 12 lane is retired,
# so this tag no longer tracks main.
docker pull lmsysorg/sglang:dev-cu12
```
docs/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx
+2 -2
View File
@@ -35,7 +35,7 @@ There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the
```bash Command
docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13
docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12
docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired)
docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121)
docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X
```
@@ -309,4 +309,4 @@ To try it, select the **Long Context** strategy in the Deploy panel above for an
The **DSpark** deploy strategy is the second speculative-decoding path for Inkling-Small. Unlike **MTP**, which drives Inkling-Small's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-Small-DSpark` — served unquantized alongside the NVFP4 target.
DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).
DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).
docs/cookbook/autoregressive/ThinkingMachines/Inkling.mdx
+2 -2
View File
@@ -35,7 +35,7 @@ There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the
```bash Command
docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13
docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12
docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired)
docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X
```
@@ -306,4 +306,4 @@ To try it, select the **Long Context** strategy in the Deploy panel above for an
The **DSpark** deploy strategy is the second speculative-decoding path for Inkling. Unlike **MTP**, which drives Inkling's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-DSpark-Preview` — served unquantized alongside the NVFP4 target.
DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).
DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).
docs/docs/developer_guide/contribution_guide.mdx
+4 -8
View File
@@ -206,9 +206,9 @@ Develop SGLang's customized DeepGEMM package on the [`dev` branch of `sgl-projec
### Update sgl-deep-ep
Develop `sgl-deep-ep` in [`sgl-project/DeepEP`](https://github.com/sgl-project/DeepEP). Use the implementation branch for the target platform: `sgl-deepep` for CUDA 13 on x86_64 or aarch64, `sgl-deepep-cu12-x86` for CUDA 12.9 on x86_64, or `sgl-deepep-cu12-arm` for CUDA 12.9 on aarch64. Merge packaging changes into `sgl-deepep-packaging`. The [`sgl-deep-ep` README](https://github.com/sgl-project/DeepEP/blob/sgl-deepep-packaging/sgl_deep_ep/README.md) describes the platform prerequisites and release matrix.
Develop `sgl-deep-ep` in [`sgl-project/DeepEP`](https://github.com/sgl-project/DeepEP). Use `sgl-deepep`, the implementation branch for CUDA 13 on x86_64 or aarch64. Merge packaging changes into `sgl-deepep-packaging`. The [`sgl-deep-ep` README](https://github.com/sgl-project/DeepEP/blob/sgl-deepep-packaging/sgl_deep_ep/README.md) describes the platform prerequisites and release matrix.
To validate locally, check out the selected implementation branch as `DeepEP-source` and the packaging branch as `DeepEP-packaging`. Install the required build dependencies first; CUDA 12.9 builds also require GDRCopy. The following CUDA 13 example builds a wheel for the host architecture, installs that exact wheel, and verifies that its guarded package import succeeds:
To validate locally, check out the selected implementation branch as `DeepEP-source` and the packaging branch as `DeepEP-packaging`. Install the required build dependencies first. The following example builds a wheel for the host architecture, installs that exact wheel, and verifies that its guarded package import succeeds:
```bash
DEEPEP_OUTPUT_DIR="$(mktemp -d)"
@@ -223,17 +223,13 @@ python3 -m pip install --force-reinstall --no-deps \
python3 -c "import deep_ep; print(deep_ep.__file__)"
```
Use `12.9` instead of `13.0` for a CUDA 12.9 build. The import check validates packaging and binary loading, but not communication correctness. On a configured multi-GPU host, also run the test appropriate for the implementation branch:
The import check validates packaging and binary loading, but not communication correctness. On a configured multi-GPU host, also run:
```bash
# CUDA 13 implementation branch
python3 DeepEP-source/tests/elastic/test_ep.py --num-processes 8
# CUDA 12.9 implementation branches
python3 DeepEP-source/tests/test_intranode.py --num-processes 8
```
Adjust `--num-processes` to the available GPUs and run the internode or low-latency tests when those transports changed. After local validation, ask the SGLang team to run the [sgl-deep-ep release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepep.yml) with the new version, CUDA target, and packaging ref. After verifying the published wheels for the supported Python versions and architectures, update the `sgl-deep-ep` pin in `python/pyproject.toml` before landing dependent SGLang changes.
Adjust `--num-processes` to the available GPUs and run the internode or low-latency tests when those transports changed. After local validation, ask the SGLang team to run the [sgl-deep-ep release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepep.yml) with the new version and packaging ref. After verifying the published wheels for the supported Python versions and architectures, update the `sgl-deep-ep` pin in `python/pyproject.toml` before landing dependent SGLang changes.
## Tips for newcomers
docs/docs/get-started/install.mdx
+6 -17
View File
@@ -29,15 +29,11 @@ uv pip install --prerelease=allow sglang
Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op.
</Note>
The major version of Cuda is 13 by default. To install sglang under Cuda 12 with pip or uv, please try the following commands:
```bash Command
pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang
uv pip install --force-reinstall torch==2.13.0 torchaudio==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu129
uv pip install --force-reinstall sglang-kernel --index-url https://docs.sglang.ai/whl/cu129/
uv pip install --force-reinstall sgl-deep-gemm --index-url https://docs.sglang.ai/whl/cu129/ --no-deps
```
<Note>
SGLang requires CUDA 13. The CUDA 12 (`cu129`) wheels and images are retired,
because PyTorch 2.14 publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last
release with a CUDA 12 lane.
</Note>
### Nightly builds
@@ -49,13 +45,6 @@ pip install uv
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
```
To install a nightly build under Cuda 12, swap the index to `cu129`:
```bash Command
pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu129/ sglang
```
### Quick fixes to common problems
- If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions:
1. Use `export CUDA_HOME=/usr/local/cuda-<your-cuda-version>` to set the `CUDA_HOME` environment variable.
@@ -113,7 +102,7 @@ docker run --gpus all \
You can also find the nightly docker images [here](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
Notes:
- SGLang is shipped with CUDA 13 environment by default. To run SGLang on CUDA 12 environment, please use images with `-cu12` or `-cu129` suffix, such as `lmsysorg/sglang:latest-cu129` or `lmsysorg/sglang:dev-cu12`.
- SGLang images ship a CUDA 13 environment. The `-cu12` / `-cu129` images are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag.
## Method 4: Using Kubernetes
docs/src/snippets/configs/MiniMaxAI/minimax-m3.jsx
+4 -3
View File
@@ -93,9 +93,10 @@ sgl-eval run mmmu_pro \\
dockerImages: {
// M3-specific dev images (multi-arch amd64+arm64). cu13 carries the sm_103
// (B300/GB300) + Grace arm64 builds; cu12 is the Hopper/CUDA-12 build;
// dev-minimax-m3 is the rolling default. M3 model support is not yet in a
// tagged release, so :latest cannot serve it.
// (B300/GB300) + Grace arm64 builds; cu12 is the final Hopper/CUDA-12 build
// (that lane is retired, so it no longer rebuilds); dev-minimax-m3 is the
// rolling default. M3 model support is not yet in a tagged release, so
// :latest cannot serve it.
b200: "lmsysorg/sglang:dev-minimax-m3",
b300: "lmsysorg/sglang:dev-cu13-minimax-m3",
gb200: "lmsysorg/sglang:dev-cu13-minimax-m3",
docs/src/snippets/configs/thinkingmachines/inkling-small.jsx
+2 -1
View File
@@ -70,7 +70,8 @@ export const config = {
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
// NVIDIA: two multi-arch CUDA builds (dev-inkling-dspark for CUDA 13,
// dev-cu12-inkling-dspark for CUDA 12) — pick by your CUDA version, not by GPU.
// dev-cu12-inkling-dspark for CUDA 12, a final build since that lane is
// retired) — pick by your CUDA version, not by GPU.
// Panel defaults to cu13. AMD: dev-rocm720-mi35x-inkling-dspark (sglang-rocm repo).
// DGX Spark uses a dedicated arm64 CUDA 13 image with NCCL 2.30.7.
// All tiers ship from the same images, DSpark included (except DGX Spark).
docs/src/snippets/configs/thinkingmachines/inkling.jsx
+2 -1
View File
@@ -73,7 +73,8 @@ export const config = {
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Hello"}] }'`,
// NVIDIA: two multi-arch CUDA builds (dev-inkling-dspark for CUDA 13,
// dev-cu12-inkling-dspark for CUDA 12) — pick by your CUDA version, not by GPU.
// dev-cu12-inkling-dspark for CUDA 12, a final build since that lane is
// retired) — pick by your CUDA version, not by GPU.
// Panel defaults to cu13. AMD: dev-rocm720-mi35x-inkling-dspark (sglang-rocm repo).
// All tiers ship from the same images, DSpark included.
dockerImages: {