[AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 (#39273)
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai> Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com> Co-authored-by: long10024070 <long10024070@users.noreply.github.com> Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com> Co-authored-by: Kevin Mi <mikevin920@yahoo.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: sunxxuns <126995791+sunxxuns@users.noreply.github.com> Co-authored-by: Long Luong <long.luong2+personal@onemount.com>
This commit is contained in:
co-authored by
Byron Hsu
Cheng Wan
long10024070
Kevin Mi
Kevin Mi
Claude Opus 5.5
sunxxuns
Long Luong
parent
80bb3fb651
commit
f17f7705a5
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: GLM-5.3-Flash
|
||||
description: "Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, and GB300, with MTP and multimodal serving."
|
||||
description: "Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, GB300, and MI355X, with MTP and multimodal serving."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
@@ -16,7 +16,9 @@ Use an SGLang build that includes GLM-5.3-Flash support (v0.5.20 or later).
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
The deployment panel can render a complete `docker run` command for the selected hardware and options. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup.
|
||||
For the listed NVIDIA hardware, the deployment panel can render a complete `docker run` command. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup.
|
||||
|
||||
For MI355X, follow the [AMD GPU installation guide](/docs/hardware-platforms/amd_gpu) and run the generated **Python** command on four GPUs. Select FP8 or [MXFP4](https://huggingface.co/OneNexus/GLM-5.3-Flash-MXFP4); SGLang downloads the model and tokenizer automatically.
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -27,9 +29,9 @@ Choose your hardware, then choose the operating point that matches your workload
|
||||
|
||||
Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
|
||||
|
||||
The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, Breakable Cuda Graph, and HiCache tiers. Changing an option that was not part of the measured command changes the badge to **Not Verified** without hiding the option.
|
||||
The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, Breakable Cuda Graph, and HiCache tiers. The MI355X commands are marked **Not Verified** because the selector defaults to HiCache off, whereas the measured AgentX configuration used a host tier, and the merged source has not been remeasured. Do not treat the CUDA benchmark rows as MI355X measurements.
|
||||
|
||||
**Breakable Cuda Graph** defaults to **Off**. Select **On** to add `--cuda-graph-backend-prefill breakable` to the generated command. This requires a build that includes [PR #38522](https://github.com/sgl-project/sglang/pull/38522).
|
||||
**Breakable Cuda Graph** defaults to **Off**. On NVIDIA, select **On** to add `--cuda-graph-backend-prefill breakable` to the generated command. This requires a build that includes [PR #38522](https://github.com/sgl-project/sglang/pull/38522); the option is disabled on MI355X.
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/zai-org/glm-5.3-flash.jsx";
|
||||
@@ -71,7 +73,7 @@ GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a h
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}>Precision</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 weights (<a href="https://huggingface.co/zai-org/GLM-5.3-Flash"><code>zai-org/GLM-5.3-Flash</code></a>); NVFP4 on Blackwell from <a href="https://huggingface.co/RadixArk/GLM-5.3-Flash-NVFP4"><code>RadixArk/GLM-5.3-Flash-NVFP4</code></a> and <a href="https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"><code>nvidia/GLM-5.3-Flash-NVFP4</code></a>; FP8 KV cache by default on Blackwell, BF16 KV cache on H100 and H200</td>
|
||||
<td style={{padding: "9px 12px"}}>FP8 weights (<a href="https://huggingface.co/zai-org/GLM-5.3-Flash"><code>zai-org/GLM-5.3-Flash</code></a>); MXFP4 on MI355X (<a href="https://huggingface.co/OneNexus/GLM-5.3-Flash-MXFP4"><code>OneNexus/GLM-5.3-Flash-MXFP4</code></a>); NVFP4 on Blackwell from <a href="https://huggingface.co/RadixArk/GLM-5.3-Flash-NVFP4"><code>RadixArk/GLM-5.3-Flash-NVFP4</code></a> and <a href="https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"><code>nvidia/GLM-5.3-Flash-NVFP4</code></a>; FP8 KV cache by default on Blackwell and MI355X, BF16 KV cache on H100 and H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Context</td>
|
||||
@@ -88,7 +90,7 @@ GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a h
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The deployment recipes use the checkpoint's generation configuration. Override sampling only when your application has its own evaluated settings.
|
||||
The deployment recipes inherit the checkpoint's 1,048,576-token context limit and generation configuration. Override sampling only when your application has its own evaluated settings.
|
||||
|
||||
## 2. Configuration tips
|
||||
|
||||
@@ -96,7 +98,7 @@ The deployment recipes use the checkpoint's generation configuration. Override s
|
||||
|
||||
Start with **Low Latency** for chat and agent workloads. It drafts from the checkpoint's MTP head at a fixed depth (5 steps, top-k 1, 6 draft tokens) with natural acceptance. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value.
|
||||
|
||||
Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes.
|
||||
Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes. On MI355X, the measured AgentX frontier used Low Latency (EAGLE 5/1/6); High Throughput is an unmeasured speculation-off starting point.
|
||||
|
||||
### Change the speculative algorithm
|
||||
|
||||
@@ -106,7 +108,7 @@ The **Speculative** card in the Playground changes the algorithm without leaving
|
||||
- **Off (greedy)** strips the whole `--speculative-*` family, which is what High Throughput already starts from.
|
||||
- **DFlash2** swaps the in-checkpoint MTP head for the trained block-diffusion draft in [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2). The draft proposes a whole block per step and the target verifies it in one forward pass, so output quality stays the target's. Its block size comes from the draft checkpoint, and the draft runs on `fa4` rather than the target's DSA backends. The hidden-state capture it needs ([PR #36708](https://github.com/sgl-project/sglang/pull/36708)) shipped with the GLM-5.3-Flash support in v0.5.20, so the image pinned above is enough. The draft repository is access-gated: request access on its model page, then download it alongside the target before serving. This combination is not yet measured on the cookbook hardware, so treat it as a starting point.
|
||||
|
||||
Neither algorithm runs with DP-Attention; the card disables the affected chips and names the reason.
|
||||
Neither algorithm runs with DP-Attention; the card disables the affected chips and names the reason. DFlash2's FA4 draft path is also disabled on MI355X.
|
||||
|
||||
### Size both memory pools
|
||||
|
||||
@@ -118,7 +120,7 @@ Keep the checkpoint's KDA lower-bound setting unchanged. In particular, do not o
|
||||
|
||||
### Keep the KV and DSA backends paired
|
||||
|
||||
On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300 this pairing measured 2.9–5.7% higher throughput and about 1.8x the KV token capacity at identical pool bytes, with GSM8K accuracy within noise of BF16. BF16 KV with TileLang DSA remains selectable in the deployment panel and is the default on H100 and H200, where FP8 KV with TRT-LLM DSA is disabled. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination.
|
||||
On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300 this pairing measured 2.9–5.7% higher throughput and about 1.8x the KV token capacity at identical pool bytes, with GSM8K accuracy within noise of BF16. BF16 KV with TileLang DSA remains selectable in the deployment panel and is the default on H100 and H200, where FP8 KV with TRT-LLM DSA is disabled. MI355X instead uses FP8 KV with TileLang DSA in the recorded ROCm recipe. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination, and TRT-LLM DSA is not the MI355X backend.
|
||||
|
||||
### Extend the cache hierarchy
|
||||
|
||||
@@ -126,9 +128,9 @@ Keep **HiCache** off when GPU memory is sufficient. Select **L1 + L2** to spill
|
||||
|
||||
### Multimodal memory
|
||||
|
||||
All strategies enable multimodal serving. The processor samples video at 2 FPS and caps video input at 240,000 visual tokens. Install `torchcodec` in the serving environment before sending video requests. For very long videos on 4x GB300, use encoder disaggregation to isolate the vision encoder's memory spikes from language decoding.
|
||||
All strategies enable multimodal serving. Image and video input need a Transformers build that ships `Glm5NextProcessor` (the MI355X checks used commit `e4052f55`, which also requires `tokenizers>=0.23.1`). With the pinned `transformers==5.12.1`, the server loads only the tokenizer and silently answers image requests from the text alone. On MI355X, text, image, and an image-bearing follow-up were qualified with that build; video was not evaluated. The processor samples video at 2 FPS and caps video input at 240,000 visual tokens. Install `torchcodec` in the serving environment before sending video requests. For very long videos on 4x GB300, use encoder disaggregation to isolate the vision encoder's memory spikes from language decoding.
|
||||
|
||||
The default multimodal feature transport is automatic, and on a single CUDA node auto resolves to CPU transport. CUDA IPC is opt-in: pass `--mm-feature-transport cuda_ipc` when lower transfer latency matters more than the GPU memory the IPC pool reserves. CUDA VMM transport applies only to multi-node GB200/GB300 systems on the MNNVL fabric, where auto selects it.
|
||||
The default multimodal feature transport is automatic, and on a single CUDA node auto resolves to CPU transport. The MI355X selection explicitly emits CPU transport, matching its image-serving check. CUDA IPC is opt-in: pass `--mm-feature-transport cuda_ipc` when lower transfer latency matters more than the GPU memory the IPC pool reserves. CUDA VMM transport applies only to multi-node GB200/GB300 systems on the MNNVL fabric, where auto selects it.
|
||||
|
||||
## 3. Advanced usage
|
||||
|
||||
|
||||
@@ -932,14 +932,13 @@ export const Playground = ({ config }) => {
|
||||
}).disabled) {
|
||||
return { flags, env };
|
||||
}
|
||||
flags = h.stripFlagsByFirstToken(flags, [
|
||||
"--speculative-algorithm", "--speculative-num-steps",
|
||||
"--speculative-eagle-topk", "--speculative-num-draft-tokens",
|
||||
"--speculative-adaptive",
|
||||
"--speculative-dspark-block-size", "--enable-linear-replayssm-spec",
|
||||
"--linear-replayssm-cache-len",
|
||||
"--speculative-ngram-max-bfs-breadth",
|
||||
]);
|
||||
// Model-specific draft precision and acceptance settings belong to the
|
||||
// old algorithm too; Off/DFlash must not inherit them from EAGLE.
|
||||
flags = flags.filter((flag) => {
|
||||
const head = flag.split(/[\s=]/)[0];
|
||||
return !head.startsWith("--speculative-") &&
|
||||
!["--enable-linear-replayssm-spec", "--linear-replayssm-cache-len"].includes(head);
|
||||
});
|
||||
const preset = (fc.options || []).find((p) => p.id === value);
|
||||
if (preset?.flags?.length) flags = h.insertBeforeTail(flags, preset.flags);
|
||||
return { flags, env };
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
export const config = {
|
||||
modelName: "GLM-5.3-Flash",
|
||||
|
||||
supportedHardware: ["gb300", "h100", "h200", "b200", "b300", "gb200"],
|
||||
supportedHardware: ["gb300", "h100", "h200", "b200", "b300", "gb200", "mi355x"],
|
||||
|
||||
matchDims: [
|
||||
{
|
||||
@@ -17,6 +17,12 @@ export const config = {
|
||||
title: "Quantization",
|
||||
options: [
|
||||
{ id: "fp8", label: "FP8" },
|
||||
{
|
||||
id: "mxfp4",
|
||||
label: "MXFP4",
|
||||
disabled: (s) => s.hw !== "mi355x",
|
||||
disableReason: "The MXFP4 recipe is qualified on MI355X only.",
|
||||
},
|
||||
{
|
||||
id: "nvfp4",
|
||||
label: "NVFP4 (RDXA)",
|
||||
@@ -34,7 +40,8 @@ export const config = {
|
||||
],
|
||||
|
||||
isRecommendedSelection(s) {
|
||||
const pairing = ["h100", "h200"].includes(s.hw) ? "bf16-tilelang" : "fp8-trtllm";
|
||||
const pairing = s.hw === "mi355x" ? "fp8-tilelang"
|
||||
: ["h100", "h200"].includes(s.hw) ? "bf16-tilelang" : "fp8-trtllm";
|
||||
return (
|
||||
s.kvDsaPair === pairing &&
|
||||
s.mmTransport === "auto" &&
|
||||
@@ -52,8 +59,8 @@ export const config = {
|
||||
{
|
||||
id: "fp8-trtllm",
|
||||
label: "FP8 + TRT-LLM",
|
||||
disabled: (s) => ["h100", "h200"].includes(s.hw),
|
||||
disableReason: "FP8 KV cache with TRT-LLM DSA is not supported on Hopper GPUs.",
|
||||
disabled: (s) => ["h100", "h200", "mi355x"].includes(s.hw),
|
||||
disableReason: "FP8 KV cache with TRT-LLM DSA is not supported on Hopper or MI355X.",
|
||||
stripPrefixes: ["--kv-cache-dtype", "--dsa-prefill-backend", "--dsa-decode-backend"],
|
||||
flags: [
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
@@ -62,6 +69,18 @@ export const config = {
|
||||
],
|
||||
hints: ["Measured on GB300: faster than BF16 + TileLang with about 1.8x the KV token capacity."],
|
||||
},
|
||||
{
|
||||
id: "fp8-tilelang",
|
||||
label: "FP8 + TileLang (ROCm)",
|
||||
disabled: (s) => s.hw !== "mi355x",
|
||||
disableReason: "This FP8 KV + TileLang DSA pairing is qualified on ROCm, not CUDA.",
|
||||
stripPrefixes: ["--kv-cache-dtype", "--dsa-prefill-backend", "--dsa-decode-backend"],
|
||||
flags: [
|
||||
"--kv-cache-dtype fp8_e4m3",
|
||||
"--dsa-prefill-backend tilelang",
|
||||
"--dsa-decode-backend tilelang",
|
||||
],
|
||||
},
|
||||
{
|
||||
id: "bf16-tilelang",
|
||||
label: "BF16 + TileLang",
|
||||
@@ -83,6 +102,8 @@ export const config = {
|
||||
{
|
||||
id: "on",
|
||||
label: "On",
|
||||
disabled: (s) => s.hw === "mi355x",
|
||||
disableReason: "The breakable prefill CUDA graph path is not qualified on ROCm.",
|
||||
flags: ["--cuda-graph-backend-prefill breakable"],
|
||||
hints: ["Enables breakable prefill CUDA graphs; requires a build with PR #38522."],
|
||||
},
|
||||
@@ -93,11 +114,13 @@ export const config = {
|
||||
title: "VLM Transport",
|
||||
default: "auto",
|
||||
options: [
|
||||
{ id: "auto", label: "Auto", subtitle: "Topology-aware" },
|
||||
{ id: "auto", label: "Auto", subtitle: "Topology-aware",
|
||||
flags: (s) => s.hw === "mi355x" ? ["--mm-feature-transport cpu"] : [] },
|
||||
{
|
||||
id: "cpu",
|
||||
label: "CPU",
|
||||
subtitle: "Save GPU memory",
|
||||
stripPrefixes: ["--mm-feature-transport"],
|
||||
flags: ["--mm-feature-transport cpu"],
|
||||
},
|
||||
],
|
||||
@@ -112,8 +135,14 @@ export const config = {
|
||||
id: "l2",
|
||||
label: "L1 + L2",
|
||||
subtitle: "Host memory",
|
||||
flags: ["--enable-hierarchical-cache", "--hicache-size 32"],
|
||||
hints: ["32 GB host tier; the default ratio can demand more host RAM than the node has free."],
|
||||
flags: (s) => s.hw === "mi355x"
|
||||
? ["--enable-hierarchical-cache", "--hicache-ratio 0.50",
|
||||
"--hicache-write-policy write_through", "--hicache-io-backend direct",
|
||||
"--hicache-mem-layout page_first_direct"]
|
||||
: ["--enable-hierarchical-cache", "--hicache-size 32"],
|
||||
hints: (s) => s.hw === "mi355x"
|
||||
? ["Host cache matches the measured 0.50-ratio, write-through MI355X recipe; check available RAM."]
|
||||
: ["32 GB host tier; the default ratio can demand more host RAM than the node has free."],
|
||||
},
|
||||
{
|
||||
id: "l3",
|
||||
@@ -156,9 +185,13 @@ export const config = {
|
||||
modelNames: {
|
||||
default: "zai-org/GLM-5.3-Flash",
|
||||
nvfp4: "RadixArk/GLM-5.3-Flash-NVFP4",
|
||||
"mi355x|mxfp4": "OneNexus/GLM-5.3-Flash-MXFP4",
|
||||
"nvfp4-nvda": "nvidia/GLM-5.3-Flash-NVFP4",
|
||||
},
|
||||
|
||||
// MI355X uses a ROCm environment; the Docker images below are NVIDIA-only.
|
||||
runModes: (s) => s.hw === "mi355x" ? ["python"] : ["python", "docker"],
|
||||
|
||||
placeholders: {
|
||||
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||
@@ -274,6 +307,13 @@ sgl-eval run gsm8k \\
|
||||
id: "deep_gemm",
|
||||
label: "DeepGemm",
|
||||
flags: ["--moe-runner-backend deep_gemm"],
|
||||
disable: [{ when: { hw: ["mi355x"] }, reason: "DeepGemm is a CUDA backend; MI355X uses AITER." }],
|
||||
},
|
||||
{
|
||||
id: "aiter",
|
||||
label: "AITER (ROCm)",
|
||||
flags: ["--moe-runner-backend aiter"],
|
||||
disable: [{ when: { hw: ["gb300", "h100", "h200", "b200", "b300", "gb200"] }, reason: "AITER is a ROCm backend." }],
|
||||
},
|
||||
],
|
||||
},
|
||||
@@ -341,6 +381,7 @@ sgl-eval run gsm8k \\
|
||||
],
|
||||
note: "⚠️ The draft checkpoint incoai/GLM-5.3-Flash-DFlash2 is access-gated: request access on its Hugging Face page, then download it alongside the target before serving.",
|
||||
disable: [
|
||||
{ when: { hw: ["mi355x"] }, reason: "The FA4 draft-attention path is CUDA-only." },
|
||||
{
|
||||
when: { dpAttnOn: [true] },
|
||||
reason: "DFLASH speculative decoding does not support DP-Attention — the server rejects the combination at startup. Turn DP-Attention off in the Attention card above.",
|
||||
@@ -353,6 +394,117 @@ sgl-eval run gsm8k \\
|
||||
},
|
||||
|
||||
cells: [
|
||||
// Four MI355X VFs, TP4/EP4. The fixed-source AgentX comparison also used
|
||||
// HiCache L2 and CPU multimodal transport. These cookbook commands are
|
||||
// supported starting points, not post-merge measurements of every overlay.
|
||||
{
|
||||
match: { hw: "mi355x", strategy: "low-latency", quant: "fp8" },
|
||||
nnodes: 1,
|
||||
verified: false,
|
||||
env: [
|
||||
"SGLANG_USE_AITER=1", "AITER_ONLINE_TUNE=0",
|
||||
"SGLANG_OPT_FUSED_KDA_VERIFY=1", "SGLANG_DSA_FUSE_TOPK=0",
|
||||
"SGLANG_ENABLE_WAR_BARRIER=1", "SGLANG_FORCE_COARSE_WAR_BARRIER=1",
|
||||
],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--trust-remote-code",
|
||||
"--quantization fp8",
|
||||
"--tp-size 4", "--ep-size 4",
|
||||
"--page-size 64",
|
||||
"--mem-fraction-static 0.82", "--max-running-requests 96",
|
||||
"--cuda-graph-max-bs-decode 96", "--min-free-slots-delay 1",
|
||||
"--chunked-prefill-size 8192", "--max-prefill-tokens 8192",
|
||||
"--prefill-decode-interval 0",
|
||||
"--dsa-prefill-backend tilelang", "--dsa-decode-backend tilelang",
|
||||
"--linear-attn-verify-backend triton", "--kv-cache-dtype fp8_e4m3",
|
||||
"--moe-runner-backend aiter",
|
||||
"--speculative-algorithm EAGLE", "--speculative-num-steps 5",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 6",
|
||||
"--speculative-attention-mode prefill",
|
||||
"--speculative-accept-threshold-single 1.0",
|
||||
"--speculative-accept-threshold-acc 1.0",
|
||||
"--speculative-moe-a2a-backend none",
|
||||
"--reasoning-parser glm45",
|
||||
"--tool-call-parser glm47", "--watchdog-timeout 1800",
|
||||
"--dist-timeout 600", "--host {{HOST_IP}}", "--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi355x", strategy: "low-latency", quant: "mxfp4" },
|
||||
nnodes: 1,
|
||||
verified: false,
|
||||
env: [
|
||||
"SGLANG_USE_AITER=1", "AITER_ONLINE_TUNE=0",
|
||||
"SGLANG_OPT_FUSED_KDA_VERIFY=1", "SGLANG_DSA_FUSE_TOPK=0",
|
||||
"SGLANG_ENABLE_WAR_BARRIER=1", "SGLANG_FORCE_COARSE_WAR_BARRIER=1",
|
||||
],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--trust-remote-code",
|
||||
"--quantization quark",
|
||||
"--tp-size 4", "--ep-size 4",
|
||||
"--page-size 64",
|
||||
"--mem-fraction-static 0.82", "--max-running-requests 96",
|
||||
"--cuda-graph-max-bs-decode 96", "--min-free-slots-delay 1",
|
||||
"--chunked-prefill-size 8192", "--max-prefill-tokens 8192",
|
||||
"--prefill-decode-interval 0",
|
||||
"--dsa-prefill-backend tilelang", "--dsa-decode-backend tilelang",
|
||||
"--linear-attn-verify-backend triton", "--kv-cache-dtype fp8_e4m3",
|
||||
"--moe-runner-backend aiter",
|
||||
"--speculative-algorithm EAGLE", "--speculative-num-steps 5",
|
||||
"--speculative-eagle-topk 1", "--speculative-num-draft-tokens 6",
|
||||
"--speculative-attention-mode prefill",
|
||||
"--speculative-accept-threshold-single 1.0",
|
||||
"--speculative-accept-threshold-acc 1.0",
|
||||
"--speculative-moe-a2a-backend none",
|
||||
"--reasoning-parser glm45",
|
||||
"--tool-call-parser glm47", "--watchdog-timeout 1800",
|
||||
"--dist-timeout 600", "--host {{HOST_IP}}", "--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi355x", strategy: "high-throughput", quant: "fp8" },
|
||||
nnodes: 1,
|
||||
verified: false,
|
||||
env: ["SGLANG_USE_AITER=1", "AITER_ONLINE_TUNE=0"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--trust-remote-code", "--quantization fp8",
|
||||
"--tp-size 4", "--ep-size 4",
|
||||
"--page-size 64", "--mem-fraction-static 0.82",
|
||||
"--max-running-requests 96", "--cuda-graph-max-bs-decode 96",
|
||||
"--min-free-slots-delay 1", "--chunked-prefill-size 8192",
|
||||
"--max-prefill-tokens 8192", "--prefill-decode-interval 0",
|
||||
"--dsa-prefill-backend tilelang", "--dsa-decode-backend tilelang",
|
||||
"--linear-attn-verify-backend triton", "--kv-cache-dtype fp8_e4m3",
|
||||
"--moe-runner-backend aiter",
|
||||
"--reasoning-parser glm45", "--tool-call-parser glm47",
|
||||
"--watchdog-timeout 1800", "--dist-timeout 600",
|
||||
"--host {{HOST_IP}}", "--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "mi355x", strategy: "high-throughput", quant: "mxfp4" },
|
||||
nnodes: 1,
|
||||
verified: false,
|
||||
env: ["SGLANG_USE_AITER=1", "AITER_ONLINE_TUNE=0"],
|
||||
flags: [
|
||||
"--model-path {{MODEL_NAME}}",
|
||||
"--trust-remote-code", "--quantization quark",
|
||||
"--tp-size 4", "--ep-size 4",
|
||||
"--page-size 64", "--mem-fraction-static 0.82",
|
||||
"--max-running-requests 96", "--cuda-graph-max-bs-decode 96",
|
||||
"--min-free-slots-delay 1", "--chunked-prefill-size 8192",
|
||||
"--max-prefill-tokens 8192", "--prefill-decode-interval 0",
|
||||
"--dsa-prefill-backend tilelang", "--dsa-decode-backend tilelang",
|
||||
"--linear-attn-verify-backend triton", "--kv-cache-dtype fp8_e4m3",
|
||||
"--moe-runner-backend aiter",
|
||||
"--reasoning-parser glm45", "--tool-call-parser glm47",
|
||||
"--watchdog-timeout 1800", "--dist-timeout 600",
|
||||
"--host {{HOST_IP}}", "--port {{PORT}}",
|
||||
],
|
||||
},
|
||||
{
|
||||
match: { hw: "gb300", strategy: "low-latency", quant: "fp8" },
|
||||
nnodes: 1,
|
||||
|
||||
@@ -31,6 +31,7 @@ namespace {
|
||||
inline constexpr int kGroupTopK = SGL_GROUP_TOPK;
|
||||
inline constexpr int kThreadsPerBlock = 1024;
|
||||
|
||||
// Keep 4K threshold-bin candidates per round; rescan larger bins without clipping.
|
||||
inline constexpr std::size_t kSmem = 8 * 1024 * sizeof(uint32_t); // 32KB (bytes)
|
||||
|
||||
struct FastTopKParams {
|
||||
@@ -53,6 +54,11 @@ __device__ __forceinline__ auto convert_to_uint32(float x) -> uint32_t {
|
||||
return (bits & 0x80000000u) ? ~bits : (bits | 0x80000000u);
|
||||
}
|
||||
|
||||
__device__ __forceinline__ auto make_topk_key(float score, int index) -> uint64_t {
|
||||
// Unique keys resolve an overfull bin even when all scores are equal.
|
||||
return (static_cast<uint64_t>(convert_to_uint32(score)) << 32) | ~static_cast<uint32_t>(index);
|
||||
}
|
||||
|
||||
template <int K>
|
||||
__device__ void
|
||||
fast_topk_cuda_tl_impl(const float* __restrict__ input, int* __restrict__ index, int row_start, int length) {
|
||||
@@ -66,6 +72,7 @@ fast_topk_cuda_tl_impl(const float* __restrict__ input, int* __restrict__ index,
|
||||
alignas(128) __shared__ int s_counter;
|
||||
alignas(128) __shared__ int s_threshold_bin_id;
|
||||
alignas(128) __shared__ int s_num_input[2];
|
||||
alignas(128) __shared__ uint64_t s_key_prefix;
|
||||
|
||||
auto& s_histogram = s_histogram_buf[0];
|
||||
extern __shared__ int s_input_idx[][SMEM_INPUT_SIZE];
|
||||
@@ -108,6 +115,7 @@ fast_topk_cuda_tl_impl(const float* __restrict__ input, int* __restrict__ index,
|
||||
|
||||
const auto threshold_bin = s_threshold_bin_id;
|
||||
topk -= s_histogram[threshold_bin + 1];
|
||||
const auto num_threshold_candidates = s_histogram[threshold_bin] - s_histogram[threshold_bin + 1];
|
||||
|
||||
if (topk == 0) {
|
||||
for (int idx = tx; idx < length; idx += BLOCK_SIZE) {
|
||||
@@ -119,6 +127,63 @@ fast_topk_cuda_tl_impl(const float* __restrict__ input, int* __restrict__ index,
|
||||
}
|
||||
__syncthreads();
|
||||
return;
|
||||
} else if (num_threshold_candidates > int(SMEM_INPUT_SIZE)) {
|
||||
// Rescan the full row only on overflow. Preserve the unsorted output contract.
|
||||
if (tx == 0) s_key_prefix = 0;
|
||||
__syncthreads();
|
||||
|
||||
#pragma unroll 8
|
||||
for (int round = 0; round < 8; ++round) {
|
||||
if (tx < RADIX + 1) s_histogram[tx] = 0;
|
||||
__syncthreads();
|
||||
const auto prefix = s_key_prefix;
|
||||
const auto offset = 56 - round * 8;
|
||||
for (int idx = tx; idx < length; idx += BLOCK_SIZE) {
|
||||
const auto raw_input = input[idx + row_start];
|
||||
if (convert_to_uint8(raw_input) != threshold_bin) continue;
|
||||
const auto key = make_topk_key(raw_input, idx);
|
||||
const bool prefix_matches = round == 0 || (key >> (64 - round * 8)) == prefix;
|
||||
if (prefix_matches) ::atomicAdd(&s_histogram[(key >> offset) & 0xFF], 1);
|
||||
}
|
||||
__syncthreads();
|
||||
run_cumsum();
|
||||
if (tx < RADIX && s_histogram[tx] > topk && s_histogram[tx + 1] <= topk) s_threshold_bin_id = tx;
|
||||
__syncthreads();
|
||||
const auto key_bin = s_threshold_bin_id;
|
||||
topk -= s_histogram[key_bin + 1];
|
||||
if (tx == 0) s_key_prefix = (prefix << 8) | static_cast<uint64_t>(key_bin);
|
||||
__syncthreads();
|
||||
|
||||
if (topk == 0) {
|
||||
const auto selected_prefix = s_key_prefix;
|
||||
const auto prefix_bits = (round + 1) * 8;
|
||||
if (tx == 0) s_counter = 0;
|
||||
__syncthreads();
|
||||
for (int idx = tx; idx < length; idx += BLOCK_SIZE) {
|
||||
const auto raw_input = input[idx + row_start];
|
||||
const auto coarse_bin = convert_to_uint8(raw_input);
|
||||
const auto key_prefix = make_topk_key(raw_input, idx) >> (64 - prefix_bits);
|
||||
if (coarse_bin > threshold_bin || (coarse_bin == threshold_bin && key_prefix > selected_prefix)) {
|
||||
const auto pos = ::atomicAdd(&s_counter, 1);
|
||||
index[pos] = idx;
|
||||
}
|
||||
}
|
||||
__syncthreads();
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const auto threshold_key = s_key_prefix;
|
||||
if (tx == 0) s_counter = 0;
|
||||
__syncthreads();
|
||||
for (int idx = tx; idx < length; idx += BLOCK_SIZE) {
|
||||
if (make_topk_key(input[idx + row_start], idx) >= threshold_key) {
|
||||
const auto pos = ::atomicAdd(&s_counter, 1);
|
||||
index[pos] = idx;
|
||||
}
|
||||
}
|
||||
__syncthreads();
|
||||
return;
|
||||
} else {
|
||||
__syncthreads();
|
||||
if (tx < RADIX + 1) {
|
||||
@@ -294,7 +359,11 @@ void setup_kernel_smem_once(host::DebugInfo where = {}) {
|
||||
[[maybe_unused]]
|
||||
static const auto result = [] {
|
||||
const auto fptr = std::bit_cast<const void*>(f);
|
||||
#if defined(__HIP_PLATFORM_AMD__)
|
||||
return ::hipFuncSetAttribute(fptr, ::hipFuncAttributeMaxDynamicSharedMemorySize, kMaxDynamicSMEM);
|
||||
#else
|
||||
return ::cudaFuncSetAttribute(fptr, ::cudaFuncAttributeMaxDynamicSharedMemorySize, kMaxDynamicSMEM);
|
||||
#endif
|
||||
}();
|
||||
host::RuntimeDeviceCheck(result, where);
|
||||
}
|
||||
@@ -326,7 +395,7 @@ struct KpoolTopKTransformKernel {
|
||||
auto S = SymbolicSize{"score_stride"};
|
||||
auto out_cols_sym = SymbolicSize{"out_cols"};
|
||||
auto device = SymbolicDevice{};
|
||||
device.set_options<kDLCUDA>();
|
||||
device.set_options<kDLGPU>();
|
||||
|
||||
TensorMatcher({B, -1}).with_strides({S, 1}).with_dtype<float>().with_device(device).verify(score);
|
||||
TensorMatcher({B}).with_dtype<int32_t>().with_device(device).verify(lengths);
|
||||
|
||||
@@ -36,12 +36,33 @@ import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.jit.utils import is_arch_support_pdl
|
||||
from sglang.kernels.jit.utils import is_arch_support_pdl, is_hip_runtime
|
||||
|
||||
# V-tile width of the fused verify kernel. Tuned on B200 at T=5 with
|
||||
# benchmark/kernels/bench_kda_verify_sweep.py; any power of two is
|
||||
# numerics-safe at num_warps=4 (bit-exact vs the BV=32 original).
|
||||
KDA_VERIFY_BLOCK_V = 4
|
||||
# gfx950 V-tile width for the measured GLM TP4 shapes: wider tiles share the q/k
|
||||
# convolution across more lanes
|
||||
KDA_VERIFY_BLOCK_V_HIP = 16
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _conv_product(x, weight, ROUND_PRODUCT: tl.constexpr):
|
||||
if ROUND_PRODUCT:
|
||||
# ROCm's unfused conv uses separately rounded fp32 products. Without
|
||||
# this boundary LLVM contracts the fused path into FMAs, which can
|
||||
# change the bf16 conv output before the recurrent update.
|
||||
return tl.inline_asm_elementwise(
|
||||
"v_mul_f32 $0, $1, $2",
|
||||
constraints="=v,v,v",
|
||||
args=[x.to(tl.float32), weight.to(tl.float32)],
|
||||
dtype=tl.float32,
|
||||
is_pure=True,
|
||||
pack=1,
|
||||
)
|
||||
else:
|
||||
return x * weight
|
||||
|
||||
|
||||
@triton.jit
|
||||
@@ -89,6 +110,7 @@ def fused_kda_conv_gating_verify_kernel(
|
||||
USE_LOWER_BOUND: tl.constexpr,
|
||||
SAVE_INTERMEDIATE_WINDOW: tl.constexpr,
|
||||
CACHE_INTERMEDIATE_STATES: tl.constexpr,
|
||||
ROUND_CONV_PRODUCTS: tl.constexpr,
|
||||
USE_GDC: tl.constexpr = False,
|
||||
# ReplaySSM fused ring-write (spec verify): per-slot rings consumed by the
|
||||
# commit-time exact fold (kda_replayssm_spec_decode.py). Off -> dead code.
|
||||
@@ -211,18 +233,18 @@ def fused_kda_conv_gating_verify_kernel(
|
||||
acc_q = tl.zeros([BK], dtype=tl.float32)
|
||||
acc_k = tl.zeros([BK], dtype=tl.float32)
|
||||
acc_v = tl.zeros([BV], dtype=tl.float32)
|
||||
acc_q += q_c0 * wq0
|
||||
acc_q += q_c1 * wq1
|
||||
acc_q += q_c2 * wq2
|
||||
acc_q += x_q * wq3
|
||||
acc_k += k_c0 * wk0
|
||||
acc_k += k_c1 * wk1
|
||||
acc_k += k_c2 * wk2
|
||||
acc_k += x_k * wk3
|
||||
acc_v += v_c0 * wv0
|
||||
acc_v += v_c1 * wv1
|
||||
acc_v += v_c2 * wv2
|
||||
acc_v += x_v * wv3
|
||||
acc_q += _conv_product(q_c0, wq0, ROUND_CONV_PRODUCTS)
|
||||
acc_q += _conv_product(q_c1, wq1, ROUND_CONV_PRODUCTS)
|
||||
acc_q += _conv_product(q_c2, wq2, ROUND_CONV_PRODUCTS)
|
||||
acc_q += _conv_product(x_q, wq3, ROUND_CONV_PRODUCTS)
|
||||
acc_k += _conv_product(k_c0, wk0, ROUND_CONV_PRODUCTS)
|
||||
acc_k += _conv_product(k_c1, wk1, ROUND_CONV_PRODUCTS)
|
||||
acc_k += _conv_product(k_c2, wk2, ROUND_CONV_PRODUCTS)
|
||||
acc_k += _conv_product(x_k, wk3, ROUND_CONV_PRODUCTS)
|
||||
acc_v += _conv_product(v_c0, wv0, ROUND_CONV_PRODUCTS)
|
||||
acc_v += _conv_product(v_c1, wv1, ROUND_CONV_PRODUCTS)
|
||||
acc_v += _conv_product(v_c2, wv2, ROUND_CONV_PRODUCTS)
|
||||
acc_v += _conv_product(x_v, wv3, ROUND_CONV_PRODUCTS)
|
||||
|
||||
# Slide the window (reference: col0=col1; col1=col2; col2=x).
|
||||
q_c0 = q_c1
|
||||
@@ -420,7 +442,8 @@ def fused_kda_conv_gating_verify(
|
||||
# T=8 safe gate. conv_state is not comparable to the reference at all.
|
||||
# The ReplaySSM ring values are bit-exact at any num_warps: they are
|
||||
# elementwise (conv FMA chain, gate, sigmoid), upstream of every tl.sum.
|
||||
num_warps: int = 4,
|
||||
# None picks 1 on ROCm (faster on gfx950 and bit-exact with the reference), else 4
|
||||
num_warps: Optional[int] = None,
|
||||
# ReplaySSM fused ring-write; same parameter names as the unfused
|
||||
# fused_sigmoid_gating_delta_rule_update so ring_kwargs pass through both.
|
||||
cache_ring: bool = False,
|
||||
@@ -435,6 +458,8 @@ def fused_kda_conv_gating_verify(
|
||||
seq_len, dim = mixed_qkv.shape
|
||||
B = seq_len // T
|
||||
W = conv_weight.shape[1]
|
||||
if num_warps is None:
|
||||
num_warps = 1 if is_hip_runtime() else 4
|
||||
|
||||
assert mixed_qkv.stride(-1) == 1, "mixed_qkv must be contiguous in dim"
|
||||
assert dim == 2 * H * K + HV * V, f"packed dim mismatch: {dim}"
|
||||
@@ -458,6 +483,16 @@ def fused_kda_conv_gating_verify(
|
||||
# the gated RMSNorm into this kernel's epilogue is a dead end; it is
|
||||
# PDL-chained behind this kernel instead (see fused_norm_gate.py).
|
||||
BV = min(triton.next_power_of_2(V), KDA_VERIFY_BLOCK_V)
|
||||
if (
|
||||
is_hip_runtime()
|
||||
and num_warps == 1
|
||||
and T in (6, 8)
|
||||
and H == HV == 16
|
||||
and K == V == 128
|
||||
and conv_weight.dtype == torch.float32
|
||||
and 3 <= B <= 16
|
||||
):
|
||||
BV = KDA_VERIFY_BLOCK_V_HIP
|
||||
NV = triton.cdiv(V, BV)
|
||||
|
||||
a2 = a.reshape(seq_len, HV * K)
|
||||
@@ -572,6 +607,7 @@ def fused_kda_conv_gating_verify(
|
||||
USE_LOWER_BOUND=lower_bound is not None,
|
||||
SAVE_INTERMEDIATE_WINDOW=intermediate_conv_window is not None,
|
||||
CACHE_INTERMEDIATE_STATES=intermediate_states_buffer is not None,
|
||||
ROUND_CONV_PRODUCTS=is_hip_runtime() and conv_weight.dtype == torch.float32,
|
||||
replayssm_rawv=replayssm_rawv,
|
||||
replayssm_rawk=replayssm_rawk,
|
||||
replayssm_g=replayssm_g,
|
||||
|
||||
@@ -1166,6 +1166,30 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
return "text", processed_messages.prompt_ids
|
||||
return "input_ids", processed_messages.prompt_ids
|
||||
|
||||
def _can_reuse_text_only_prompt_ids(
|
||||
self, processed_messages: MessageProcessingResult, is_multimodal: bool
|
||||
) -> bool:
|
||||
# Moss-VL invokes its processor for text-only requests, and that processor
|
||||
# requires the rendered text rather than pre-tokenized ids.
|
||||
is_moss_vl = (
|
||||
"MossVLForConditionalGeneration"
|
||||
in self.tokenizer_manager.model_config.hf_config.architectures
|
||||
)
|
||||
return (
|
||||
is_multimodal
|
||||
and not is_moss_vl
|
||||
and self.chat_encoding_spec is None
|
||||
and self.template_manager.chat_template_name is None
|
||||
and not self._prompt_text_round_trip_is_lossy
|
||||
and not self._tokenizer_auto_adds_specials
|
||||
and isinstance(processed_messages.prompt_ids, list)
|
||||
and bool(processed_messages.prompt_ids)
|
||||
and not processed_messages.image_data
|
||||
and not processed_messages.video_data
|
||||
and not processed_messages.audio_data
|
||||
and not processed_messages.modalities
|
||||
)
|
||||
|
||||
def _convert_to_internal_request(
|
||||
self,
|
||||
request: ChatCompletionRequest,
|
||||
@@ -1233,7 +1257,9 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
)
|
||||
|
||||
# Handle single vs multiple requests
|
||||
if request.input_ids is not None:
|
||||
if request.input_ids is not None or self._can_reuse_text_only_prompt_ids(
|
||||
processed_messages, is_multimodal
|
||||
):
|
||||
prompt_kwargs = {"input_ids": processed_messages.prompt_ids}
|
||||
else:
|
||||
prompt_key, prompt_value = self._engine_prompt(
|
||||
@@ -1337,8 +1363,12 @@ class OpenAIServingChat(OpenAIServingBase):
|
||||
tool_call_constraint = None
|
||||
|
||||
effective_tools = self._effective_tools(request)
|
||||
glm_constraint = self.tool_call_parser == "glm47" and not any(
|
||||
tool.function.strict for tool in effective_tools
|
||||
# Only tool-bearing requests get the full-assistant EBNF: its terminal
|
||||
# state finishes a request even under ignore_eos.
|
||||
glm_constraint = (
|
||||
self.tool_call_parser == "glm47"
|
||||
and bool(effective_tools)
|
||||
and not any(tool.function.strict for tool in effective_tools)
|
||||
)
|
||||
if glm_constraint:
|
||||
enable_thinking = (request.chat_template_kwargs or {}).get(
|
||||
|
||||
@@ -23,7 +23,6 @@ from sglang.srt.layers.attention.dsa.utils import (
|
||||
dsa_use_prefill_cp,
|
||||
)
|
||||
from sglang.srt.layers.attention.mqa_logits_utils import (
|
||||
MQA_LOGITS_BYTES_PER_ELEM,
|
||||
MQA_LOGITS_MAX_BYTES_ROCM,
|
||||
mqa_logits_budget_bytes,
|
||||
mqa_logits_row_bytes,
|
||||
@@ -901,7 +900,8 @@ class IndexerKPool(MultiPlatformOp):
|
||||
num_q, num_k = q_fp8.shape[0], k_fp8.shape[0]
|
||||
rows_per_call = mqa_logits_rows_per_chunk(
|
||||
num_rows=num_q,
|
||||
row_bytes=num_k * MQA_LOGITS_BYTES_PER_ELEM,
|
||||
# aiter pads each output row to 256 columns
|
||||
row_bytes=mqa_logits_row_bytes(num_k),
|
||||
budget_bytes=MQA_LOGITS_MAX_BYTES_ROCM,
|
||||
)
|
||||
if rows_per_call is None:
|
||||
|
||||
@@ -122,7 +122,7 @@ def _gather_index_k_scale_prefix_into_kernel(
|
||||
token_id = tl.program_id(0)
|
||||
page_idx = token_id // PAGE_SIZE
|
||||
token_offset_in_page = token_id % PAGE_SIZE
|
||||
page = tl.load(page_indices_ptr + page_idx)
|
||||
page = tl.load(page_indices_ptr + page_idx).to(tl.int64)
|
||||
|
||||
offs = tl.arange(0, BLOCK_D)
|
||||
mask = offs < HEAD_DIM
|
||||
|
||||
@@ -44,10 +44,19 @@ class DsaGraphVariants:
|
||||
|
||||
|
||||
def create_attention_graph_variants(hf_config) -> Optional[AttentionGraphVariants]:
|
||||
from sglang.srt.configs.model_config import get_dsa_index_topk, is_deepseek_dsa
|
||||
from sglang.srt.configs.model_config import (
|
||||
get_dsa_index_kpool,
|
||||
get_dsa_index_topk,
|
||||
is_deepseek_dsa,
|
||||
)
|
||||
from sglang.srt.utils import is_hip
|
||||
|
||||
if is_hip() and is_deepseek_dsa(hf_config):
|
||||
# KPool has no dense-skip path: both variants run the full indexer.
|
||||
# Capture it once, keeping the regular DSA dense/sparse split intact.
|
||||
if get_dsa_index_kpool(hf_config) > 1:
|
||||
return None
|
||||
|
||||
index_topk = get_dsa_index_topk(hf_config)
|
||||
logger.info(
|
||||
"[dense-decode] DSA dual-graph enabled: capturing "
|
||||
|
||||
@@ -20,7 +20,7 @@ from sglang.srt.layers.attention.linear.utils import (
|
||||
select_verify_intermediate_state_indices,
|
||||
)
|
||||
from sglang.srt.layers.radix_linear_attention import RadixLinearAttention
|
||||
from sglang.srt.utils import is_cpu, is_cuda, is_npu
|
||||
from sglang.srt.utils import is_cpu, is_cuda, is_hip, is_npu
|
||||
from sglang.srt.utils.common import is_gfx95_supported, rank0_log
|
||||
|
||||
# KDA always uses the triton causal_conv1d_fn (no CUDA override).
|
||||
@@ -1285,6 +1285,16 @@ class KDAAttnBackend(MambaAttnBackendBase):
|
||||
# other batch/architecture combinations are measured. The snapshot
|
||||
# path and the separate CuTe path are unchanged.
|
||||
return False
|
||||
if is_hip() and not (
|
||||
is_gfx95_supported()
|
||||
and 1 <= batch_size <= 16
|
||||
and draft_token_num in (6, 8)
|
||||
and mixed_qkv.dtype == torch.bfloat16
|
||||
and layer.conv_weights.dtype == torch.float32
|
||||
):
|
||||
# measured gfx950 wins only: larger batches lose the launch saving to
|
||||
# duplicated convolution work
|
||||
return False
|
||||
expected_dim = (
|
||||
2 * layer.num_q_heads * layer.head_k_dim
|
||||
+ layer.num_v_heads * layer.head_v_dim
|
||||
|
||||
@@ -952,6 +952,9 @@ class ReqKvInfo:
|
||||
mamba_last_track_idx: Optional[int] = None # 0 or 1
|
||||
# Seq len of the last cached mamba state
|
||||
mamba_last_track_seqlen: Optional[int] = None
|
||||
# Seq len of the other ping-pong slot's state. None means what is in
|
||||
# that slot cannot be named (never written, donated, freed).
|
||||
mamba_prev_track_seqlen: Optional[int] = None
|
||||
# Deferred COW: source mamba pool index from radix cache node (copy on forward stream)
|
||||
mamba_cow_src_index: Optional[torch.Tensor] = None
|
||||
# Deferred clear: newly allocated mamba slot needs zeroing on forward stream
|
||||
@@ -1475,7 +1478,14 @@ class Req(ReqDllmMixin):
|
||||
# overallocated range and are reclaimed by release_kv_cache. #22373.
|
||||
if get_serving().strip_thinking_cache and self.reasoning_tokens > 0:
|
||||
return min(self.kv.kv_committed_len, len(self.origin_input_ids))
|
||||
return self.kv.kv_committed_len
|
||||
if self.finished_len is None:
|
||||
return self.kv.kv_committed_len
|
||||
# A verify step commits a whole accepted chunk, so output_ids can run
|
||||
# past the stop; the client never sees those tokens, so nothing matches.
|
||||
return min(
|
||||
self.kv.kv_committed_len,
|
||||
len(self.origin_input_ids) + len(self.output_ids_through_stop),
|
||||
)
|
||||
|
||||
def update_spec_correct_drafts_histogram(self, num_correct_drafts: int):
|
||||
"""Record one step accepted draft count (excludes bonus token) into the histogram."""
|
||||
@@ -1988,6 +1998,7 @@ class Req(ReqDllmMixin):
|
||||
self.kv.mamba_next_track_idx = None
|
||||
self.kv.mamba_last_track_idx = None
|
||||
self.kv.mamba_last_track_seqlen = None
|
||||
self.kv.mamba_prev_track_seqlen = None
|
||||
self.mamba_branching_seqlen = None
|
||||
self.kv.mamba_cow_src_index = None
|
||||
self.kv.mamba_needs_clear = False
|
||||
@@ -2212,7 +2223,8 @@ def set_mamba_track_indices_from_reqs(
|
||||
if track_positions is None:
|
||||
# Guard: mamba_next_track_idx may be None for requests that haven't
|
||||
# gone through _alloc_ping_pong_buffer yet (e.g., spec v2 verify path).
|
||||
# Default to 0 (first ping-pong slot) to avoid TypeError.
|
||||
# Keep the gather in bounds for those requests. Freed rows are
|
||||
# invalidated below so they cannot scatter into a recycled slot.
|
||||
track_positions = [
|
||||
(
|
||||
req.kv.mamba_next_track_idx
|
||||
@@ -2221,6 +2233,9 @@ def set_mamba_track_indices_from_reqs(
|
||||
)
|
||||
for req in batch.reqs
|
||||
]
|
||||
freed_rows = [
|
||||
i for i, req in enumerate(batch.reqs) if req.kv.mamba_next_track_idx is None
|
||||
]
|
||||
batch.mamba_track_buffer_indices = list(track_positions)
|
||||
idx = (
|
||||
torch.tensor(
|
||||
@@ -2234,6 +2249,12 @@ def set_mamba_track_indices_from_reqs(
|
||||
batch.mamba_track_indices = (
|
||||
torch.gather(all_buffers, 1, idx).squeeze(1).to(torch.int64)
|
||||
)
|
||||
if freed_rows:
|
||||
# Overlap can leave a request in TARGET_VERIFY after its Mamba state has
|
||||
# been freed and reused by another request. The downstream scatter
|
||||
# treats a negative destination as a no-op, while the old fallback to
|
||||
# position 0 could overwrite the reused live slot.
|
||||
batch.mamba_track_indices[freed_rows] = -1
|
||||
|
||||
|
||||
def release_req(
|
||||
|
||||
@@ -648,7 +648,12 @@ class PrefillAdder:
|
||||
self.rem_chunk_tokens = rem_chunk_tokens
|
||||
self.chunked_req_limit: Optional[int] = None
|
||||
self.dllm_config = dllm_config
|
||||
self.exact_chunk_fill = _use_exact_chunk_fill() and dllm_config is None
|
||||
# Mamba checkpoints only land on page-aligned chunk ends.
|
||||
self.exact_chunk_fill = (
|
||||
_use_exact_chunk_fill()
|
||||
and dllm_config is None
|
||||
and not tree_cache.supports_mamba()
|
||||
)
|
||||
|
||||
if self.dllm_config is not None:
|
||||
self._init_dllm_meta(dllm_config)
|
||||
|
||||
@@ -1366,6 +1366,9 @@ class SchedulerBatchResultProcessor:
|
||||
|
||||
if completed_mamba_boundary and not lazy:
|
||||
req.kv.mamba_last_track_idx = batch.mamba_track_buffer_indices[i]
|
||||
# The slot that stops being the latest still holds its
|
||||
# checkpoint; name it so a short key can fall back to it.
|
||||
req.kv.mamba_prev_track_seqlen = req.kv.mamba_last_track_seqlen
|
||||
req.kv.mamba_last_track_seqlen = req.kv.kv_committed_len - lookahead
|
||||
elif (
|
||||
req.finished()
|
||||
@@ -1495,13 +1498,17 @@ class SchedulerBatchResultProcessor:
|
||||
track_idx = req.kv.mamba_next_track_idx
|
||||
if not known_boundary and batch.mamba_track_buffer_indices is not None:
|
||||
track_idx = batch.mamba_track_buffer_indices[i]
|
||||
previous_track_seqlen = req.kv.mamba_last_track_seqlen
|
||||
if not known_boundary:
|
||||
req.kv.mamba_last_track_seqlen = track_seqlen
|
||||
if lazy:
|
||||
# Lazy frees the slot it stops tracking, so nothing names the
|
||||
# previous checkpoint there; mamba_prev_track_seqlen stays None.
|
||||
self.mamba_lazy_post_decode_at_boundary(req, batch, track_idx)
|
||||
else:
|
||||
if not known_boundary:
|
||||
req.kv.mamba_last_track_idx = track_idx
|
||||
req.kv.mamba_prev_track_seqlen = previous_track_seqlen
|
||||
req.kv.mamba_next_track_idx = (
|
||||
batch.req_to_token_pool.get_mamba_ping_pong_other_idx(track_idx)
|
||||
)
|
||||
|
||||
@@ -81,6 +81,8 @@ class InsertParams:
|
||||
|
||||
# Mamba specific
|
||||
mamba_value: Optional[torch.Tensor] = None
|
||||
# The ping-pong slot prepare picked; cleanup keeps that same slot.
|
||||
mamba_keep_idx: Optional[int] = None
|
||||
|
||||
# DSV4 NPU C128 sidecar pages, one page id per physical C128 page group.
|
||||
c128_value: Optional[torch.Tensor] = None
|
||||
|
||||
@@ -346,7 +346,12 @@ def _release_overallocated_kv_indices(
|
||||
# strip_thinking_cache intentionally reports output tokens as overallocated
|
||||
# so they fall into the free path below (#22373).
|
||||
if spec_algo is None and not get_serving().strip_thinking_cache:
|
||||
assert start_p == end_p, (
|
||||
# A stop landing before the last committed token does the same, via
|
||||
# effective_kv_committed_len().
|
||||
assert start_p == end_p or (
|
||||
req.finished_len is not None
|
||||
and len(req.origin_input_ids) + req.finished_len < req.kv.kv_committed_len
|
||||
), (
|
||||
f"Unexpected overallocated KV cache, {req.kv.kv_committed_len=}, {req.kv.kv_allocated_len=}"
|
||||
)
|
||||
|
||||
|
||||
@@ -1620,6 +1620,7 @@ class HybridReqToTokenPool(ReqToTokenPool):
|
||||
)
|
||||
buf[:n] = slots
|
||||
req.kv.mamba_ping_pong_track_buffer = buf
|
||||
req.kv.mamba_prev_track_seqlen = None
|
||||
req.kv.mamba_next_track_idx = 0
|
||||
req.kv.mamba_last_track_idx = (
|
||||
0
|
||||
@@ -1720,6 +1721,7 @@ class HybridReqToTokenPool(ReqToTokenPool):
|
||||
req.kv.mamba_next_track_idx = None
|
||||
req.kv.mamba_last_track_idx = None
|
||||
req.kv.mamba_last_track_seqlen = None
|
||||
req.kv.mamba_prev_track_seqlen = None
|
||||
req.kv.mamba_cow_src_index = None
|
||||
req.kv.mamba_needs_clear = False
|
||||
|
||||
|
||||
@@ -536,6 +536,28 @@ class MambaComponent(TreeComponent):
|
||||
else:
|
||||
self.cache.req_to_token_pool.mamba_allocator.free(mamba_value)
|
||||
|
||||
def _select_finished_checkpoint(
|
||||
self, req: Req, token_ids_len: int
|
||||
) -> Optional[tuple[int, int]]:
|
||||
# None means donate nothing, not "no slot found".
|
||||
pool = self.cache.req_to_token_pool
|
||||
keep_idx = pool.get_mamba_ping_pong_keep_idx(req)
|
||||
cache_len = req.kv.mamba_last_track_seqlen or 0
|
||||
|
||||
if cache_len <= token_ids_len:
|
||||
return cache_len, keep_idx
|
||||
|
||||
# Overshoot: the latest state ran past the key. The other slot always
|
||||
# holds a tensor; only this seqlen says whether a key can name it.
|
||||
previous_cache_len = req.kv.mamba_prev_track_seqlen
|
||||
if (
|
||||
pool.mamba_ping_pong_track_buffer_size != 2
|
||||
or previous_cache_len is None
|
||||
or previous_cache_len > token_ids_len
|
||||
):
|
||||
return None
|
||||
return previous_cache_len, pool.get_mamba_ping_pong_other_idx(keep_idx)
|
||||
|
||||
def prepare_for_caching_req(
|
||||
self,
|
||||
req: Req,
|
||||
@@ -564,9 +586,11 @@ class MambaComponent(TreeComponent):
|
||||
if cache_len is None:
|
||||
cache_len = 0
|
||||
if self.cache.enable_mamba_extra_buffer:
|
||||
keep_idx = self.cache.req_to_token_pool.get_mamba_ping_pong_keep_idx(
|
||||
req
|
||||
)
|
||||
checkpoint = self._select_finished_checkpoint(req, token_ids_len)
|
||||
if checkpoint is None:
|
||||
return 0
|
||||
cache_len, keep_idx = checkpoint
|
||||
insert_params.mamba_keep_idx = keep_idx
|
||||
active_value = (
|
||||
req.kv.mamba_ping_pong_track_buffer[keep_idx].unsqueeze(-1).clone()
|
||||
)
|
||||
@@ -639,11 +663,11 @@ class MambaComponent(TreeComponent):
|
||||
return
|
||||
|
||||
if self.cache.enable_mamba_extra_buffer:
|
||||
keep_idx = (
|
||||
pool.get_mamba_ping_pong_keep_idx(req)
|
||||
if mamba_value_inserted
|
||||
else None
|
||||
# Keep the slot prepare picked, so cleanup cannot pick another.
|
||||
prepared_keep_idx = (
|
||||
insert_params.mamba_keep_idx if insert_params is not None else None
|
||||
)
|
||||
keep_idx = prepared_keep_idx if mamba_value_inserted else None
|
||||
pool.free_mamba_cache(
|
||||
req, mamba_ping_pong_track_buffer_to_keep=keep_idx
|
||||
)
|
||||
@@ -657,6 +681,7 @@ class MambaComponent(TreeComponent):
|
||||
):
|
||||
self._free_mamba_value(insert_params.mamba_value)
|
||||
req.kv.mamba_last_track_seqlen = None
|
||||
req.kv.mamba_prev_track_seqlen = None
|
||||
|
||||
def build_external_linker_transfer(
|
||||
self,
|
||||
|
||||
@@ -526,8 +526,8 @@ class DecodeCudaGraphRunner(BaseCudaGraphRunner):
|
||||
declared is SharedReadEnds.IN_REPLAY
|
||||
and self.in_graph_metadata_prep_done is None
|
||||
):
|
||||
# TODO: this lands EARLIER than declared; POST_REPLAY is the sound one.
|
||||
return SharedReadEnds.PRE_REPLAY
|
||||
# no in-graph marker (e.g. HIP): only a post-replay event covers its reads
|
||||
return SharedReadEnds.POST_REPLAY
|
||||
return declared
|
||||
|
||||
def _publish_read_done(self, in_graph: bool):
|
||||
|
||||
@@ -638,9 +638,7 @@ class DeepseekMHAForwardMixin:
|
||||
|
||||
Returns: (kv_a, k_pe) both in BF16
|
||||
"""
|
||||
backend = get_attn_backend()
|
||||
if isinstance(backend, TboAttnBackend): # if enable tbo, get primary backend
|
||||
backend = backend.primary
|
||||
backend = resolve_attn_backend(forward_batch)
|
||||
kv_indices = backend.forward_metadata.page_table_1_flattened
|
||||
assert kv_indices is not None, (
|
||||
"page_table_1_flattened should have been generated for FP8 MHA path"
|
||||
|
||||
@@ -25,6 +25,7 @@ from sglang.srt.eplb.expert_location import ModelConfigForExpertLocation
|
||||
from sglang.srt.layers.attention import vision_utils
|
||||
from sglang.srt.layers.attention.vision import VisionAttention
|
||||
from sglang.srt.layers.aux_hidden_states import AuxHiddenStateList
|
||||
from sglang.srt.layers.conv import Conv2dLayer
|
||||
from sglang.srt.layers.layer_boundary import (
|
||||
PLAIN_RESIDUAL_OPS,
|
||||
MHCState,
|
||||
@@ -317,11 +318,14 @@ class Glm5NextVisionModel(GlmOcrVisionModel):
|
||||
swiglu_limit=vision_config.swiglu_limit,
|
||||
)
|
||||
|
||||
self.downsample = nn.Conv2d(
|
||||
# These non-overlapping patches are equivalent to unfold + linear and
|
||||
# avoid MIOpen's expensive per-shape convolution search on ROCm.
|
||||
self.downsample = Conv2dLayer(
|
||||
in_channels=vision_config.hidden_size,
|
||||
out_channels=vision_config.out_hidden_size,
|
||||
kernel_size=vision_config.spatial_merge_size,
|
||||
stride=vision_config.spatial_merge_size,
|
||||
disable_linear=False,
|
||||
)
|
||||
self.post_layernorm = GlmOcrRMSNorm(
|
||||
vision_config.hidden_size, eps=vision_config.rms_norm_eps
|
||||
@@ -1596,8 +1600,10 @@ class Glm5NextForConditionalGeneration(nn.Module):
|
||||
params_dict = dict(self.named_parameters())
|
||||
|
||||
def maybe_map_fp8_block_scale_name(name: str) -> str:
|
||||
if name.endswith("weight_scale"):
|
||||
candidate = name.removesuffix("weight_scale") + "weight_scale_inv"
|
||||
# Quark stores dequantization scales without native block-FP8's
|
||||
# "_inv" suffix, including fused w13/w2 expert parameters.
|
||||
if name not in params_dict and name.endswith("weight_scale"):
|
||||
candidate = name + "_inv"
|
||||
if candidate in params_dict:
|
||||
return candidate
|
||||
return name
|
||||
|
||||
@@ -59,6 +59,8 @@ from sglang.srt.utils.device_timer import device_timer_ctx
|
||||
if TYPE_CHECKING:
|
||||
from sglang.srt.speculative.eagle_worker_v2 import EagleDraftWorker
|
||||
|
||||
_GREEDY_VARIANT = "greedy"
|
||||
|
||||
|
||||
@dataclass
|
||||
class EagleDraftInputBuffers(ForwardInputBuffers):
|
||||
@@ -307,8 +309,19 @@ class EAGLEDraftCudaGraphRunner(DecodeCudaGraphRunner):
|
||||
return torch.int64
|
||||
|
||||
def _make_graph_key(self, bs, stream_idx=None, variant_label=None):
|
||||
# EAGLE doesn't use stream_idx / lora variants.
|
||||
return ShapeKey(size=bs)
|
||||
# EAGLE doesn't use stream_idx; variant_label is _GREEDY_VARIANT or None.
|
||||
return ShapeKey(size=bs, variant_label=variant_label)
|
||||
|
||||
def _sampling_variant(self, forward_batch: ForwardBatch) -> Optional[str]:
|
||||
# Under rejection sampling an all-greedy batch replays the graph that
|
||||
# skips the draft proposal distribution (see draft_forward).
|
||||
if (
|
||||
get_spec().speculative_use_rejection_sampling
|
||||
and forward_batch.sampling_info is not None
|
||||
and forward_batch.sampling_info.is_all_greedy
|
||||
):
|
||||
return _GREEDY_VARIANT
|
||||
return None
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
# can_run_graph
|
||||
@@ -332,7 +345,13 @@ class EAGLEDraftCudaGraphRunner(DecodeCudaGraphRunner):
|
||||
cuda_graph_bs = forward_batch.batch_size
|
||||
|
||||
is_bs_supported = (
|
||||
self.backend.can_run(forward_batch, self._make_graph_key(cuda_graph_bs))
|
||||
self.backend.can_run(
|
||||
forward_batch,
|
||||
self._make_graph_key(
|
||||
cuda_graph_bs,
|
||||
variant_label=self._sampling_variant(forward_batch),
|
||||
),
|
||||
)
|
||||
if self.disable_padding
|
||||
else cuda_graph_bs <= self.max_bs
|
||||
)
|
||||
@@ -498,7 +517,6 @@ class EAGLEDraftCudaGraphRunner(DecodeCudaGraphRunner):
|
||||
# per-step forwards inside draft_forward must not re-plan.
|
||||
forward_batch.mark_forward_metadata_ready()
|
||||
self.deepep_adapter.capture(is_extend_in_batch=False)
|
||||
shape_key = self._make_graph_key(num_seqs)
|
||||
post_warmup_hook = getattr(
|
||||
self.draft_attn_backend, "on_after_cuda_graph_warmup", None
|
||||
)
|
||||
@@ -508,12 +526,19 @@ class EAGLEDraftCudaGraphRunner(DecodeCudaGraphRunner):
|
||||
post_warmup_hook=post_warmup_hook,
|
||||
run_lm_head=True,
|
||||
)
|
||||
self.backend.capture_one(
|
||||
shape_key,
|
||||
run_once,
|
||||
capture_inputs=None,
|
||||
post_warmup_hook=post_warmup_hook,
|
||||
variants = (
|
||||
(None, _GREEDY_VARIANT)
|
||||
if get_spec().speculative_use_rejection_sampling
|
||||
else (None,)
|
||||
)
|
||||
for variant in variants:
|
||||
sampling_info.is_all_greedy = variant == _GREEDY_VARIANT
|
||||
self.backend.capture_one(
|
||||
self._make_graph_key(num_seqs, variant_label=variant),
|
||||
run_once,
|
||||
capture_inputs=None,
|
||||
post_warmup_hook=post_warmup_hook,
|
||||
)
|
||||
|
||||
def _postprocess_output_to_raw_bs(self, out, raw_bs):
|
||||
parent_list, top_scores_index, draft_tokens, draft_probs = (
|
||||
@@ -694,7 +719,9 @@ class EAGLEDraftCudaGraphRunner(DecodeCudaGraphRunner):
|
||||
self.bs = bs
|
||||
|
||||
# Replay via backend
|
||||
shape_key = self._make_graph_key(bs)
|
||||
shape_key = self._make_graph_key(
|
||||
bs, variant_label=self._sampling_variant(forward_batch)
|
||||
)
|
||||
with device_timer_ctx(self.model_runner.device_timer, "eagle_draft"):
|
||||
out = self._replay_graph(shape_key, forward_batch)
|
||||
if self.buffers.dsa_seed_topk is not None:
|
||||
|
||||
@@ -7,6 +7,7 @@ from typing import TYPE_CHECKING, Callable, Optional
|
||||
import torch
|
||||
|
||||
from sglang.srt.compilation.torch_compile_decoration import set_torch_compile_config
|
||||
from sglang.srt.layers.attention.base_attn_backend import SharedReadEnds
|
||||
from sglang.srt.layers.dp_attention import (
|
||||
DpPaddingMode,
|
||||
deployment_attn_dp_size,
|
||||
@@ -631,12 +632,14 @@ class EAGLEDraftExtendCudaGraphRunner(DecodeCudaGraphRunner):
|
||||
)
|
||||
self.draft_extend_attn_backend.init_forward_metadata_out_graph(fb_view)
|
||||
|
||||
# Snapshot built -- the forward is done reading the shared pool. Publish
|
||||
# a read-done event the scheduler's WAR barrier waits on (draft extend
|
||||
# is the EAGLE-family last shared-read phase; last write wins the mailbox).
|
||||
read_done = self.device_module.Event()
|
||||
read_done.record()
|
||||
self.model_runner.shared_read_done_event = read_done
|
||||
# Draft extend is the last shared reader. UNKNOWN must discard any
|
||||
# earlier phase's event so the scheduler uses its whole-forward fence.
|
||||
self.model_runner.shared_read_done_event = None
|
||||
shared_read_ends = self.draft_extend_attn_backend.shared_read_ends(
|
||||
self.forward_mode
|
||||
)
|
||||
if shared_read_ends is SharedReadEnds.PRE_REPLAY:
|
||||
self._publish_read_done(in_graph=False)
|
||||
|
||||
self.raw_bs = raw_bs
|
||||
self.bs = bs
|
||||
@@ -644,6 +647,11 @@ class EAGLEDraftExtendCudaGraphRunner(DecodeCudaGraphRunner):
|
||||
with device_timer_ctx(self.model_runner.device_timer, "eagle_draft_extend"):
|
||||
out = self._replay_graph(shape_key, forward_batch)
|
||||
|
||||
# This runner has no in-graph read-done marker. A later event is safe;
|
||||
# an event before replay cannot cover metadata reads inside the graph.
|
||||
if shared_read_ends in (SharedReadEnds.IN_REPLAY, SharedReadEnds.POST_REPLAY):
|
||||
self._publish_read_done(in_graph=False)
|
||||
|
||||
out = LogitsProcessorOutput(
|
||||
next_token_logits=out.next_token_logits[:raw_bs],
|
||||
# CUDA graph replay reuses its captured output storage. These states
|
||||
|
||||
@@ -700,7 +700,13 @@ class EagleDraftWorker(EagleDraftWorkerBase):
|
||||
score_list: List[torch.Tensor] = []
|
||||
token_list: List[torch.Tensor] = []
|
||||
parents_list: List[torch.Tensor] = []
|
||||
if get_spec().speculative_use_rejection_sampling:
|
||||
# Verify commits argmax for an all-greedy batch and never reads the
|
||||
# draft distribution, so skip building it.
|
||||
needs_draft_probs = (
|
||||
get_spec().speculative_use_rejection_sampling
|
||||
and not forward_batch.sampling_info.is_all_greedy
|
||||
)
|
||||
if needs_draft_probs:
|
||||
draft_probs_list: List[torch.Tensor] = [spec_info.draft_probs]
|
||||
|
||||
topk1_chain_fits = (
|
||||
@@ -715,7 +721,7 @@ class EagleDraftWorker(EagleDraftWorkerBase):
|
||||
topk1_chain_fits
|
||||
and _is_cuda
|
||||
and self.hot_token_id is None
|
||||
and not get_spec().speculative_use_rejection_sampling
|
||||
and not needs_draft_probs
|
||||
):
|
||||
draft_tokens_topk1 = torch.empty(
|
||||
(topk_index.shape[0], self.speculative_num_steps),
|
||||
@@ -778,7 +784,7 @@ class EagleDraftWorker(EagleDraftWorkerBase):
|
||||
maybe_detect_inf(
|
||||
logits_output.next_token_logits, f"draft_forward step {i}"
|
||||
)
|
||||
if get_spec().speculative_use_rejection_sampling:
|
||||
if needs_draft_probs:
|
||||
probs, topk_p, topk_index = sample_draft_proposal(
|
||||
logits_output.next_token_logits,
|
||||
forward_batch.sampling_info.temperatures,
|
||||
@@ -804,7 +810,7 @@ class EagleDraftWorker(EagleDraftWorkerBase):
|
||||
probs = renorm_draft_probs(
|
||||
logits_output.next_token_logits,
|
||||
forward_batch.sampling_info,
|
||||
get_spec().speculative_use_rejection_sampling,
|
||||
needs_draft_probs,
|
||||
)
|
||||
topk_p, topk_index = fast_topk(probs, self.topk, dim=-1)
|
||||
forward_batch.positions.add_(1)
|
||||
@@ -821,9 +827,7 @@ class EagleDraftWorker(EagleDraftWorkerBase):
|
||||
hidden_states = logits_output.hidden_states
|
||||
|
||||
draft_probs = (
|
||||
torch.stack(draft_probs_list, dim=1)
|
||||
if get_spec().speculative_use_rejection_sampling
|
||||
else None
|
||||
torch.stack(draft_probs_list, dim=1) if needs_draft_probs else None
|
||||
)
|
||||
|
||||
# Organize the results
|
||||
|
||||
@@ -12,9 +12,11 @@ from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
from sglang.kernels.ops.mamba.causal_conv1d_triton import (
|
||||
causal_conv1d_update,
|
||||
)
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
|
||||
|
||||
register_cuda_ci(est_time=90, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||
# backend-specific: the gfx950 product-rounding boundary must stay bit-exact.
|
||||
register_amd_ci(est_time=90, suite="stage-b-test-1-gpu-small-amd-mi35x")
|
||||
|
||||
_DEVICE = "cuda"
|
||||
|
||||
@@ -29,10 +31,19 @@ _CASES = [
|
||||
(1, 4, 8, 8, 64, 64, 4, True, None, False, 8),
|
||||
]
|
||||
|
||||
# ReplaySSM ring-write cases: _CASES plus HV != H shapes, which exercise the
|
||||
_RING_CASES = list(_CASES)
|
||||
|
||||
_CASES += [
|
||||
# GLM-5.3 Flash TP4: 16 local heads, six-token EAGLE verification,
|
||||
# and a negative safe-gate lower bound.
|
||||
(1, 6, 16, 16, 128, 128, 4, False, -5.0, False, 9),
|
||||
(16, 8, 16, 16, 128, 128, 4, False, -5.0, True, 10),
|
||||
]
|
||||
|
||||
# ReplaySSM ring-write cases add HV != H shapes, which exercise the
|
||||
# per-k-head rawk vs per-v-head g/beta writer split (the GQA hazard: a wrong
|
||||
# head index scribbles another head's ring silently).
|
||||
_RING_CASES = _CASES + [
|
||||
_RING_CASES += [
|
||||
(2, 4, 2, 4, 128, 128, 4, True, None, False, 20),
|
||||
(1, 5, 2, 8, 64, 64, 4, True, 1.5, False, 21),
|
||||
(3, 4, 4, 8, 128, 128, 4, True, None, True, 22),
|
||||
@@ -59,17 +70,29 @@ def _make_ring_buffers(H, HV, K, V):
|
||||
}
|
||||
|
||||
|
||||
def _make_inputs(B, T, H, HV, K, V, W, has_bias, neg_slot, seed):
|
||||
def _make_inputs(
|
||||
B,
|
||||
T,
|
||||
H,
|
||||
HV,
|
||||
K,
|
||||
V,
|
||||
W,
|
||||
has_bias,
|
||||
neg_slot,
|
||||
seed,
|
||||
weight_dtype=torch.bfloat16,
|
||||
):
|
||||
torch.manual_seed(seed)
|
||||
dim = 2 * H * K + HV * V
|
||||
seq_len = B * T
|
||||
lines = slots = 8
|
||||
lines = slots = max(8, B + 2)
|
||||
|
||||
inputs = {
|
||||
"mixed": torch.randn(seq_len, dim, device=_DEVICE, dtype=torch.bfloat16) * 0.5,
|
||||
"w": torch.randn(dim, W, device=_DEVICE, dtype=torch.bfloat16) * 0.3,
|
||||
"w": torch.randn(dim, W, device=_DEVICE, dtype=weight_dtype) * 0.3,
|
||||
"bias": (
|
||||
torch.randn(dim, device=_DEVICE, dtype=torch.bfloat16) * 0.1
|
||||
torch.randn(dim, device=_DEVICE, dtype=weight_dtype) * 0.1
|
||||
if has_bias
|
||||
else None
|
||||
),
|
||||
@@ -185,7 +208,7 @@ def _run_fused(inp, B, T, H, HV, K, V, lower_bound, num_warps, rings=None):
|
||||
head_k_dim=K,
|
||||
head_v_dim=V,
|
||||
lower_bound=lower_bound,
|
||||
num_warps=num_warps,
|
||||
**({"num_warps": num_warps} if num_warps is not None else {}),
|
||||
cache_ring=rings is not None,
|
||||
replayssm_rawv=rings["rawv"] if rings is not None else None,
|
||||
replayssm_rawk=rings["rawk"] if rings is not None else None,
|
||||
@@ -195,9 +218,9 @@ def _run_fused(inp, B, T, H, HV, K, V, lower_bound, num_warps, rings=None):
|
||||
return o, conv, win, ic
|
||||
|
||||
|
||||
def _compare_case(case, num_warps, use_ring=False):
|
||||
def _compare_case(case, num_warps=None, use_ring=False, weight_dtype=torch.bfloat16):
|
||||
B, T, H, HV, K, V, W, has_bias, lower_bound, neg_slot, seed = case
|
||||
inp = _make_inputs(B, T, H, HV, K, V, W, has_bias, neg_slot, seed)
|
||||
inp = _make_inputs(B, T, H, HV, K, V, W, has_bias, neg_slot, seed, weight_dtype)
|
||||
if use_ring:
|
||||
template = _make_ring_buffers(H, HV, K, V)
|
||||
rings_ref = {name: buf.clone() for name, buf in template.items()}
|
||||
@@ -239,11 +262,32 @@ def test_matches_unfused_reference(case):
|
||||
_compare_case(case, num_warps=4)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("case_index", [8, 9])
|
||||
def test_glm_fp32_weights_match_unfused_reference(case_index):
|
||||
_compare_case(_CASES[case_index], weight_dtype=torch.float32)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not torch.version.hip, reason="ROCm default launch shape")
|
||||
@pytest.mark.parametrize("case_index", [8, 9])
|
||||
def test_glm_fp32_weights_bit_exact_on_rocm(case_index):
|
||||
# One wave keeps the reference reduction order and separately rounded
|
||||
# conv products keep LLVM from contracting them into FMAs.
|
||||
B, T, H, HV, K, V, W, bias, lower, neg_slot, seed = _CASES[case_index]
|
||||
inp = _make_inputs(B, T, H, HV, K, V, W, bias, neg_slot, seed, torch.float32)
|
||||
o_ref, _, _, ic_ref = _run_reference(inp, B, T, H, HV, K, V, lower)
|
||||
o_fus, _, _, ic_fus = _run_fused(inp, B, T, H, HV, K, V, lower, None)
|
||||
valid = [i for i, slot in enumerate(inp["idx_vals"]) if slot >= 0]
|
||||
assert torch.equal(
|
||||
o_fus.reshape(B, T, HV, V)[valid], o_ref.reshape(B, T, HV, V)[valid]
|
||||
)
|
||||
assert torch.equal(ic_fus[valid], ic_ref[valid])
|
||||
|
||||
|
||||
def test_output_does_not_depend_on_cta_scheduling():
|
||||
"""The verify output must not change with how the CTAs happen to be
|
||||
scheduled. H=1 with HV=16 shares one Q/K history across 16 V tiles."""
|
||||
if torch.cuda.get_device_capability()[0] < 9:
|
||||
pytest.skip("green contexts need SM90 or newer")
|
||||
if torch.version.hip or torch.cuda.get_device_capability()[0] < 9:
|
||||
pytest.skip("green contexts need CUDA SM90 or newer")
|
||||
from flashinfer.green_ctx import split_device_green_ctx_by_sm_count
|
||||
|
||||
case = (1, 6, 1, 16, 128, 128, 4, False, None, False, 1)
|
||||
|
||||
@@ -10,21 +10,24 @@ The radix selector does not specify an output order and DSA attention is
|
||||
permutation-invariant over the selected set, so the pooled columns are compared
|
||||
as a set. The tail columns are positional and are compared exactly.
|
||||
|
||||
Registered for AMD only: the kernel had no direct coverage on any platform, and
|
||||
adding CUDA coverage for it is not this change's call to make.
|
||||
The overfull-bin rescan is shared by CUDA and ROCm, so the tie and graph-replay
|
||||
cases run on both.
|
||||
"""
|
||||
|
||||
import unittest
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from sglang.kernels.ops.attention.dsa.kpool_topk_transform import (
|
||||
fast_kpool_topk_transform_fused,
|
||||
)
|
||||
from sglang.test.ci.ci_register import register_amd_ci
|
||||
from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
|
||||
from sglang.test.test_utils import CustomTestCase
|
||||
|
||||
register_amd_ci(est_time=60, stage="jit-kernel-unit", runner_config="amd")
|
||||
register_cuda_ci(est_time=90, stage="base-b-kernel-unit", runner_config="1-gpu-large")
|
||||
# backend-specific: compile the HIP JIT path and exercise wave64 graph replay.
|
||||
register_amd_ci(est_time=90, stage="jit-kernel-unit", runner_config="amd")
|
||||
|
||||
|
||||
@unittest.skipUnless(torch.cuda.is_available(), "Test requires a GPU")
|
||||
@@ -105,5 +108,113 @@ class TestKpoolTopkTransformFused(CustomTestCase):
|
||||
self.assertEqual(tuple(out.shape), (1, topk + self.POOL_SIZE - 1))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"distribution,start",
|
||||
[
|
||||
("equal", 0),
|
||||
("late_higher", 8192),
|
||||
],
|
||||
)
|
||||
@pytest.mark.skipif(not torch.cuda.is_available(), reason="Requires a GPU")
|
||||
def test_topk_membership_including_overfull_coarse_bins(distribution, start):
|
||||
torch.manual_seed(1234)
|
||||
rows, width, length, group_topk, pool_size = 4, 50000, 32768, 512, 4
|
||||
scores = torch.randn(rows, width, device="cuda")
|
||||
if distribution == "equal":
|
||||
scores.fill_(1)
|
||||
elif distribution == "late_higher":
|
||||
scores.fill_(1)
|
||||
scores[:, start + length // 2 : start + length] = 1.001
|
||||
lengths = torch.full((rows,), length, dtype=torch.int32, device="cuda")
|
||||
starts = torch.full_like(lengths, start)
|
||||
result = fast_kpool_topk_transform_fused(
|
||||
scores,
|
||||
lengths,
|
||||
pool_size,
|
||||
group_topk * pool_size,
|
||||
row_starts=starts,
|
||||
seq_lens=lengths * pool_size + 3,
|
||||
)
|
||||
groups = result[:, :2048:pool_size].long() // pool_size
|
||||
assert bool(((groups >= 0) & (groups < length)).all())
|
||||
for row in groups:
|
||||
assert torch.unique(row).numel() == group_topk
|
||||
torch.testing.assert_close(
|
||||
result[:, :2048].reshape(rows, group_topk, pool_size).long(),
|
||||
groups.unsqueeze(-1) * pool_size + torch.arange(pool_size, device="cuda"),
|
||||
atol=0,
|
||||
rtol=0,
|
||||
)
|
||||
# Ties may select any tied group and output order is unspecified.
|
||||
actual = scores.gather(1, groups + start).sort(dim=1).values
|
||||
expected = (
|
||||
scores[:, start : start + length].topk(group_topk).values.sort(dim=1).values
|
||||
)
|
||||
torch.testing.assert_close(actual, expected, atol=0, rtol=0)
|
||||
torch.testing.assert_close(
|
||||
result[:, -3:],
|
||||
torch.arange(
|
||||
length * pool_size, length * pool_size + 3, dtype=torch.int32, device="cuda"
|
||||
).expand(rows, -1),
|
||||
atol=0,
|
||||
rtol=0,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not torch.cuda.is_available(), reason="Requires a GPU")
|
||||
@pytest.mark.parametrize("long_length", [600, 32768])
|
||||
def test_graph_replay_short_rows_page_mapping_and_tail(long_length):
|
||||
width = long_length + 40
|
||||
scores = (
|
||||
torch.arange(width, dtype=torch.float32, device="cuda")
|
||||
.expand(3, -1)
|
||||
.contiguous()
|
||||
)
|
||||
lengths = torch.tensor([0, 3, long_length], dtype=torch.int32, device="cuda")
|
||||
seq_lens = lengths * 4 + torch.tensor([0, 2, 1], device="cuda", dtype=torch.int32)
|
||||
page_width = width * 4
|
||||
pages = torch.arange(4 * page_width, dtype=torch.int32, device="cuda").reshape(
|
||||
4, page_width
|
||||
)
|
||||
row_index = torch.tensor([2, 0, 3], dtype=torch.int32, device="cuda")
|
||||
|
||||
def run():
|
||||
return fast_kpool_topk_transform_fused(
|
||||
scores,
|
||||
lengths,
|
||||
4,
|
||||
2048,
|
||||
page_table=pages,
|
||||
page_table_row_index=row_index,
|
||||
seq_lens=seq_lens,
|
||||
)
|
||||
|
||||
run() # Compile before capture.
|
||||
torch.cuda.synchronize()
|
||||
graph = torch.cuda.CUDAGraph()
|
||||
with torch.cuda.graph(graph):
|
||||
result = run()
|
||||
for reverse in (False, True):
|
||||
if reverse:
|
||||
scores.copy_(scores.flip(1))
|
||||
graph.replay()
|
||||
torch.cuda.synchronize()
|
||||
expected = torch.full_like(result, -1)
|
||||
for row, (length, tail) in enumerate(((0, 0), (3, 2), (long_length, 1))):
|
||||
selected = scores[row, :length].topk(min(length, 512)).indices
|
||||
tokens = (selected[:, None] * 4 + torch.arange(4, device="cuda")).flatten()
|
||||
n = tokens.numel()
|
||||
expected[row, :n] = pages[row_index[row], tokens]
|
||||
expected[row, n : n + tail] = pages[
|
||||
row_index[row], length * 4 : length * 4 + tail
|
||||
]
|
||||
torch.testing.assert_close(
|
||||
result.sort(dim=1).values,
|
||||
expected.sort(dim=1).values,
|
||||
atol=0,
|
||||
rtol=0,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -620,6 +620,76 @@ class ServingChatTestCase(CustomTestCase):
|
||||
self.assertEqual(adapted.session_id, "session-1")
|
||||
self.assertEqual(processed, self.basic_req)
|
||||
|
||||
def test_text_only_prompt_reuses_rendered_token_ids(self):
|
||||
self.tm.model_config.is_multimodal = True
|
||||
self.chat.chat_encoding_spec = None
|
||||
self.template_manager.chat_template_name = None
|
||||
self.chat._tokenizer_auto_adds_specials = False
|
||||
self.chat._prompt_text_round_trip_is_lossy = False
|
||||
processed = MessageProcessingResult(
|
||||
prompt="rendered prompt",
|
||||
prompt_ids=[11, 22, 33],
|
||||
image_data=None,
|
||||
audio_data=None,
|
||||
video_data=None,
|
||||
modalities=[],
|
||||
stop=[],
|
||||
)
|
||||
|
||||
with patch.object(self.chat, "_process_messages", return_value=processed):
|
||||
adapted, _ = self.chat._convert_to_internal_request(self.basic_req)
|
||||
|
||||
self.assertEqual(adapted.input_ids, [11, 22, 33])
|
||||
self.assertIsNone(adapted.text)
|
||||
|
||||
def test_prompt_reuse_keeps_media_path(self):
|
||||
self.tm.model_config.is_multimodal = True
|
||||
self.chat.chat_encoding_spec = None
|
||||
self.template_manager.chat_template_name = None
|
||||
self.chat._tokenizer_auto_adds_specials = False
|
||||
self.chat._prompt_text_round_trip_is_lossy = False
|
||||
processed = MessageProcessingResult(
|
||||
prompt="rendered prompt",
|
||||
prompt_ids=[11, 22, 33],
|
||||
image_data=None,
|
||||
audio_data=None,
|
||||
video_data=None,
|
||||
modalities=[],
|
||||
stop=[],
|
||||
)
|
||||
processed.image_data = ["image"]
|
||||
|
||||
with patch.object(self.chat, "_process_messages", return_value=processed):
|
||||
adapted, _ = self.chat._convert_to_internal_request(self.basic_req)
|
||||
|
||||
self.assertEqual(adapted.text, "rendered prompt")
|
||||
self.assertIsNone(adapted.input_ids)
|
||||
|
||||
def test_prompt_reuse_keeps_always_on_processor_path(self):
|
||||
self.tm.model_config.is_multimodal = True
|
||||
self.tm.model_config.hf_config.architectures = [
|
||||
"MossVLForConditionalGeneration"
|
||||
]
|
||||
self.chat.chat_encoding_spec = None
|
||||
self.template_manager.chat_template_name = None
|
||||
self.chat._tokenizer_auto_adds_specials = False
|
||||
self.chat._prompt_text_round_trip_is_lossy = False
|
||||
processed = MessageProcessingResult(
|
||||
prompt="rendered prompt",
|
||||
prompt_ids=[11, 22, 33],
|
||||
image_data=None,
|
||||
audio_data=None,
|
||||
video_data=None,
|
||||
modalities=[],
|
||||
stop=[],
|
||||
)
|
||||
|
||||
with patch.object(self.chat, "_process_messages", return_value=processed):
|
||||
adapted, _ = self.chat._convert_to_internal_request(self.basic_req)
|
||||
|
||||
self.assertEqual(adapted.text, "rendered prompt")
|
||||
self.assertIsNone(adapted.input_ids)
|
||||
|
||||
def test_chat_applies_pd_header_overrides(self):
|
||||
request = ChatCompletionRequest(
|
||||
model="x",
|
||||
@@ -1162,6 +1232,23 @@ class ServingChatTestCase(CustomTestCase):
|
||||
kwargs = self.tm.tokenizer.apply_chat_template.call_args.kwargs
|
||||
self.assertEqual(kwargs["tools"], expected_tools)
|
||||
|
||||
def test_glm47_without_tools_has_no_tool_call_constraint(self):
|
||||
"""A plain GLM47 chat must not get the full-assistant EBNF: its
|
||||
terminal state finishes the request even under ignore_eos."""
|
||||
self.template_manager.chat_template_name = None
|
||||
self.template_manager.jinja_template_content_format = "string"
|
||||
self.tm.tokenizer.apply_chat_template.return_value = [1, 2, 3]
|
||||
self.chat.tool_call_parser = "glm47"
|
||||
|
||||
req = ChatCompletionRequest(
|
||||
model="x",
|
||||
messages=[{"role": "user", "content": "What is 2+2?"}],
|
||||
)
|
||||
|
||||
processed = self.chat._process_messages(req, is_multimodal=False)
|
||||
|
||||
self.assertIsNone(processed.tool_call_constraint)
|
||||
|
||||
def test_jinja_tool_schema_fallback_to_flat_function(self):
|
||||
"""Fallback to function-only schema when template rejects OpenAI wrapper."""
|
||||
self.template_manager.chat_template_name = None
|
||||
|
||||
@@ -19,8 +19,8 @@ torch = pytest.importorskip("torch")
|
||||
|
||||
from sglang.srt.layers.attention.dsa import dsa_indexer_kpool # noqa: E402
|
||||
from sglang.srt.layers.attention.mqa_logits_utils import ( # noqa: E402
|
||||
MQA_LOGITS_BYTES_PER_ELEM,
|
||||
MQA_LOGITS_MAX_BYTES_ROCM,
|
||||
mqa_logits_row_bytes,
|
||||
mqa_logits_should_chunk,
|
||||
)
|
||||
from sglang.test.ci.ci_register import register_cpu_ci # noqa: E402
|
||||
@@ -102,9 +102,10 @@ def _kpool_mqa_logits(num_q, num_k, cap_bytes):
|
||||
|
||||
def test_kpool_splits_the_query_rows_to_stay_under_the_aiter_cap():
|
||||
"""Above 2 GiB of logits the aiter kernel abort()s the process instead of
|
||||
raising, so the k-pool wrapper must never hand it a larger tensor."""
|
||||
raising, so the k-pool wrapper must never hand it a larger tensor. aiter
|
||||
pads each row to 256 columns, so the cap counts padded rows."""
|
||||
num_q, num_k = 8, 4
|
||||
row_bytes = num_k * MQA_LOGITS_BYTES_PER_ELEM
|
||||
row_bytes = mqa_logits_row_bytes(num_k)
|
||||
logits, rows_per_call = _kpool_mqa_logits(num_q, num_k, 3 * row_bytes + 1)
|
||||
|
||||
assert rows_per_call == [3, 3, 2]
|
||||
|
||||
@@ -5,7 +5,11 @@ from unittest.mock import MagicMock, patch
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.srt.managers.schedule_batch import Req, ScheduleBatch
|
||||
from sglang.srt.managers.schedule_batch import (
|
||||
Req,
|
||||
ScheduleBatch,
|
||||
set_mamba_track_indices_from_reqs,
|
||||
)
|
||||
from sglang.srt.managers.scheduler import Scheduler
|
||||
from sglang.srt.managers.scheduler_components.batch_result_processor import (
|
||||
SchedulerBatchResultProcessor,
|
||||
@@ -23,6 +27,20 @@ register_cpu_ci(est_time=12, suite="base-a-test-cpu")
|
||||
TRACK_INTERVAL = 4
|
||||
|
||||
|
||||
def _set_mamba_track_indices_on_cpu(batch, track_positions=None):
|
||||
real_tensor = torch.tensor
|
||||
|
||||
def tensor_without_pinning(*args, **kwargs):
|
||||
kwargs["pin_memory"] = False
|
||||
return real_tensor(*args, **kwargs)
|
||||
|
||||
with patch(
|
||||
"sglang.srt.managers.schedule_batch.torch.tensor",
|
||||
side_effect=tensor_without_pinning,
|
||||
):
|
||||
set_mamba_track_indices_from_reqs(batch, track_positions)
|
||||
|
||||
|
||||
def _make_batch() -> tuple[Req, ScheduleBatch]:
|
||||
sampling_params = SamplingParams(max_new_tokens=32)
|
||||
sampling_params.normalize(None)
|
||||
@@ -179,5 +197,54 @@ class TestMambaBoundaryMaskReuse(unittest.TestCase):
|
||||
self.assertTrue(cache_update.call_args.kwargs["known_boundary"])
|
||||
|
||||
|
||||
class TestMambaPrevTrackSeqlen(unittest.TestCase):
|
||||
def test_flip_names_the_checkpoint_in_the_other_slot(self):
|
||||
req, batch = _make_batch()
|
||||
processor = _make_processor()
|
||||
req.kv.mamba_ping_pong_track_buffer = torch.tensor([10, 11])
|
||||
req.kv.mamba_next_track_idx = 0
|
||||
req.kv.mamba_last_track_idx = 1
|
||||
batch.mamba_track_buffer_indices = None
|
||||
batch.req_to_token_pool = SimpleNamespace(
|
||||
get_mamba_ping_pong_other_idx=lambda idx: 1 - idx
|
||||
)
|
||||
with (
|
||||
get_context().override_server_args(
|
||||
mamba_radix_cache_strategy="extra_buffer",
|
||||
mamba_track_interval=TRACK_INTERVAL,
|
||||
_mamba_cache_chunk_size=TRACK_INTERVAL,
|
||||
),
|
||||
patch.object(
|
||||
SchedulerBatchResultProcessor,
|
||||
"_mamba_check_track_boundary",
|
||||
side_effect=[(True, TRACK_INTERVAL), (True, 2 * TRACK_INTERVAL)],
|
||||
),
|
||||
):
|
||||
processor._mamba_prefix_cache_update(req, batch, _make_result(), 0)
|
||||
self.assertIsNone(req.kv.mamba_prev_track_seqlen)
|
||||
processor._mamba_prefix_cache_update(req, batch, _make_result(), 0)
|
||||
self.assertEqual(req.kv.mamba_last_track_seqlen, 2 * TRACK_INTERVAL)
|
||||
self.assertEqual(req.kv.mamba_prev_track_seqlen, TRACK_INTERVAL)
|
||||
|
||||
|
||||
class TestMambaFreedTrackIndicesCPU(unittest.TestCase):
|
||||
def test_freed_rows_are_inert_and_live_rows_preserve_mapping(self):
|
||||
for positions, expected in ((None, [-1, 11]), ([1, 1], [-1, 12])):
|
||||
with self.subTest(positions=positions):
|
||||
mapping = torch.tensor([[7, 8], [11, 12]], dtype=torch.int64)
|
||||
batch = SimpleNamespace(
|
||||
reqs=[
|
||||
SimpleNamespace(kv=SimpleNamespace(mamba_next_track_idx=p))
|
||||
for p in (None, 0)
|
||||
],
|
||||
req_pool_indices=torch.arange(2),
|
||||
req_to_token_pool=SimpleNamespace(
|
||||
req_index_to_mamba_ping_pong_track_buffer_mapping=mapping
|
||||
),
|
||||
)
|
||||
_set_mamba_track_indices_on_cpu(batch, positions)
|
||||
self.assertEqual(batch.mamba_track_indices.tolist(), expected)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -11,6 +11,7 @@ length finish. Drives the real `Req.update_finish_state`; pure CPU."""
|
||||
|
||||
import unittest
|
||||
from array import array
|
||||
from unittest import mock
|
||||
|
||||
from sglang.test.ci.ci_register import register_cpu_ci
|
||||
from sglang.test.test_utils import CustomTestCase, maybe_stub_sgl_kernel
|
||||
@@ -23,6 +24,7 @@ from sglang.srt.managers.schedule_batch import (
|
||||
FINISH_MATCHED_TOKEN,
|
||||
Req,
|
||||
)
|
||||
from sglang.srt.runtime_context import get_context
|
||||
from sglang.srt.sampling.sampling_params import SamplingParams
|
||||
|
||||
register_cpu_ci(est_time=10, suite="base-a-test-cpu")
|
||||
@@ -198,5 +200,26 @@ class TestPostEosBonusTokenIncident(CustomTestCase):
|
||||
self.assertEqual(list(req.output_ids_through_stop), [10, 11, 12, EOT_ID])
|
||||
|
||||
|
||||
class TestSpecOvershootCacheLen(CustomTestCase):
|
||||
def test_commit_past_stop_is_not_a_cache_key(self):
|
||||
req = _make_req([10, 11, 12, EOS_ID, 20], max_new_tokens=100)
|
||||
req.kv.kv_committed_len = len(req.origin_input_ids) + len(req.output_ids)
|
||||
req.update_finish_state(new_accepted_len=5)
|
||||
with get_context().override_server_args(strip_thinking_cache=False):
|
||||
self.assertEqual(req.owned_kv_len(), 5)
|
||||
|
||||
def test_abort_stopless_keeps_committed_len(self):
|
||||
"""An abort keeps kv_committed_len; clamping trips the overalloc assert."""
|
||||
req = _make_req([101, 102], max_new_tokens=10, vocab_size=10_000)
|
||||
req.kv.kv_committed_len = 100
|
||||
req.kv.kv_allocated_len = 100
|
||||
with mock.patch("sglang.srt.managers.schedule_batch.get_parallel") as gp:
|
||||
gp().tp_rank = 0
|
||||
req.set_finish_with_abort("boom")
|
||||
self.assertIsNone(req.finished_len)
|
||||
with get_context().override_server_args(strip_thinking_cache=False):
|
||||
self.assertEqual(req.owned_kv_len(), req.kv.kv_committed_len)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -282,6 +282,29 @@ class TestPrefillAdder(CustomTestCase):
|
||||
self.assertIs(adder.add_chunked_req(req), req)
|
||||
self.assertEqual(req.extend_range.length, 4096)
|
||||
|
||||
def test_exact_chunk_fill_keeps_mamba_chunks_page_aligned(self):
|
||||
# A Mamba checkpoint only lands on a page-aligned chunk end, so an
|
||||
# off-grid chunk leaves the rest of the prompt uncacheable.
|
||||
self.mock_token_allocator.available_size.return_value = 32768
|
||||
self.mock_tree_cache.is_tree_cache.return_value = False
|
||||
for supports_mamba, expected in ((False, 100), (True, 64)):
|
||||
with (
|
||||
self.subTest(supports_mamba=supports_mamba),
|
||||
patch.object(
|
||||
schedule_policy, "_use_exact_chunk_fill", return_value=True
|
||||
),
|
||||
):
|
||||
self.mock_tree_cache.supports_mamba.return_value = supports_mamba
|
||||
adder = self.create_adder(
|
||||
self.create_running_batch(), page_size=64, rem_chunk_tokens=100
|
||||
)
|
||||
req = self.create_shared_req("chunked")
|
||||
req.full_untruncated_fill_ids = list(range(300))
|
||||
adder.add_one_req(
|
||||
req, has_chunked_req=False, truncation_align_size=None
|
||||
)
|
||||
self.assertEqual(req.extend_range.length, expected)
|
||||
|
||||
def test_shared_admission_reserves_all_pending_requests(self):
|
||||
adder = self.create_shared_adder()
|
||||
first, second = (
|
||||
|
||||
@@ -1809,6 +1809,15 @@ class UnifiedRadixCacheSuite:
|
||||
req.extra_key = None
|
||||
if self.cfg.has_mamba:
|
||||
req.kv.mamba_last_track_seqlen = kv_len
|
||||
if self.cfg.enable_mamba_extra_buffer:
|
||||
# The prompt-only key needs its own checkpoint, not the
|
||||
# latest state that has already consumed thinking + answer.
|
||||
req.kv.mamba_prev_track_seqlen = len(prompt_ids)
|
||||
prompt_slot = req.kv.mamba_ping_pong_track_buffer[
|
||||
req_to_token_pool.get_mamba_ping_pong_other_idx(
|
||||
req.kv.mamba_last_track_idx
|
||||
)
|
||||
].clone()
|
||||
req.reasoning_tokens = 1
|
||||
|
||||
# owned_kv_len reads get_serving().strip_thinking_cache
|
||||
@@ -1829,6 +1838,16 @@ class UnifiedRadixCacheSuite:
|
||||
MatchPrefixParams(key=RadixKey(array("q", prompt_ids + output_ids)))
|
||||
)
|
||||
self.assertEqual(len(m.device_indices), prompt_aligned)
|
||||
if self.cfg.has_mamba and self.cfg.enable_mamba_extra_buffer:
|
||||
node_value = cache.tree_core.get_component_device_value(
|
||||
m.last_device_node, ComponentType.MAMBA
|
||||
)
|
||||
self.assertTrue(
|
||||
torch.equal(
|
||||
node_value.reshape(-1),
|
||||
prompt_slot.reshape(-1),
|
||||
)
|
||||
)
|
||||
# Only prompt-aligned pages remain owned by the tree.
|
||||
self.assertEqual(
|
||||
allocator.available_size(), avail_before + kv_len - prompt_aligned
|
||||
@@ -8719,6 +8738,68 @@ class TestMambaCheckpointGrid(CustomTestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestMambaFinishedOvershootCheckpoint(CustomTestCase):
|
||||
"""A donated state must have consumed exactly the prefix its key names."""
|
||||
|
||||
_rid = 0
|
||||
cfg = CacheConfig(
|
||||
page_size=4,
|
||||
components=(ComponentType.FULL, ComponentType.MAMBA),
|
||||
enable_mamba_extra_buffer=True,
|
||||
kv_size=64,
|
||||
max_context_len=64,
|
||||
)
|
||||
|
||||
def _build_req(self, allocator, pool, previous_track_seqlen):
|
||||
tokens = list(range(12))
|
||||
req = UnifiedRadixCacheSuite._make_req(self, pool)
|
||||
req.origin_input_ids = array("q", tokens)
|
||||
req.output_ids = array("q")
|
||||
req.extra_key = None
|
||||
req.swa_uuid_for_lock = None
|
||||
req.kv.kv_committed_len = len(tokens)
|
||||
req.kv.kv_allocated_len = len(tokens)
|
||||
req.kv.cache_protected_len = 0
|
||||
req.kv.mamba_last_track_seqlen = len(tokens)
|
||||
req.kv.mamba_prev_track_seqlen = previous_track_seqlen
|
||||
req.kv.mamba_last_track_idx = 0
|
||||
req.kv.mamba_next_track_idx = pool.get_mamba_ping_pong_other_idx(0)
|
||||
indices = allocator.alloc(len(tokens))
|
||||
self.assertIsNotNone(indices)
|
||||
pool.write((req.kv.req_pool_idx, slice(0, len(tokens))), indices)
|
||||
return req, tokens
|
||||
|
||||
def test_previous_checkpoint_or_no_donation(self):
|
||||
for previous_len, expected_len in ((8, 8), (None, 0), (12, 0)):
|
||||
with self.subTest(previous_len=previous_len):
|
||||
cache, allocator, pool = build_fixture(
|
||||
self.cfg, mamba_cache_chunk_size=4
|
||||
)
|
||||
req, tokens = self._build_req(allocator, pool, previous_len)
|
||||
previous_slot = req.kv.mamba_ping_pong_track_buffer[
|
||||
req.kv.mamba_next_track_idx
|
||||
].clone()
|
||||
req.last_node = cache.root_node_handle()
|
||||
finish_req(cache, req, 11)
|
||||
match = cache.match_prefix(
|
||||
MatchPrefixParams(key=RadixKey(array("q", tokens)))
|
||||
)
|
||||
self.assertEqual(len(match.device_indices), expected_len)
|
||||
if expected_len:
|
||||
node_value = cache.tree_core.get_component_device_value(
|
||||
match.last_device_node, ComponentType.MAMBA
|
||||
)
|
||||
self.assertTrue(
|
||||
torch.equal(
|
||||
node_value.reshape(-1),
|
||||
previous_slot.reshape(-1),
|
||||
)
|
||||
)
|
||||
else:
|
||||
self.assertIsNone(req.kv.mamba_pool_idx)
|
||||
cache.sanity_check()
|
||||
|
||||
|
||||
class TestUnifiedRadixCacheInt8MambaCheckpoint(CustomTestCase):
|
||||
cfg = CacheConfig(
|
||||
components=(ComponentType.FULL, ComponentType.MAMBA),
|
||||
|
||||
+4
-2
@@ -37,8 +37,10 @@ def _backend(declared: SharedReadEnds):
|
||||
[
|
||||
# The backend's declaration decides where the record lands.
|
||||
(SharedReadEnds.IN_REPLAY, True, SharedReadEnds.IN_REPLAY),
|
||||
# Nowhere to record in-graph -> fall back to the pre-replay record.
|
||||
(SharedReadEnds.IN_REPLAY, False, SharedReadEnds.PRE_REPLAY),
|
||||
# No external-event support (e.g. HIP): IN_REPLAY cannot be recorded
|
||||
# in-graph, and a PRE_REPLAY fence would not cover the replay reads.
|
||||
# The sound fallback is POST_REPLAY.
|
||||
(SharedReadEnds.IN_REPLAY, False, SharedReadEnds.POST_REPLAY),
|
||||
# Only an in-graph declaration is demoted; the rest pass through.
|
||||
(SharedReadEnds.POST_REPLAY, False, SharedReadEnds.POST_REPLAY),
|
||||
],
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.srt.layers.quantization.modelopt_quant import ModelOptFp4Config
|
||||
from sglang.srt.models import glm5_next
|
||||
@@ -10,7 +12,7 @@ from sglang.srt.runtime_context import get_parallel
|
||||
from sglang.test.ci.ci_register import register_cpu_ci
|
||||
from sglang.test.test_utils import CustomTestCase
|
||||
|
||||
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
|
||||
register_cpu_ci(est_time=6, suite="base-a-test-cpu")
|
||||
|
||||
NVIDIA_EXCLUDE_MODULES = [
|
||||
"model.language_model.embed_tokens",
|
||||
@@ -118,5 +120,44 @@ class TestGlm5NextModelOpt(CustomTestCase):
|
||||
self.assertEqual(mapper._map_name(source), expected)
|
||||
|
||||
|
||||
class TestGlm5NextNextNScaleLoading(CustomTestCase):
|
||||
def _load(self, weights, params):
|
||||
model = SimpleNamespace(
|
||||
config=SimpleNamespace(
|
||||
num_nextn_predict_layers=1, num_hidden_layers=45, n_routed_experts=2
|
||||
),
|
||||
num_fused_shared_experts=0,
|
||||
fuse_qkv_a_proj=False,
|
||||
quant_config=None,
|
||||
named_parameters=lambda: params.items(),
|
||||
model=SimpleNamespace(decoder=SimpleNamespace(self_attn=None)),
|
||||
)
|
||||
weights = [(f"model.language_model.layers.45.{k}", v) for k, v in weights]
|
||||
Glm5NextForConditionalGenerationNextN.load_weights(model, weights)
|
||||
|
||||
def _param(self):
|
||||
param = torch.nn.Parameter(torch.full((1, 2), -1.0), requires_grad=False)
|
||||
param.weight_loader = Mock(
|
||||
side_effect=lambda p, w, *args, **kwargs: p.data.copy_(w)
|
||||
)
|
||||
return param
|
||||
|
||||
def test_existing_scale_parameter_takes_priority(self):
|
||||
for source, target in (
|
||||
("self_attn.q_b_proj", "self_attn.q_b_proj."),
|
||||
("mlp.experts.0.gate_proj", "mlp.experts.w13_"),
|
||||
):
|
||||
with self.subTest(source=source):
|
||||
name = f"model.decoder.{target}weight_scale"
|
||||
param, unused = self._param(), self._param()
|
||||
scale = torch.tensor([[0.25, 0.5]])
|
||||
self._load(
|
||||
[(f"{source}.weight_scale", scale)],
|
||||
{name: param, f"{name}_inv": unused},
|
||||
)
|
||||
torch.testing.assert_close(param, scale)
|
||||
unused.weight_loader.assert_not_called()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -81,3 +81,36 @@ def test_nope_mha_k_cast_rocm(backend, zero_width_k_pe):
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main([__file__]))
|
||||
|
||||
|
||||
def test_fp8_dsa_prefix_read_uses_the_hybrid_full_attn_child():
|
||||
"""A hybrid KDA/DSA wrapper holds no metadata; the FP8 prefix read crashed on it."""
|
||||
page_table = torch.tensor([3, 5, 8])
|
||||
child = SimpleNamespace(
|
||||
forward_metadata=SimpleNamespace(page_table_1_flattened=page_table)
|
||||
)
|
||||
outer = SimpleNamespace(
|
||||
full_attn_backend=child, forward_metadata=None, kv_index_translator=None
|
||||
)
|
||||
read_indices = []
|
||||
|
||||
def get_mla_kv_buffer(layer, indices, dtype):
|
||||
read_indices.append(indices)
|
||||
return torch.zeros(len(indices), 1, 4), torch.zeros(len(indices), 1, 2)
|
||||
|
||||
fake_pool = SimpleNamespace(get_mla_kv_buffer=get_mla_kv_buffer)
|
||||
with (
|
||||
mock.patch.object(forward_mha, "_use_aiter_gfx95", True),
|
||||
mock.patch.object(forward_mha, "get_attn_backend", return_value=outer),
|
||||
mock.patch.object(forward_mha, "get_token_to_kv_pool", return_value=fake_pool),
|
||||
mock.patch.object(
|
||||
forward_mha,
|
||||
"filter_dcp_local_kv_indices",
|
||||
side_effect=lambda kv_indices: kv_indices,
|
||||
),
|
||||
):
|
||||
forward_mha.DeepseekMHAForwardMixin._get_mla_kv_buffer_from_fp8_for_dsa(
|
||||
SimpleNamespace(attn_mha=object()), SimpleNamespace(forward_mode=None)
|
||||
)
|
||||
|
||||
assert len(read_indices) == 1 and torch.equal(read_indices[0], page_table)
|
||||
|
||||
Reference in New Issue
Block a user