minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

Support Intern-S2-Mobius FP8 (#34908)

Browse Source
This commit is contained in:
Xinyuan Tong
2026-08-19 10:58:01 -07:00
committed by GitHub
parent 5f12839591
commit 157d8ad27a
3 changed files with 146 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/InternLM/Intern-S2-Mobius.mdx
+12 -5
View File
@@ -1,6 +1,6 @@
---
title: Intern-S2-Mobius
description: "Deploy Intern-S2-Mobius with SGLang — InternLM's Mobius-v0 multimodal model with a globally shared Knowledge Memory, hybrid GDN + full attention, MTP (NEXTN) speculative decoding, and 256K context, on NVIDIA H200 and B200."
description: "Deploy the BF16 and FP8 Intern-S2-Mobius checkpoints with SGLang on NVIDIA H200 and B200."
tag: NEW
---
@@ -52,7 +52,7 @@ import { benchmarks } from "/src/snippets/configs/internlm/intern-s2-mobius-benc
<Deployment config={config} benchmarks={benchmarks} />
<Note>
Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on 2×H200 TP=2 against `main @ e0828ee3` + PR [#33691](https://github.com/sgl-project/sglang/pull/33691) head (since merged 2026-08-08 — `lmsysorg/sglang:dev` is the live equivalent). GSM8K is the full 1319-example test split; GPQA is Diamond 198 problems × 8 repeats (pass@1 avg-of-8). Both ran with no server-side sampling override, so the checkpoint's `generation_config.json` defaults applied (temperature 1.0, top_p 0.95, top_k 20). The B200 recipes are inferred from the H200 ones and unverified — same flags, just a TP=2 or TP=1 Blackwell equivalent.
Speed and accuracy numbers apply to the BF16 checkpoint. They were measured with `--random-range-ratio 1.0`, `--flush-cache`, on 2×H200 TP=2 against `main @ e0828ee3` + PR [#33691](https://github.com/sgl-project/sglang/pull/33691) head (since merged 2026-08-08 — `lmsysorg/sglang:dev` is the live equivalent). GSM8K is the full 1319-example test split; GPQA is Diamond 198 problems × 8 repeats (pass@1 avg-of-8). Both ran with no server-side sampling override, so the checkpoint's `generation_config.json` defaults applied (temperature 1.0, top_p 0.95, top_k 20). The FP8 H200 recipes have launch, text, vision, and stop-reason verification but no benchmark numbers yet. All B200 recipes are inferred and unverified.
</Note>
## Playground
@@ -94,20 +94,27 @@ On the serving side the model is a hybrid: 30 of 40 transformer layers use **GDN
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>262,144</td>
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/internlm/Intern-S2-Mobius/blob/main/LICENSE">Apache-2.0</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong><a href="https://huggingface.co/internlm/Intern-S2-Mobius-FP8">Intern-S2-Mobius-FP8</a></strong></td>
<td style={{padding: "9px 12px"}}>Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · FP8 E4M3</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>262,144</td>
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/internlm/Intern-S2-Mobius-FP8/blob/main/LICENSE">Apache-2.0</a></td>
</tr>
</tbody>
</table>
**Recommended generation:** `temperature=0.8`, `top_p=1.0`, `top_k=50`, `min_p=0.0` — the values the [model card](https://huggingface.co/internlm/Intern-S2-Mobius) recommends. Note these are *not* what the checkpoint ships in `generation_config.json` (`temperature=1.0`, `top_p=0.95`, `top_k=20`), and SGLang applies that file by default (`--sampling-defaults model`) — so send the recommended values explicitly per request if you want them.
**Recommended generation:** `temperature=0.8`, `top_p=1.0`, `top_k=50`, `min_p=0.0` — the values the [model card](https://huggingface.co/internlm/Intern-S2-Mobius) recommends. Both checkpoints ship `generation_config.json` with different values (`temperature=1.0`, `top_p=0.95`, `top_k=20`), and SGLang applies that file by default (`--sampling-defaults model`) — so send the recommended values explicitly per request if you want them.
**Resources:** [HuggingFace](https://huggingface.co/internlm/Intern-S2-Mobius) · [GitHub (InternLM/Intern-S2-Mobius)](https://github.com/InternLM/Intern-S2-Mobius).
**Resources:** [HuggingFace (BF16)](https://huggingface.co/internlm/Intern-S2-Mobius) · [HuggingFace (FP8)](https://huggingface.co/internlm/Intern-S2-Mobius-FP8) · [GitHub (InternLM/Intern-S2-Mobius)](https://github.com/InternLM/Intern-S2-Mobius).
## 2. Configuration Tips
- **Trust remote code is required.** Intern-S2-Mobius ships a custom `configuration_interns2_mobius.py` / `modeling_interns2_mobius.py` on its HF repo; every recipe adds `--trust-remote-code`.
- **FP8 uses one H200.** The FP8 checkpoint stores E4M3 weights with 128×128 block scales and dynamic activation quantization. The verified H200 recipes use TP=1, `deep_gemm`, `--mem-fraction-static 0.6`, disabled prefill CUDA graphs, and decode graphs up to batch size 16. Higher memory fractions or the default graph range exhausted H200 memory during DeepGEMM warmup; the running MTP server used about 99 GB.
- **Speculative decoding schedule.** The checkpoint ships one MTP layer. Enable MTP for the lowest latency (`--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`). We measured accept-length ~3.9/4 draft tokens at 8K-in / 1K-out, roughly tripling single-stream decode speed (median TPOT 9.79 ms → 3.13 ms at conc=1, 14.26 ms → 6.84 ms at conc=16) and roughly doubling mid-concurrency total throughput (9358 → 18029 tokens/s at conc=16, 21395 → 26033 tokens/s at conc=64). The high-throughput recipe stays spec-off because once you can batch wide, its saturation point is higher (34786 tokens/s at conc=256 vs the spec recipe's peak at conc=64).
- **Mamba pool sizing.** GDN layers live in a separate Mamba state pool; the `--mamba-full-memory-ratio` (defaults to 0.9) controls the split between the 10 full-attention layers' KV pool and the 30 GDN layers' conv+SSM state pool. Default split comfortably handles conc=64 on a 2×H200 node; if you need higher concurrency than `--max-running-requests` allows for your workload, raise `--mamba-full-memory-ratio` slightly (each +1% mamba ratio costs full-attn KV).
- **Vision input.** Images are accepted via the standard `image_url` chat message type. Vision tokens are counted into the prompt (`prompt_tokens_details.image_tokens` shows the count), and the model honors `<|vision_start|>` / `<|vision_end|>` boundaries exactly.
- **B200 sizing.** B200 (192 GB HBM) fits the BF16 weights + KV + Mamba pool on a **single GPU** with `--tp 1`. The B200 cells in the panel inherit the H200 recipe with only `--tp` changed — unverified; treat them as a starting point until the Intern-S2-Mobius team publishes a Blackwell measurement.
- **B200 sizing.** B200 (192 GB HBM) fits the BF16 weights + KV + Mamba pool on a **single GPU** with `--tp 1`. The FP8 B200 cells inherit the verified H200 TP=1 recipe but remain unverified; treat them as a starting point until a Blackwell measurement is available.
## 3. Advanced Usage
docs/src/snippets/configs/internlm/intern-s2-mobius.jsx
+84 -5
View File
@@ -6,12 +6,12 @@ export const config = {
supportedHardware: ["h200", "b200"],
// Single released checkpoint — no variant axis.
variants: [
{ id: "default", label: "Intern-S2-Mobius", subtitle: "Mobius-v0 · BF16" },
{ id: "default", label: "Intern-S2-Mobius", subtitle: "Mobius-v0" },
],
quantizations: [
{ id: "bf16", label: "BF16" },
{ id: "fp8", label: "FP8" },
],
strategies: [
{ id: "low-latency", label: "Low-Latency" },
@@ -23,6 +23,7 @@ export const config = {
modelNames: {
"default|bf16": "internlm/Intern-S2-Mobius",
"default|fp8": "internlm/Intern-S2-Mobius-FP8",
},
placeholders: {
@@ -105,9 +106,7 @@ sgl-eval run gpqa \\
// — models/interns2_mobius.py), so EP has nothing to shard. The runtime enforces
// that: server_args._handle_model_specific_adjustments raises for this arch on
// `--ep-size != 1` (and `--pp-size != 1`), so an EP chip would emit a command
// that cannot start. `--moe-a2a-backend deepep` is out for the same reason, and
// arg_groups/overrides.py pins moe_runner_backend to triton_kernel as the only
// runner validated for the 2,560-expert bank.
// that cannot start. `--moe-a2a-backend deepep` is out for the same reason.
// ----- Card: "Parsers" -----
parsers: {
@@ -175,6 +174,46 @@ sgl-eval run gpqa \\
"--port {{PORT}}",
],
},
{
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
verified: true,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 1",
"--mem-fraction-static 0.6",
"--context-length 262144",
"--reasoning-parser qwen3",
"--moe-runner-backend deep_gemm",
"--disable-prefill-cuda-graph",
"--cuda-graph-max-bs-decode 16",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "h200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
verified: true,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 1",
"--mem-fraction-static 0.6",
"--context-length 262144",
"--reasoning-parser qwen3",
"--moe-runner-backend deep_gemm",
"--disable-prefill-cuda-graph",
"--cuda-graph-max-bs-decode 16",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
// ==== B200, 2 GPUs, BF16, low-latency (MTP NEXTN on) — INFERRED from H200 ====
{
match: { hw: "b200", variant: "default", quant: "bf16", strategy: "low-latency", nodes: "single" },
@@ -211,5 +250,45 @@ sgl-eval run gpqa \\
"--port {{PORT}}",
],
},
{
match: { hw: "b200", variant: "default", quant: "fp8", strategy: "low-latency", nodes: "single" },
verified: false,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 1",
"--mem-fraction-static 0.6",
"--context-length 262144",
"--reasoning-parser qwen3",
"--moe-runner-backend deep_gemm",
"--disable-prefill-cuda-graph",
"--cuda-graph-max-bs-decode 16",
"--speculative-algorithm NEXTN",
"--speculative-num-steps 3",
"--speculative-eagle-topk 1",
"--speculative-num-draft-tokens 4",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
{
match: { hw: "b200", variant: "default", quant: "fp8", strategy: "high-throughput", nodes: "single" },
verified: false,
env: [],
flags: [
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 1",
"--mem-fraction-static 0.6",
"--context-length 262144",
"--reasoning-parser qwen3",
"--moe-runner-backend deep_gemm",
"--disable-prefill-cuda-graph",
"--cuda-graph-max-bs-decode 16",
"--host {{HOST_IP}}",
"--port {{PORT}}",
],
},
],
};
python/sglang/srt/models/interns2_mobius.py
+50 -7
View File
@@ -91,6 +91,10 @@ def _normalize_mobius_weight_name(name: str) -> str:
return name
def _is_optional_mobius_parameter(name: str) -> bool:
return name.endswith((".attn.k_scale", ".attn.v_scale"))
def _load_fused_mobius_expert_weight(
*,
name: str,
@@ -99,8 +103,17 @@ def _load_fused_mobius_expert_weight(
num_experts: int,
record_slot,
) -> None:
if name.endswith("experts.gate_up_proj"):
parameter_name = name.replace("experts.gate_up_proj", "experts.w13_weight")
gate_up_suffixes = {
"experts.gate_up_proj": "experts.w13_weight",
"experts.gate_up_proj_scale_inv": "experts.w13_weight_scale_inv",
}
gate_up_suffix = next(
(suffix for suffix in gate_up_suffixes if name.endswith(suffix)), None
)
if gate_up_suffix is not None:
parameter_name = (
name.removesuffix(gate_up_suffix) + gate_up_suffixes[gate_up_suffix]
)
if parameter_name not in params_dict:
raise KeyError(
f"Mobius fused gate/up destination is missing: {parameter_name}"
@@ -127,8 +140,15 @@ def _load_fused_mobius_expert_weight(
)
return
if name.endswith("experts.down_proj"):
parameter_name = name.replace("experts.down_proj", "experts.w2_weight")
down_suffixes = {
"experts.down_proj": "experts.w2_weight",
"experts.down_proj_scale_inv": "experts.w2_weight_scale_inv",
}
down_suffix = next(
(suffix for suffix in down_suffixes if name.endswith(suffix)), None
)
if down_suffix is not None:
parameter_name = name.removesuffix(down_suffix) + down_suffixes[down_suffix]
if parameter_name not in params_dict:
raise KeyError(
f"Mobius fused down destination is missing: {parameter_name}"
@@ -166,11 +186,15 @@ def _expected_mobius_load_slots(
if parameter_id in seen_parameters:
continue
seen_parameters.add(parameter_id)
if ".meta_mlp." in name and name.endswith("experts.w13_weight"):
if ".meta_mlp." in name and name.endswith(
("experts.w13_weight", "experts.w13_weight_scale_inv")
):
for expert_id in range(num_experts):
expected.add((name, "w1", expert_id))
expected.add((name, "w3", expert_id))
elif ".meta_mlp." in name and name.endswith("experts.w2_weight"):
elif ".meta_mlp." in name and name.endswith(
("experts.w2_weight", "experts.w2_weight_scale_inv")
):
for expert_id in range(num_experts):
expected.add((name, "w2", expert_id))
elif ".qkv_proj." in name and name.startswith("model.layers."):
@@ -185,6 +209,8 @@ def _expected_mobius_load_slots(
elif ".in_proj_ba." in name:
expected.add((name, 0, None))
expected.add((name, 1, None))
elif _is_optional_mobius_parameter(name):
continue
else:
expected.add((name, None, None))
return expected
@@ -215,7 +241,12 @@ def _load_mobius_weights_strict(
name = _normalize_mobius_weight_name(source_name)
if ".meta_mlp." in name and name.endswith(
("experts.gate_up_proj", "experts.down_proj")
(
"experts.gate_up_proj",
"experts.down_proj",
"experts.gate_up_proj_scale_inv",
"experts.down_proj_scale_inv",
)
):
_load_fused_mobius_expert_weight(
name=name,
@@ -250,6 +281,8 @@ def _load_mobius_weights_strict(
)
parameter = params_dict[name]
loader = getattr(parameter, "weight_loader", default_weight_loader)
if _is_optional_mobius_parameter(name):
expected_slots.add((name, None, None))
record_slot(name)
loader(parameter, loaded_weight)
@@ -802,6 +835,16 @@ class InternS2MobiusForConditionalGeneration(Qwen3_5ForConditionalGeneration):
prefix: str = "",
language_model_cls=InternS2MobiusForCausalLM,
) -> None:
ignored_layers = getattr(quant_config, "ignored_layers", None)
if (
getattr(quant_config, "is_checkpoint_fp8_serialized", False)
and ignored_layers
):
# HF treats these parent entries as exact names; SGLang prefix matching
# would also skip their quantized qkv/z and output projections.
quant_config.ignored_layers = [
name for name in ignored_layers if not name.endswith(".linear_attn")
]
super().__init__(config, quant_config, prefix, language_model_cls)
def should_apply_lora(self, module_name: str) -> bool: