minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[CPU] Fix model failures on Xeon (#29497)

Browse Source
This commit is contained in:
Zaili Wang
2026-07-02 13:20:18 +08:00
committed by GitHub
parent aff44d748d
commit cb06c4e6ce
6 changed files with 86 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/xeon.Dockerfile
+1 -2
View File
@@ -42,8 +42,7 @@ RUN source /opt/.venv/bin/activate && \
uv pip install . && \
cd ../sgl-kernel && \
cp pyproject_cpu.toml pyproject.toml && \
uv pip install . && \
uv pip install pytest
uv pip install .
ENV SGLANG_USE_CPU_ENGINE=1
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4:/usr/lib/x86_64-linux-gnu/libtbbmalloc.so:/opt/.venv/lib/libiomp5.so
docs_new/docs/hardware-platforms/cpu_server.mdx
+18 -69
View File
@@ -5,82 +5,31 @@ The document addresses how to set up the [SGLang](https://github.com/sgl-project
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
which are 4th generation or newer Intel® Xeon® Scalable Processors.
## Optimized Model List
A list of popular LLMs are optimized and run efficiently on CPU,
A number of popular LLMs are optimized and run efficiently on CPU,
including the most notable open-source models like Llama series, Qwen series,
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "22%"}} />
<col style={{width: "26%"}} />
<col style={{width: "30%"}} />
<col style={{width: "22%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8_INT8</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8">meituan/DeepSeek-R1-Channel-INT8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-R1">deepseek-ai/DeepSeek-R1</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1-Terminus</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8">IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus">deepseek-ai/DeepSeek-V3.1-Terminus</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct">meta-llama/Llama-3.2-3B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8">RedHatAI/Llama-3.2-3B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct">meta-llama/Llama-3.1-8B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8">RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QwQ-32B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8">RedHatAI/QwQ-32B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-Distilled-Llama</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8">RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8">Qwen/Qwen3-235B-A22B-FP8</a></td>
</tr>
</tbody>
</table>
**Note:** The model identifiers listed in the table above
have been verified on 6th Gen Intel® Xeon® P-core platforms.
Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro)
in which the support status and example commands can be found.
## Installation
### Install Using Docker
It is recommended to use Docker for setting up the SGLang environment.
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation.
#### Pull from Docker Hub
Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository.
The [CPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xeon) end with `xeon` suffix.
The image pulling command is like:
```bash Command
docker pull lmsysorg/sglang:v0.5.13-xeon
```
#### Build from Dockerfile
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation from latest source code.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
```bash Command
@@ -112,7 +61,7 @@ the setup process is as follows:
Please install the required packages and libraries beforehand if
they are not already present on your system.
You can refer to the Ubuntu-based installation commands in
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L11)
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L7)
for guidance.
1. Install `uv` package manager, then create and activate a virtual environment:
python/pyproject_cpu.toml
+2 -1
View File
@@ -49,6 +49,7 @@ dependencies = [
"pybase64",
"pydantic",
"python-multipart",
"pytest",
"pyzmq>=25.1.2",
"requests",
"scipy",
@@ -69,6 +70,7 @@ dependencies = [
"uvicorn",
"uvloop",
"xgrammar==0.2.1",
"zstandard",
]
[project.optional-dependencies]
@@ -105,7 +107,6 @@ test = [
"matplotlib",
"pandas",
"peft>=0.18.0",
"pytest",
"sentence_transformers",
]
all = []
python/sglang/srt/layers/quantization/mxfp4.py
+58 -25
View File
@@ -857,6 +857,29 @@ class Mxfp4MoEMethod(FusedMoEMethodBase):
layer.w2_weight_bias = Parameter(
layer.w2_weight_bias.float(), requires_grad=False
)
return
# Fallback if the TP-sharded layer cannot be AMX-packed
from sglang.srt.layers.quantization.mxfp4_tensor import MXFP4QuantizeUtil
w13_weight = MXFP4QuantizeUtil.dequantize(
quantized_data=layer.w13_weight,
dtype=torch.bfloat16,
scale=layer.w13_weight_scale,
block_sizes=[32],
)
w2_weight = MXFP4QuantizeUtil.dequantize(
quantized_data=layer.w2_weight,
dtype=torch.bfloat16,
scale=layer.w2_weight_scale,
block_sizes=[32],
)
del layer.w13_weight
del layer.w2_weight
del layer.w13_weight_scale
del layer.w2_weight_scale
layer.w13_weight = Parameter(w13_weight, requires_grad=False)
layer.w2_weight = Parameter(w2_weight, requires_grad=False)
return
else:
from triton_kernels.numerics_details.mxfp import upcast_from_mxfp
@@ -1085,32 +1108,42 @@ class Mxfp4MoEMethod(FusedMoEMethodBase):
x = dispatch_output.hidden_states
topk_output = dispatch_output.topk_output
if use_intel_amx_backend(layer):
from sglang.srt.layers.moe.topk import apply_topk_weights_cpu
if _is_cpu:
if use_intel_amx_backend(layer):
from sglang.srt.layers.moe.topk import apply_topk_weights_cpu
topk_weights, topk_ids, _ = dispatch_output.topk_output
x, topk_weights = apply_topk_weights_cpu(
self.moe_runner_config.apply_router_weight_on_input, topk_weights, x
)
output = torch.ops.sgl_kernel.fused_experts_cpu(
x,
layer.w13_weight,
layer.w2_weight,
topk_weights,
topk_ids,
False, # inplace See [Note] inplace should be False in fused_experts.
CPUQuantMethod.MXFP4,
layer.w13_weight_scale, # w1_scale
layer.w2_weight_scale, # w2_scale
None, # w1_zp
None, # w2_zp
None, # block_size
getattr(layer, "w13_weight_bias", None),
getattr(layer, "w2_weight_bias", None),
layer.moe_runner_config.gemm1_alpha,
layer.moe_runner_config.gemm1_clamp_limit,
True, # is_vnni
)
topk_weights, topk_ids, _ = dispatch_output.topk_output
x, topk_weights = apply_topk_weights_cpu(
self.moe_runner_config.apply_router_weight_on_input, topk_weights, x
)
output = torch.ops.sgl_kernel.fused_experts_cpu(
x,
layer.w13_weight,
layer.w2_weight,
topk_weights,
topk_ids,
False, # inplace See [Note] inplace should be False in fused_experts.
CPUQuantMethod.MXFP4,
layer.w13_weight_scale, # w1_scale
layer.w2_weight_scale, # w2_scale
None, # w1_zp
None, # w2_zp
None, # block_size
getattr(layer, "w13_weight_bias", None),
getattr(layer, "w2_weight_bias", None),
layer.moe_runner_config.gemm1_alpha,
layer.moe_runner_config.gemm1_clamp_limit,
True, # is_vnni
)
else:
from sglang.srt.layers.moe.fused_moe_native import moe_forward_native
output = moe_forward_native(
layer,
x,
topk_output,
self.moe_runner_config,
)
return StandardCombineInput(hidden_states=output)
if self.use_marlin:
python/sglang/srt/model_executor/cuda_graph_buffer_registry.py
+4 -1
View File
@@ -514,6 +514,7 @@ def build_decode_registry(
enable_mamba_track: bool = False,
is_encoder_decoder: bool = False,
encoder_len_fill_value: int = 0,
encoder_lens_dtype: torch.dtype = torch.int32,
enable_num_token_non_padded: bool = False,
require_gathered_buffer: bool = False,
enable_prefill_cp: bool = False,
@@ -632,7 +633,7 @@ def build_decode_registry(
GraphSlot(
"encoder_lens",
_bs,
torch.int32,
encoder_lens_dtype,
axis="bs",
padding_policy=PaddingPolicy.FILL_ONCE,
pad_value=encoder_len_fill_value,
@@ -898,6 +899,7 @@ def build_eager_registry(
enable_mamba_track: bool = False,
is_encoder_decoder: bool = False,
encoder_len_fill_value: int = 0,
encoder_lens_dtype: torch.dtype = torch.int32,
dp_size: int = 1,
) -> CudaGraphBufferRegistry:
"""One fixed-max input registry for the ``EagerRunner``, serving BOTH eager
@@ -924,6 +926,7 @@ def build_eager_registry(
enable_mamba_track=enable_mamba_track,
is_encoder_decoder=is_encoder_decoder,
encoder_len_fill_value=encoder_len_fill_value,
encoder_lens_dtype=encoder_lens_dtype,
enable_num_token_non_padded=False,
register_global_num_tokens=False,
require_gathered_buffer=False,
python/sglang/srt/model_executor/runner/eager_runner.py
+3
View File
@@ -130,6 +130,9 @@ class EagerRunner(BaseRunner):
if is_encoder_decoder
else 0
),
encoder_lens_dtype=(
torch.int64 if torch.device(mr.device).type == "cpu" else torch.int32
),
dp_size=sa.dp_size,
)
# Eager has no capture step, so warm up here (run-once via mr._kernel_warmed_up).