minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[CPU] Fix model failures on Xeon (#29497)

Browse Source
This commit is contained in:
Zaili Wang
2026-07-02 13:20:18 +08:00
committed by GitHub
parent aff44d748d
commit cb06c4e6ce
6 changed files with 86 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/xeon.Dockerfile
+1 -2
View File
@@ -42,8 +42,7 @@ RUN source /opt/.venv/bin/activate && \
uv pip install . && \ uv pip install . && \
cd ../sgl-kernel && \ cd ../sgl-kernel && \
cp pyproject_cpu.toml pyproject.toml && \ cp pyproject_cpu.toml pyproject.toml && \
uv pip install . && \ uv pip install .
uv pip install pytest
ENV SGLANG_USE_CPU_ENGINE=1 ENV SGLANG_USE_CPU_ENGINE=1
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4:/usr/lib/x86_64-linux-gnu/libtbbmalloc.so:/opt/.venv/lib/libiomp5.so ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4:/usr/lib/x86_64-linux-gnu/libtbbmalloc.so:/opt/.venv/lib/libiomp5.so
docs_new/docs/hardware-platforms/cpu_server.mdx
+18 -69
View File
@@ -5,82 +5,31 @@ The document addresses how to set up the [SGLang](https://github.com/sgl-project
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions, SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
which are 4th generation or newer Intel® Xeon® Scalable Processors. which are 4th generation or newer Intel® Xeon® Scalable Processors.
## Optimized Model List A number of popular LLMs are optimized and run efficiently on CPU,
A list of popular LLMs are optimized and run efficiently on CPU,
including the most notable open-source models like Llama series, Qwen series, including the most notable open-source models like Llama series, Qwen series,
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus. and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}> in which the support status and example commands can be found.
<colgroup>
<col style={{width: "22%"}} />
<col style={{width: "26%"}} />
<col style={{width: "30%"}} />
<col style={{width: "22%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8_INT8</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8">meituan/DeepSeek-R1-Channel-INT8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-R1">deepseek-ai/DeepSeek-R1</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1-Terminus</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8">IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus">deepseek-ai/DeepSeek-V3.1-Terminus</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct">meta-llama/Llama-3.2-3B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8">RedHatAI/Llama-3.2-3B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct">meta-llama/Llama-3.1-8B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8">RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QwQ-32B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8">RedHatAI/QwQ-32B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-Distilled-Llama</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8">RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8">Qwen/Qwen3-235B-A22B-FP8</a></td>
</tr>
</tbody>
</table>
**Note:** The model identifiers listed in the table above
have been verified on 6th Gen Intel® Xeon® P-core platforms.
## Installation ## Installation
### Install Using Docker ### Install Using Docker
It is recommended to use Docker for setting up the SGLang environment. It is recommended to use Docker for setting up the SGLang environment.
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation.
#### Pull from Docker Hub
Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository.
The [CPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xeon) end with `xeon` suffix.
The image pulling command is like:
```bash Command
docker pull lmsysorg/sglang:v0.5.13-xeon
```
#### Build from Dockerfile
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation from latest source code.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
```bash Command ```bash Command
@@ -112,7 +61,7 @@ the setup process is as follows:
Please install the required packages and libraries beforehand if Please install the required packages and libraries beforehand if
they are not already present on your system. they are not already present on your system.
You can refer to the Ubuntu-based installation commands in You can refer to the Ubuntu-based installation commands in
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L11) [the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L7)
for guidance. for guidance.
1. Install `uv` package manager, then create and activate a virtual environment: 1. Install `uv` package manager, then create and activate a virtual environment:
python/pyproject_cpu.toml
+2 -1
View File
@@ -49,6 +49,7 @@ dependencies = [
"pybase64", "pybase64",
"pydantic", "pydantic",
"python-multipart", "python-multipart",
"pytest",
"pyzmq>=25.1.2", "pyzmq>=25.1.2",
"requests", "requests",
"scipy", "scipy",
@@ -69,6 +70,7 @@ dependencies = [
"uvicorn", "uvicorn",
"uvloop", "uvloop",
"xgrammar==0.2.1", "xgrammar==0.2.1",
"zstandard",
] ]
[project.optional-dependencies] [project.optional-dependencies]
@@ -105,7 +107,6 @@ test = [
"matplotlib", "matplotlib",
"pandas", "pandas",
"peft>=0.18.0", "peft>=0.18.0",
"pytest",
"sentence_transformers", "sentence_transformers",
] ]
all = [] all = []
python/sglang/srt/layers/quantization/mxfp4.py
+58 -25
View File
@@ -857,6 +857,29 @@ class Mxfp4MoEMethod(FusedMoEMethodBase):
layer.w2_weight_bias = Parameter( layer.w2_weight_bias = Parameter(
layer.w2_weight_bias.float(), requires_grad=False layer.w2_weight_bias.float(), requires_grad=False
) )
return
# Fallback if the TP-sharded layer cannot be AMX-packed
from sglang.srt.layers.quantization.mxfp4_tensor import MXFP4QuantizeUtil
w13_weight = MXFP4QuantizeUtil.dequantize(
quantized_data=layer.w13_weight,
dtype=torch.bfloat16,
scale=layer.w13_weight_scale,
block_sizes=[32],
)
w2_weight = MXFP4QuantizeUtil.dequantize(
quantized_data=layer.w2_weight,
dtype=torch.bfloat16,
scale=layer.w2_weight_scale,
block_sizes=[32],
)
del layer.w13_weight
del layer.w2_weight
del layer.w13_weight_scale
del layer.w2_weight_scale
layer.w13_weight = Parameter(w13_weight, requires_grad=False)
layer.w2_weight = Parameter(w2_weight, requires_grad=False)
return return
else: else:
from triton_kernels.numerics_details.mxfp import upcast_from_mxfp from triton_kernels.numerics_details.mxfp import upcast_from_mxfp
@@ -1085,32 +1108,42 @@ class Mxfp4MoEMethod(FusedMoEMethodBase):
x = dispatch_output.hidden_states x = dispatch_output.hidden_states
topk_output = dispatch_output.topk_output topk_output = dispatch_output.topk_output
if use_intel_amx_backend(layer): if _is_cpu:
from sglang.srt.layers.moe.topk import apply_topk_weights_cpu if use_intel_amx_backend(layer):
from sglang.srt.layers.moe.topk import apply_topk_weights_cpu
topk_weights, topk_ids, _ = dispatch_output.topk_output topk_weights, topk_ids, _ = dispatch_output.topk_output
x, topk_weights = apply_topk_weights_cpu( x, topk_weights = apply_topk_weights_cpu(
self.moe_runner_config.apply_router_weight_on_input, topk_weights, x self.moe_runner_config.apply_router_weight_on_input, topk_weights, x
) )
output = torch.ops.sgl_kernel.fused_experts_cpu( output = torch.ops.sgl_kernel.fused_experts_cpu(
x, x,
layer.w13_weight, layer.w13_weight,
layer.w2_weight, layer.w2_weight,
topk_weights, topk_weights,
topk_ids, topk_ids,
False, # inplace See [Note] inplace should be False in fused_experts. False, # inplace See [Note] inplace should be False in fused_experts.
CPUQuantMethod.MXFP4, CPUQuantMethod.MXFP4,
layer.w13_weight_scale, # w1_scale layer.w13_weight_scale, # w1_scale
layer.w2_weight_scale, # w2_scale layer.w2_weight_scale, # w2_scale
None, # w1_zp None, # w1_zp
None, # w2_zp None, # w2_zp
None, # block_size None, # block_size
getattr(layer, "w13_weight_bias", None), getattr(layer, "w13_weight_bias", None),
getattr(layer, "w2_weight_bias", None), getattr(layer, "w2_weight_bias", None),
layer.moe_runner_config.gemm1_alpha, layer.moe_runner_config.gemm1_alpha,
layer.moe_runner_config.gemm1_clamp_limit, layer.moe_runner_config.gemm1_clamp_limit,
True, # is_vnni True, # is_vnni
) )
else:
from sglang.srt.layers.moe.fused_moe_native import moe_forward_native
output = moe_forward_native(
layer,
x,
topk_output,
self.moe_runner_config,
)
return StandardCombineInput(hidden_states=output) return StandardCombineInput(hidden_states=output)
if self.use_marlin: if self.use_marlin:
python/sglang/srt/model_executor/cuda_graph_buffer_registry.py
+4 -1
View File
@@ -514,6 +514,7 @@ def build_decode_registry(
enable_mamba_track: bool = False, enable_mamba_track: bool = False,
is_encoder_decoder: bool = False, is_encoder_decoder: bool = False,
encoder_len_fill_value: int = 0, encoder_len_fill_value: int = 0,
encoder_lens_dtype: torch.dtype = torch.int32,
enable_num_token_non_padded: bool = False, enable_num_token_non_padded: bool = False,
require_gathered_buffer: bool = False, require_gathered_buffer: bool = False,
enable_prefill_cp: bool = False, enable_prefill_cp: bool = False,
@@ -632,7 +633,7 @@ def build_decode_registry(
GraphSlot( GraphSlot(
"encoder_lens", "encoder_lens",
_bs, _bs,
torch.int32, encoder_lens_dtype,
axis="bs", axis="bs",
padding_policy=PaddingPolicy.FILL_ONCE, padding_policy=PaddingPolicy.FILL_ONCE,
pad_value=encoder_len_fill_value, pad_value=encoder_len_fill_value,
@@ -898,6 +899,7 @@ def build_eager_registry(
enable_mamba_track: bool = False, enable_mamba_track: bool = False,
is_encoder_decoder: bool = False, is_encoder_decoder: bool = False,
encoder_len_fill_value: int = 0, encoder_len_fill_value: int = 0,
encoder_lens_dtype: torch.dtype = torch.int32,
dp_size: int = 1, dp_size: int = 1,
) -> CudaGraphBufferRegistry: ) -> CudaGraphBufferRegistry:
"""One fixed-max input registry for the ``EagerRunner``, serving BOTH eager """One fixed-max input registry for the ``EagerRunner``, serving BOTH eager
@@ -924,6 +926,7 @@ def build_eager_registry(
enable_mamba_track=enable_mamba_track, enable_mamba_track=enable_mamba_track,
is_encoder_decoder=is_encoder_decoder, is_encoder_decoder=is_encoder_decoder,
encoder_len_fill_value=encoder_len_fill_value, encoder_len_fill_value=encoder_len_fill_value,
encoder_lens_dtype=encoder_lens_dtype,
enable_num_token_non_padded=False, enable_num_token_non_padded=False,
register_global_num_tokens=False, register_global_num_tokens=False,
require_gathered_buffer=False, require_gathered_buffer=False,
python/sglang/srt/model_executor/runner/eager_runner.py
+3
View File
@@ -130,6 +130,9 @@ class EagerRunner(BaseRunner):
if is_encoder_decoder if is_encoder_decoder
else 0 else 0
), ),
encoder_lens_dtype=(
torch.int64 if torch.device(mr.device).type == "cpu" else torch.int32
),
dp_size=sa.dp_size, dp_size=sa.dp_size,
) )
# Eager has no capture step, so warm up here (run-once via mr._kernel_warmed_up). # Eager has no capture step, so warm up here (run-once via mr._kernel_warmed_up).