update quantization code owner and document quantization contributions (#26784)
Co-authored-by: ronnie_zheng <zl19940307@163.com>
This commit is contained in:
co-authored by
ronnie_zheng
parent
8524678889
commit
c98d31143d
+3
-2
@@ -32,7 +32,8 @@
|
||||
/python/sglang/srt/hardware_backend/mlx @yeahdongcn
|
||||
/python/sglang/srt/hardware_backend/musa @yeahdongcn
|
||||
/python/sglang/srt/hardware_backend/npu @ping1jing2 @iforgetmyname @whybeyoung
|
||||
/python/sglang/srt/hardware_backend/npu/quantization @OrangeRedeng @TamirBaydasov @iforgetmyname
|
||||
/python/sglang/srt/hardware_backend/gpu/quantization @Alisehen
|
||||
/python/sglang/srt/hardware_backend/npu/quantization @OrangeRedeng @TamirBaydasov @Alisehen
|
||||
/python/sglang/srt/layers @merrymercy @Ying1123 @Fridge003 @ispobock @HaiShaw @ch-wan @BBuf @Edwardf0t1
|
||||
/python/sglang/srt/layers/attention @merrymercy @Fridge003 @ispobock @Qiaolin-Yu @hebiao064 @HaiShaw
|
||||
/python/sglang/srt/layers/attention/fla @yizhang2077 @hebiao064 @yuan-luo
|
||||
@@ -40,7 +41,7 @@
|
||||
/python/sglang/srt/layers/attention/mamba @yizhang2077 @hebiao064
|
||||
/python/sglang/srt/layers/attention/dsa @1am9trash @hubertlu-tw @kkHuang-amd @HaiShaw @Fridge003 @YAMY1234 @rainj-me
|
||||
/python/sglang/srt/layers/attention/vision.py @mickqian @yuan-luo @yhyang201
|
||||
/python/sglang/srt/layers/quantization @ch-wan @BBuf @Edwardf0t1 @FlamingoPg @AniZpZ @HaiShaw @b8zhong
|
||||
/python/sglang/srt/layers/quantization @ch-wan @BBuf @Edwardf0t1 @FlamingoPg @AniZpZ @HaiShaw @b8zhong @OrangeRedeng @TamirBaydasov @Alisehen
|
||||
/python/sglang/srt/layers/quantization/quark @kkHuang-amd @yichiche @hubertlu-tw @1am9trash @BowenBao
|
||||
/python/sglang/srt/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
|
||||
/python/sglang/srt/managers @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann
|
||||
|
||||
+2
-1
@@ -809,7 +809,8 @@
|
||||
"group": "Development",
|
||||
"pages": [
|
||||
"docs/developer_guide/development_guide_using_docker",
|
||||
"docs/developer_guide/development_jit_kernel_guide"
|
||||
"docs/developer_guide/development_jit_kernel_guide",
|
||||
"docs/developer_guide/quantization_contribution_guide"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -6,6 +6,7 @@ description: Contributing to SGLang — development setup, benchmarking, and eva
|
||||
- [Contribution Guide](./contribution_guide)
|
||||
- [Development Guide (Docker)](./development_guide_using_docker)
|
||||
- [JIT Kernels](./development_jit_kernel_guide)
|
||||
- [Quantization Contribution Guide](./quantization_contribution_guide)
|
||||
- [Benchmark and Profiling](./benchmark_and_profiling)
|
||||
- [Bench Serving](./bench_serving)
|
||||
- [Evaluating New Models](./evaluating_new_models)
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: "Quantization Contribution Guide"
|
||||
metatags:
|
||||
description: "Guidance for contributing SGLang quantization methods, schemes, backend kernels, tests, and validation results."
|
||||
---
|
||||
|
||||
This guide describes how to add or refactor quantization support in SGLang. It focuses on the common structure used by weight-only and weight-activation quantization methods such as AWQ, GPTQ, compressed-tensors, ModelSlim, Quark, and related backend kernels.
|
||||
|
||||
## Design Goals
|
||||
|
||||
Quantization code should keep the quantization format semantics separate from hardware-specific execution. This makes it easier to add new formats, reuse kernels across formats, and review platform-specific changes independently.
|
||||
|
||||
Follow the architecture proposed in [Quantization Modifications](https://github.com/sgl-project/sglang/issues/15194):
|
||||
|
||||
- **Config**: parses model and runtime quantization parameters, validates supported options, and selects the proper scheme.
|
||||
- **Scheme**: owns quantized weight creation, weight loading, post-processing, and quantized layer wiring for Linear, MoE, embedding, or other module types.
|
||||
- **Backend kernel**: wraps hardware-specific execution, layout conversion, kernel selection, and kernel calls for GPU (CUDA/HIP/XPU), NPU, or other backends.
|
||||
|
||||
Avoid putting config parsing, weight loading, and backend kernel calls in a single monolithic file. If a method needs multiple formats or backends, add a package under `python/sglang/srt/layers/quantization/<method>/` and split schemes into `schemes/`.
|
||||
|
||||
## Recommended File Layout
|
||||
|
||||
Use this layout for a quantization method that has multiple schemes or backend-specific execution paths:
|
||||
|
||||
```text
|
||||
python/sglang/srt/layers/quantization/<method>/
|
||||
__init__.py
|
||||
<method>.py
|
||||
schemes/
|
||||
__init__.py
|
||||
<method>_scheme.py
|
||||
<method>_linear.py
|
||||
<method>_moe.py
|
||||
<method>_<variant>.py
|
||||
```
|
||||
|
||||
Backend kernels should live under the hardware backend they target:
|
||||
|
||||
```text
|
||||
python/sglang/srt/hardware_backend/gpu/quantization/<method>_kernels.py
|
||||
python/sglang/srt/hardware_backend/npu/quantization/<method>_kernels.py
|
||||
```
|
||||
|
||||
Keep shared method selection in the quantization package and keep backend imports narrow. This prevents circular imports and keeps non-target platforms from importing unavailable kernel dependencies.
|
||||
|
||||
## Adding or Refactoring a Quantization Method
|
||||
|
||||
1. Define the config entry point and register it through `python/sglang/srt/layers/quantization/__init__.py` when needed.
|
||||
2. Add explicit scheme selection helpers such as `get_linear_scheme` and `get_moe_scheme`.
|
||||
3. Move layer-specific weight creation and weight loading into scheme classes.
|
||||
4. Move GPU (CUDA/HIP/XPU), NPU, or other hardware kernel calls into backend kernel modules.
|
||||
5. Keep Linear, MoE, embedding, and non-linear module handling explicit. Do not assign a Linear quantization method to a module type that needs different semantics.
|
||||
6. Preserve compatibility for existing quantized checkpoints and runtime flags.
|
||||
7. Add tests that cover both config parsing and execution paths touched by the change.
|
||||
|
||||
For examples, see the AWQ and GPTQ refactors:
|
||||
|
||||
- [PR #21126](https://github.com/sgl-project/sglang/pull/21126): splits AWQ schemes, weight initialization, and backend kernel calls.
|
||||
- [PR #26402](https://github.com/sgl-project/sglang/pull/26402): applies the same scheme/kernel split to GPTQ.
|
||||
|
||||
## Tests and Validation
|
||||
|
||||
Quantization changes can affect both accuracy and performance. Include validation that matches the blast radius of the change.
|
||||
|
||||
For Python-only structure changes:
|
||||
|
||||
```bash
|
||||
ruff check <changed-python-files>
|
||||
git diff --check
|
||||
```
|
||||
|
||||
For quantized model behavior:
|
||||
|
||||
- Launch at least one representative model for each touched quantization method.
|
||||
- Send a `/generate` request and confirm the output path succeeds.
|
||||
- Run an accuracy sanity test if the change can affect numerics.
|
||||
- Include warmup-aware benchmark results when the change affects kernel calls, layout conversion, or dispatch.
|
||||
|
||||
For backend-specific changes:
|
||||
|
||||
- Validate GPU changes on a supported GPU environment (NVIDIA, AMD, or Intel).
|
||||
- Validate NPU changes on a supported Ascend environment.
|
||||
- Include the exact model, quantization flag, backend flag, hardware, and command used in the PR description.
|
||||
|
||||
## PR Checklist
|
||||
|
||||
Before requesting review, make sure the PR description includes:
|
||||
|
||||
- The quantization method and backend paths changed.
|
||||
- The issue, design proposal, or roadmap item the PR follows.
|
||||
- Any compatibility notes for existing checkpoints or flags.
|
||||
- Accuracy results when model outputs can change.
|
||||
- Benchmark or profiling results when runtime performance can change.
|
||||
- The exact local checks and model launch tests that were run.
|
||||
|
||||
Use the general [Contribution Guide](./contribution_guide) for source setup, formatting, unit tests, CI triggering, and review process details.
|
||||
Reference in New Issue
Block a user