update quantization code owner and document quantization contributions (#26784)
Co-authored-by: ronnie_zheng <zl19940307@163.com>
This commit is contained in:
co-authored by
ronnie_zheng
parent
8524678889
commit
c98d31143d
+3
-2
@@ -32,7 +32,8 @@
|
|||||||
/python/sglang/srt/hardware_backend/mlx @yeahdongcn
|
/python/sglang/srt/hardware_backend/mlx @yeahdongcn
|
||||||
/python/sglang/srt/hardware_backend/musa @yeahdongcn
|
/python/sglang/srt/hardware_backend/musa @yeahdongcn
|
||||||
/python/sglang/srt/hardware_backend/npu @ping1jing2 @iforgetmyname @whybeyoung
|
/python/sglang/srt/hardware_backend/npu @ping1jing2 @iforgetmyname @whybeyoung
|
||||||
/python/sglang/srt/hardware_backend/npu/quantization @OrangeRedeng @TamirBaydasov @iforgetmyname
|
/python/sglang/srt/hardware_backend/gpu/quantization @Alisehen
|
||||||
|
/python/sglang/srt/hardware_backend/npu/quantization @OrangeRedeng @TamirBaydasov @Alisehen
|
||||||
/python/sglang/srt/layers @merrymercy @Ying1123 @Fridge003 @ispobock @HaiShaw @ch-wan @BBuf @Edwardf0t1
|
/python/sglang/srt/layers @merrymercy @Ying1123 @Fridge003 @ispobock @HaiShaw @ch-wan @BBuf @Edwardf0t1
|
||||||
/python/sglang/srt/layers/attention @merrymercy @Fridge003 @ispobock @Qiaolin-Yu @hebiao064 @HaiShaw
|
/python/sglang/srt/layers/attention @merrymercy @Fridge003 @ispobock @Qiaolin-Yu @hebiao064 @HaiShaw
|
||||||
/python/sglang/srt/layers/attention/fla @yizhang2077 @hebiao064 @yuan-luo
|
/python/sglang/srt/layers/attention/fla @yizhang2077 @hebiao064 @yuan-luo
|
||||||
@@ -40,7 +41,7 @@
|
|||||||
/python/sglang/srt/layers/attention/mamba @yizhang2077 @hebiao064
|
/python/sglang/srt/layers/attention/mamba @yizhang2077 @hebiao064
|
||||||
/python/sglang/srt/layers/attention/dsa @1am9trash @hubertlu-tw @kkHuang-amd @HaiShaw @Fridge003 @YAMY1234 @rainj-me
|
/python/sglang/srt/layers/attention/dsa @1am9trash @hubertlu-tw @kkHuang-amd @HaiShaw @Fridge003 @YAMY1234 @rainj-me
|
||||||
/python/sglang/srt/layers/attention/vision.py @mickqian @yuan-luo @yhyang201
|
/python/sglang/srt/layers/attention/vision.py @mickqian @yuan-luo @yhyang201
|
||||||
/python/sglang/srt/layers/quantization @ch-wan @BBuf @Edwardf0t1 @FlamingoPg @AniZpZ @HaiShaw @b8zhong
|
/python/sglang/srt/layers/quantization @ch-wan @BBuf @Edwardf0t1 @FlamingoPg @AniZpZ @HaiShaw @b8zhong @OrangeRedeng @TamirBaydasov @Alisehen
|
||||||
/python/sglang/srt/layers/quantization/quark @kkHuang-amd @yichiche @hubertlu-tw @1am9trash @BowenBao
|
/python/sglang/srt/layers/quantization/quark @kkHuang-amd @yichiche @hubertlu-tw @1am9trash @BowenBao
|
||||||
/python/sglang/srt/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
|
/python/sglang/srt/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
|
||||||
/python/sglang/srt/managers @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann
|
/python/sglang/srt/managers @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann
|
||||||
|
|||||||
+2
-1
@@ -809,7 +809,8 @@
|
|||||||
"group": "Development",
|
"group": "Development",
|
||||||
"pages": [
|
"pages": [
|
||||||
"docs/developer_guide/development_guide_using_docker",
|
"docs/developer_guide/development_guide_using_docker",
|
||||||
"docs/developer_guide/development_jit_kernel_guide"
|
"docs/developer_guide/development_jit_kernel_guide",
|
||||||
|
"docs/developer_guide/quantization_contribution_guide"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ description: Contributing to SGLang — development setup, benchmarking, and eva
|
|||||||
- [Contribution Guide](./contribution_guide)
|
- [Contribution Guide](./contribution_guide)
|
||||||
- [Development Guide (Docker)](./development_guide_using_docker)
|
- [Development Guide (Docker)](./development_guide_using_docker)
|
||||||
- [JIT Kernels](./development_jit_kernel_guide)
|
- [JIT Kernels](./development_jit_kernel_guide)
|
||||||
|
- [Quantization Contribution Guide](./quantization_contribution_guide)
|
||||||
- [Benchmark and Profiling](./benchmark_and_profiling)
|
- [Benchmark and Profiling](./benchmark_and_profiling)
|
||||||
- [Bench Serving](./bench_serving)
|
- [Bench Serving](./bench_serving)
|
||||||
- [Evaluating New Models](./evaluating_new_models)
|
- [Evaluating New Models](./evaluating_new_models)
|
||||||
|
|||||||
@@ -0,0 +1,96 @@
|
|||||||
|
---
|
||||||
|
title: "Quantization Contribution Guide"
|
||||||
|
metatags:
|
||||||
|
description: "Guidance for contributing SGLang quantization methods, schemes, backend kernels, tests, and validation results."
|
||||||
|
---
|
||||||
|
|
||||||
|
This guide describes how to add or refactor quantization support in SGLang. It focuses on the common structure used by weight-only and weight-activation quantization methods such as AWQ, GPTQ, compressed-tensors, ModelSlim, Quark, and related backend kernels.
|
||||||
|
|
||||||
|
## Design Goals
|
||||||
|
|
||||||
|
Quantization code should keep the quantization format semantics separate from hardware-specific execution. This makes it easier to add new formats, reuse kernels across formats, and review platform-specific changes independently.
|
||||||
|
|
||||||
|
Follow the architecture proposed in [Quantization Modifications](https://github.com/sgl-project/sglang/issues/15194):
|
||||||
|
|
||||||
|
- **Config**: parses model and runtime quantization parameters, validates supported options, and selects the proper scheme.
|
||||||
|
- **Scheme**: owns quantized weight creation, weight loading, post-processing, and quantized layer wiring for Linear, MoE, embedding, or other module types.
|
||||||
|
- **Backend kernel**: wraps hardware-specific execution, layout conversion, kernel selection, and kernel calls for GPU (CUDA/HIP/XPU), NPU, or other backends.
|
||||||
|
|
||||||
|
Avoid putting config parsing, weight loading, and backend kernel calls in a single monolithic file. If a method needs multiple formats or backends, add a package under `python/sglang/srt/layers/quantization/<method>/` and split schemes into `schemes/`.
|
||||||
|
|
||||||
|
## Recommended File Layout
|
||||||
|
|
||||||
|
Use this layout for a quantization method that has multiple schemes or backend-specific execution paths:
|
||||||
|
|
||||||
|
```text
|
||||||
|
python/sglang/srt/layers/quantization/<method>/
|
||||||
|
__init__.py
|
||||||
|
<method>.py
|
||||||
|
schemes/
|
||||||
|
__init__.py
|
||||||
|
<method>_scheme.py
|
||||||
|
<method>_linear.py
|
||||||
|
<method>_moe.py
|
||||||
|
<method>_<variant>.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Backend kernels should live under the hardware backend they target:
|
||||||
|
|
||||||
|
```text
|
||||||
|
python/sglang/srt/hardware_backend/gpu/quantization/<method>_kernels.py
|
||||||
|
python/sglang/srt/hardware_backend/npu/quantization/<method>_kernels.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep shared method selection in the quantization package and keep backend imports narrow. This prevents circular imports and keeps non-target platforms from importing unavailable kernel dependencies.
|
||||||
|
|
||||||
|
## Adding or Refactoring a Quantization Method
|
||||||
|
|
||||||
|
1. Define the config entry point and register it through `python/sglang/srt/layers/quantization/__init__.py` when needed.
|
||||||
|
2. Add explicit scheme selection helpers such as `get_linear_scheme` and `get_moe_scheme`.
|
||||||
|
3. Move layer-specific weight creation and weight loading into scheme classes.
|
||||||
|
4. Move GPU (CUDA/HIP/XPU), NPU, or other hardware kernel calls into backend kernel modules.
|
||||||
|
5. Keep Linear, MoE, embedding, and non-linear module handling explicit. Do not assign a Linear quantization method to a module type that needs different semantics.
|
||||||
|
6. Preserve compatibility for existing quantized checkpoints and runtime flags.
|
||||||
|
7. Add tests that cover both config parsing and execution paths touched by the change.
|
||||||
|
|
||||||
|
For examples, see the AWQ and GPTQ refactors:
|
||||||
|
|
||||||
|
- [PR #21126](https://github.com/sgl-project/sglang/pull/21126): splits AWQ schemes, weight initialization, and backend kernel calls.
|
||||||
|
- [PR #26402](https://github.com/sgl-project/sglang/pull/26402): applies the same scheme/kernel split to GPTQ.
|
||||||
|
|
||||||
|
## Tests and Validation
|
||||||
|
|
||||||
|
Quantization changes can affect both accuracy and performance. Include validation that matches the blast radius of the change.
|
||||||
|
|
||||||
|
For Python-only structure changes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ruff check <changed-python-files>
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
For quantized model behavior:
|
||||||
|
|
||||||
|
- Launch at least one representative model for each touched quantization method.
|
||||||
|
- Send a `/generate` request and confirm the output path succeeds.
|
||||||
|
- Run an accuracy sanity test if the change can affect numerics.
|
||||||
|
- Include warmup-aware benchmark results when the change affects kernel calls, layout conversion, or dispatch.
|
||||||
|
|
||||||
|
For backend-specific changes:
|
||||||
|
|
||||||
|
- Validate GPU changes on a supported GPU environment (NVIDIA, AMD, or Intel).
|
||||||
|
- Validate NPU changes on a supported Ascend environment.
|
||||||
|
- Include the exact model, quantization flag, backend flag, hardware, and command used in the PR description.
|
||||||
|
|
||||||
|
## PR Checklist
|
||||||
|
|
||||||
|
Before requesting review, make sure the PR description includes:
|
||||||
|
|
||||||
|
- The quantization method and backend paths changed.
|
||||||
|
- The issue, design proposal, or roadmap item the PR follows.
|
||||||
|
- Any compatibility notes for existing checkpoints or flags.
|
||||||
|
- Accuracy results when model outputs can change.
|
||||||
|
- Benchmark or profiling results when runtime performance can change.
|
||||||
|
- The exact local checks and model launch tests that were run.
|
||||||
|
|
||||||
|
Use the general [Contribution Guide](./contribution_guide) for source setup, formatting, unit tests, CI triggering, and review process details.
|
||||||
Reference in New Issue
Block a user