diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx index ac00fba81..f87657a83 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx @@ -1,7 +1,9 @@ --- title: "How to Support New Models" -description: "This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations." +metatags: + description: "This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations." --- + This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations. @@ -33,7 +35,7 @@ standard LLM support: 3. **Multimodal Data Processor**: Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your model’s dedicated processor. - See [multimodal_processor.py](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors) + See [Multimodal Processors](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors) for more details. 4. **Handle Multimodal Tokens**: @@ -60,34 +62,6 @@ Refer to `vit_npu_graph_runner.py` under `hardware_backend/npu/graph_runner/` an Please note all your testing and benchmarking results in PR description. -### Interactive Debugging - -For interactive debugging, compare the outputs of Hugging Face/Transformers and SGLang. The following two commands -should give the same text output and very similar prefill logits: - -- Get the reference output: - ```bash Command - python3 scripts/playground/reference_hf.py --model-path [new model] --model-type {text,vlm} - ``` -- Get the SGLang output: - ```bash Command - python3 -m sglang.bench_one_batch --correct --model [new model] - ``` - -### Add the Model to the Test Suite - -To ensure the new model is well maintained, add it to the test suite by including it in the `ALL_OTHER_MODELS` list in -the [test_generation_models.py](https://github.com/sgl-project/sglang/blob/main/test/registered/models/test_generation_models.py) -file, test the new model on your local machine and report the results on demonstrative benchmarks (GSM8K, MMLU, MMMU, -MMMU-Pro, etc.) in your PR. -For VLMs, also include a test in `test_vision_openai_server_{x}.py` (e.g. [test_vision_openai_server_a.py](https://github.com/sgl-project/sglang/blob/main/test/registered/vlm/test_vision_openai_server_a.py)). - -This is an example command to run to test a new model on your local machine: - -```bash Run Test -ONLY_RUN=Qwen/Qwen2-1.5B python3 -m unittest test_generation_models.TestGenerationModels.test_others -``` - ### Benchmark - **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow https://docs.sglang.io/developer_guide/benchmark_and_profiling.html to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). @@ -113,10 +87,12 @@ To port a model from vLLM to SGLang: - The major differences include: - **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`). - **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.** - - **Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.** + - **(For multimodal models) Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.** - **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers. - - **Remove `Sample`.** - - **Change the `forward()` functions** and add a `forward_batch()` method. + - **Change the `forward()` functions** to accept a `forward_batch: ForwardBatch` argument carrying the full batch + state (positions, `input_ids`, KV cache indices, sampling info, etc.). SGLang’s top-level model `forward()` is the + entry point and returns the final `LogitsProcessorOutput` directly — it does both the backbone forward and the + logits computation, unlike vLLM which splits these into `forward()` + `compute_logits()`. - **Add `EntryClass`** at the end. - **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components. - **For Ascend NPU**: Reference existing NPU-adapted models (e.g., `llama.py`, `deepseek_v2.py`) for NPU-specific @@ -157,7 +133,7 @@ launch_server(server_args) ## Example: Implementing and Serving a Llama Wrapper Model -Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](../basic_usage/offline_engine_api). +Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](/docs/basic_usage/offline_engine_api). ### Implementing Our Model @@ -275,10 +251,9 @@ llama_dir = snapshot_download( Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`. That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model. -```python Example +```json Example { "architectures": [ - # "LlamaForCausalLM" "LlamaWrapper" ], ... @@ -538,7 +513,7 @@ python -m sglang.launch_server \ ## Documentation -Add to table of supported models in [generative_models.md](./generative_models) or [multimodal_language_models.md](./multimodal_language_models) +Add to table of supported models in [generative_models.mdx](/docs/supported-models/generative_models) or [multimodal_language_models.mdx](/docs/supported-models/multimodal_language_models) For NPU-adapted models, also add entries to the NPU support models table in