minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Delegate ModelExpress loading to package (#24723)

Browse Source 
Signed-off-by: Zheng Luo <zheluo@nvidia.com>
This commit is contained in:
Zheng Luo
2026-05-16 11:16:44 -07:00
committed by GitHub
parent 0be539024f
commit 435ea41cf0
7 changed files with 60 additions and 503 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/advanced_features/rfork.mdx
+5 -11
View File
@@ -47,7 +47,7 @@ To learn more details about R-Fork, please check **<a href="https://lmsys.org/bl
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>modelexpress-config</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>JSON config for <code>modelexpress</code> backend. Keys: <code>"url"</code> (required, gRPC host:port of ModelExpress server), <code>"model_name"</code> (optional, defaults to <code>--model-path</code>), <code>"source"</code> (optional bool, <code>true</code> for seed mode).</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>JSON config for <code>modelexpress</code> backend. Keys: <code>"url"</code> (optional gRPC host:port override) and <code>"transport"</code> (<code>"nixl"</code> or <code>"transfer_engine"</code>, defaults to <code>"nixl"</code>).</td>
</tr>
</tbody>
</table>
@@ -87,22 +87,16 @@ python -m sglang.launch_server [args] \
### ModelExpress as backend
[ModelExpress](https://github.com/ai-dynamo/modelexpress) is a coordination service that manages P2P weight transfer metadata. It removes the need for direct seed IP/port configuration by providing a centralized registry that seeds publish to and clients discover from. Under the hood it uses TransferEngine (Mooncake) for the actual RDMA data transfer.
[ModelExpress](https://github.com/ai-dynamo/modelexpress) is a coordination service that manages P2P weight transfer metadata. It removes the need for direct seed IP/port configuration by providing a centralized registry that instances publish to and discover from. The ModelExpress Python package must be installed in the SGLang image.
A running ModelExpress server is required. See the [ModelExpress documentation](https://github.com/ai-dynamo/modelexpress) for setup instructions.
seed instance:
```bash Command
python -m sglang.launch_server [args] \
--modelexpress-config '{"url": "[modelexpress_grpc_host:port]", "model_name": "[model_name]", "source": true}'
```
client instance:
server instance:
```bash Command
python -m sglang.launch_server [args] \
--load-format remote_instance \
--remote-instance-weight-loader-backend modelexpress \
--modelexpress-config '{"url": "[modelexpress_grpc_host:port]", "model_name": "[model_name]"}'
--modelexpress-config '{"url": "[modelexpress_grpc_host:port]", "transport": "nixl"}'
```
The seed publishes its TransferEngine session ID and tensor layout to ModelExpress. The client queries ModelExpress to discover the seed, then pulls weights directly via RDMA. This enables dynamic seed discovery without hardcoding IPs, and supports multiple models through a single ModelExpress instance.
All SGLang instances use the same command shape. If no ready source exists, the instance loads weights natively and publishes metadata to ModelExpress. If a compatible source exists, it loads weights through ModelExpress P2P transfer. Set <code>"transport": "transfer_engine"</code> to use Mooncake TransferEngine instead of the default NIXL transport.