minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

docs(nixl): document OBJ throughput target (#30405)

Browse Source
This commit is contained in:
Lukas Humbel
2026-08-13 13:04:13 -07:00
committed by GitHub
parent 29b067245b
commit 8ad04a9bee
2 changed files with 21 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/sglang/srt/mem_cache/storage/nixl/README.md
+13 -1
View File
@@ -628,7 +628,7 @@ Configures the 3FS (third-party filesystem) backend.
#### Description
Configures an object storage backend compatible with S3 APIs (e.g., AWS S3, MinIO, Ceph).
Configures an object storage backend compatible with S3 APIs (e.g., AWS S3, MinIO, Ceph). See the [NIXL OBJ plugin documentation](https://github.com/ai-dynamo/nixl/blob/main/src/plugins/obj/README.md) for backend-specific options and behavior.
#### Configuration Keys
@@ -637,6 +637,8 @@ Configures an object storage backend compatible with S3 APIs (e.g., AWS S3, MinI
| ------------------------ | ------- | ------------ | ---------------------------------------------- |
| `num_threads` | integer | `4` | Number of client worker threads. |
| `endpoint_override` | string | `""` | Custom endpoint URL (for non-AWS S3 services). |
| `crtMinLimit` | integer | disabled | Minimum object size in bytes for S3 CRT. |
| `throughput_target_gbps` | integer | `10` | Whole-Gbps target for S3 CRT. |
| `scheme` | string | `"http"` | Connection scheme (`http` or `https`). |
| `region` | string | `""` | Cloud region (if applicable). |
| `req_checksum` | string | `"required"` | Request checksum behavior. |
@@ -648,6 +650,16 @@ Configures an object storage backend compatible with S3 APIs (e.g., AWS S3, MinI
| `bucket` | string | `""` | Default bucket name. |
| `active` | boolean | N/A | Controls whether this plugin is eligible for backend selection. |
`throughput_target_gbps` only affects transfers routed through the S3 CRT client. Set `crtMinLimit` to enable the CRT path for objects at or above the chosen size. The throughput target must be a whole number. Values below the 5 MiB AWS S3 minimum multipart part size are accepted for `crtMinLimit`, but the AWS CRT SDK clamps the part size to 5 MiB.
To saturate a high-bandwidth link with low tensor parallelism, set `SGLANG_AUTO_NUMA_BIND=false`. Disabling automatic NUMA pinning allows the CRT thread pool to use all available CPU cores for workers.
For example, the following inline configuration uses the CRT client for objects of at least 5 MiB and targets 25 Gbps:
```bash
--hicache-storage-backend-extra-config '{"plugin":{"obj":{"active":true,"crtMinLimit":"5242880","throughput_target_gbps":"25"}}}'
```
##### `req_checksum` Valid Values
| Value | Description |
python/sglang/srt/mem_cache/storage/nixl/nixl.config.toml.sample
+8
View File
@@ -108,6 +108,14 @@ num_threads = 4
# Override endpoint (useful for non-AWS S3 services)
endpoint_override = ""
# Minimum object size (bytes) that uses the high-performance S3 CRT client.
# Disabled by default. AWS S3 multipart uploads require parts of at least 5 MiB.
crtMinLimit = 5242880
# Target throughput for the S3 CRT client in whole Gbps, default: 10.
# The CRT scheduler uses this value to size its parallel connection count.
throughput_target_gbps = 25
# Connection scheme: http or https, default: http
scheme = "http"