[Feature][Intel XPU] Add memory saver support for Intel XPU via upstream torch_memory_saver (#29935)

2026-09-08 09:38:51 +08:00
parent b8a81f055d
commit 2358916d5a
4 changed files with 474 additions and 7 deletions
@@ -326,10 +326,63 @@ via `--cuda-graph-backend-prefill` or `--cuda-graph-config`.

 | Feature | Status |
 |---|---|
-| Memory saver (`--enable-memory-saver`) | Not yet supported |
 | Two-batch overlap (`--enable-two-batch-overlap`) | Not yet supported |
 | Speculative decoding | Not yet implemented |

+## Memory Saver (release/resume memory occupation) on Intel XPU [Experimental]
+
+SGLang can temporarily release most of the GPU memory it holds — model weights
+and/or KV cache — and reclaim it later without restarting the process. This is
+the same `release_memory_occupation` / `resume_memory_occupation` feature
+available on CUDA, used for RL rollout/training hand-off and for freeing the
+device between inference bursts.
+
+This is backed by the [`torch_memory_saver`](https://github.com/fzyzcjy/torch_memory_saver)
+package — the same package used on CUDA — which gained an Intel XPU backend
+built natively on Level Zero (keeping virtual addresses fixed while
+releasing/re-committing physical pages via `zeVirtualMemUnmap` /
+`zeVirtualMemMap`).
+
+**Install `torch_memory_saver`.** Unlike CUDA (prebuilt wheel), the XPU backend
+is built from source against your local oneAPI + `torch+xpu` runtime (the `.so`
+links `libsycl.so.<N>`, which must match the installed `intel-sycl-rt`).
+`TMS_PLATFORM=xpu` forces the XPU backend, and `--no-build-isolation` lets the
+build import your installed `torch` so it can match the `libsycl` major to it:
+
+The published wheels are CUDA-only, so install from git and let it build. The ref
+below is the `v0.0.10b2` release, pinned so the build is reproducible.
+
+```bash
+source /opt/intel/oneapi/setvars.sh
+TMS_PLATFORM=xpu pip install --no-build-isolation \
+    git+https://github.com/fzyzcjy/torch_memory_saver.git@a5c99f11b18ebb8e9fda71a68812e476ae49e417
+```
+
+**Use it** by launching with `--enable-memory-saver` (the XPU backend is
+selected automatically); optionally add `--enable-weights-cpu-backup` to keep
+weights in host RAM across a release:
+
+```bash
+python -m sglang.launch_server --model-path Qwen/Qwen3-0.6B \
+    --trust-remote-code --device xpu --enable-memory-saver
+```
+
+```bash
+# Release GPU memory while idle, then reclaim it (server must be idle).
+curl -X POST http://127.0.0.1:30000/release_memory_occupation
+curl -X POST http://127.0.0.1:30000/resume_memory_occupation
+```
+
+The Python engine API (`engine.release_memory_occupation(tags=...)` /
+`engine.resume_memory_occupation(tags=...)`) and the `weights` / `kv_cache` tags
+behave the same as on CUDA. Pauseable CUDA-graph capture is not used on XPU, so
+the `cuda_graph` tag is a no-op there.
+
+> **Verifying memory was freed:** neither `torch.xpu.memory_allocated()` nor
+> `torch.xpu.mem_get_info()` drops when physical pages are released — the first is
+> allocator accounting, and the second stays flat because torch keeps the freed block
+> cached. Query actual device memory via sysman (`ZES_ENABLE_SYSMAN=1`) instead.
+
 ## Prefill-Decode (P/D) Disaggregation on Intel XPU [Experimental]

 SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend.