diff --git a/docs/advanced_features/pd_disaggregation.md b/docs/advanced_features/pd_disaggregation.md index e1edc56b8..7c12c26e0 100644 --- a/docs/advanced_features/pd_disaggregation.md +++ b/docs/advanced_features/pd_disaggregation.md @@ -133,6 +133,12 @@ To enable NVLink transport for KV cache transfers with the mooncake backend (rec export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK export MC_FORCE_MNNVL=True ``` +To utilize Intra-Node NVLink for KV cache transfers with the Mooncake backend (recommended for A100, H20, H100, etc.), set the following environment variables. Please note that auxiliary data still needs to be transferred via TCP. + +```bash +export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK +export MC_INTRANODE_NVLINK=true +``` The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom memory pool. Supported values are `NVLINK` (or `True`), `BAREX`, and `INTRA_NODE_NVLINK`. diff --git a/docs_new/docs/advanced_features/pd_disaggregation.mdx b/docs_new/docs/advanced_features/pd_disaggregation.mdx index 86f4bf025..ff92b55db 100644 --- a/docs_new/docs/advanced_features/pd_disaggregation.mdx +++ b/docs_new/docs/advanced_features/pd_disaggregation.mdx @@ -137,6 +137,12 @@ export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK export MC_FORCE_MNNVL=True ``` +To utilize Intra-Node NVLink for KV cache transfers with the Mooncake backend (recommended for A100, H20, H100, etc.), set the following environment variables. Please note that auxiliary data still needs to be transferred via TCP. + +```bash +export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK +export MC_INTRANODE_NVLINK=true +``` The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom memory pool. Supported values are `NVLINK` (or `True`), `BAREX`, and `INTRA_NODE_NVLINK`. #### Prefill Server Configuration