[HiCache] Support heterogeneous tp for hicache storage (#18541)

Co-authored-by: hzh0425 <hzh0425@apache.org>

docs/advanced_features/hicache_best_practices.md

@@ -39,6 +39,23 @@ Notes:
 - `page_first`: Only compatible with `kernel` I/O backend, automatically switches to `layer_first` with `direct` backend
 - `page_first_direct`: Specifically designed for `direct` I/O backend with optimized memory organization
 
+### Heterogeneous TP Support (GQA/MHA models)
+
+HiCache storage supports cross-cluster KV reuse when different deployments use different TP sizes (for example, `tp=4` and `tp=8`) and share the same storage backend namespace.
+
+Use `tp_lcm_size` in `--hicache-storage-backend-extra-config`:
+
+```bash
+# Example: heterogeneous TP = {4, 8}, so lcm = 8
+--hicache-storage-backend-extra-config '{"tp_lcm_size": 8}'
+```
+
+Guidelines:
+
+- Set `tp_lcm_size` to the least common multiple (LCM) of all TP sizes that will share the same HiCache storage.
+- For MHA models with Mooncake and `page_head` layout, HiCache will split head shards based on `tp_lcm_size` to make keys reusable across heterogeneous TP deployments.
+- If all clusters use the same TP size, this option is not needed.
+
 ### Prefetch Policies
 
 ```bash