 fzyzcjy
|
62222bd27e
|
Minor tool for comparison of benchmark results (#7974)
|
2025-07-27 00:27:50 -07:00 |
|
|
fzyzcjy
|
ed0fdbf35b
|
Tool to dump and compare internal activation tensors (#7976)
|
2025-07-27 00:27:21 -07:00 |
|
|
fzyzcjy
|
da0c026084
|
Tiny assert EPLB is used together with expert parallel (#8381)
|
2025-07-26 03:20:39 -07:00 |
|
|
fzyzcjy
|
519ff5c8e6
|
Super tiny fix typo (#8046)
|
2025-07-17 21:15:51 +08:00 |
|
|
fzyzcjy
|
c46e069d34
|
Tiny fix mooncake log warning wrong output (#7952)
|
2025-07-12 21:22:44 -07:00 |
|
|
fzyzcjy
|
2a2d3478af
|
Fix wrong gemm branch cause 250us slower (#7969)
|
2025-07-11 19:45:09 -07:00 |
|
|
fzyzcjy
|
880221bd3b
|
Revert "[PD Disaggregation] replace transfer with batch transfer for better performance (#7236)" (#7968)
|
2025-07-11 19:03:01 -07:00 |
|
|
fzyzcjy
|
a291439a59
|
Support logprobs in two-batch overlap (#7709)
|
2025-07-05 19:05:32 -07:00 |
|
|
fzyzcjy
|
0c9c6c75a8
|
Move files related to EPLB (#7580)
|
2025-06-29 15:39:38 -07:00 |
|
|
fzyzcjy
|
00c7b1ad07
|
Let EP prefill support new DeepGEMM (#7310)
|
2025-06-28 01:45:30 -07:00 |
|
|
fzyzcjy
|
82eccae44e
|
Let ep_scatter support arbitrary strides / ue8m0 format (#7309)
|
2025-06-28 01:38:33 -07:00 |
|
|
fzyzcjy
|
9efb2993da
|
Tiny add logs for expert location updater (#7308)
|
2025-06-27 19:12:33 -07:00 |
|
|
fzyzcjy
|
3d7cdb2ebd
|
Fix MTP error when enabling two-batch overlap (#7569)
|
2025-06-26 15:40:54 -07:00 |
|
|
fzyzcjy
|
edc21cc8ae
|
Tiny add logging for GC (#7406)
|
2025-06-22 12:40:02 +08:00 |
|
|
fzyzcjy
|
650127a173
|
Reintroduce tiny fix sampler error when prob is not contiguous (#7354)
|
2025-06-19 09:39:34 -07:00 |
|
|
fzyzcjy
|
9c6a0656a3
|
Fix profiler error when there are idle passes (#7003)
|
2025-06-18 10:55:01 -07:00 |
|
|
fzyzcjy
|
91a066ec6a
|
Tiny remove comments about DeepEP on H20 (#7234)
|
2025-06-16 09:13:57 -07:00 |
|
|
fzyzcjy
|
349bb2c92a
|
Fix error when disabling new DeepGEMM (#7198)
|
2025-06-14 19:24:54 -07:00 |
|
|
fzyzcjy
|
0b8939bc2c
|
Fix NCCL 2.27.3 not in docker image (#7195)
|
2025-06-14 18:36:11 -07:00 |
|
|
fzyzcjy
|
b57d87c297
|
Fix shared experts fusion + weight requant (#7177)
|
2025-06-14 02:35:18 -07:00 |
|
|
fzyzcjy
|
93cec4335f
|
Support new DeepGEMM (#7172)
|
2025-06-13 23:00:17 -07:00 |
|
|
fzyzcjy
|
b4c41f7276
|
Refactor DeepGEMM integration (#7150)
|
2025-06-13 20:41:03 -07:00 |
|
|
fzyzcjy
|
8b8f2e7463
|
Support new DeepGEMM input format in silu_and_mul_masked_post_quant_fwd (#7153)
|
2025-06-13 20:40:24 -07:00 |
|
|
fzyzcjy
|
5b1afa7814
|
Re-quantize DeepSeek model weights to support DeepGEMM new input format (#7156)
|
2025-06-13 15:57:45 -07:00 |
|
|
fzyzcjy
|
c49c1d9226
|
Remove 200us slow concat kernel (part 2: srt) (#7020)
|
2025-06-13 15:19:31 -07:00 |
|
|
fzyzcjy
|
0f1dfa1efe
|
Tiny add sanity checks for DeepGEMM inputs (#7157)
|
2025-06-13 14:36:27 -07:00 |
|
|
fzyzcjy
|
e3ec6bf4b6
|
Minor speed up block_quant_dequant (#6814)
|
2025-06-13 14:32:46 -07:00 |
|
|
fzyzcjy
|
b04df75acd
|
Fix DeepEP error in some environments (#7154)
|
2025-06-13 14:27:25 -07:00 |
|
|
fzyzcjy
|
bec3e48402
|
Support new DeepGEMM format in per token group quant (part 2: srt) (#7155)
|
2025-06-13 14:25:40 -07:00 |
|
|
fzyzcjy
|
5c66c4424f
|
Support new DeepGEMM format in per token group quant (#7146)
|
2025-06-13 02:00:22 -07:00 |
|
|
fzyzcjy
|
aa46ed34d2
|
Remove 200us slow concat kernel (part 1: kernel) (#7145)
|
2025-06-13 01:58:29 -07:00 |
|
|
fzyzcjy
|
da47621ccc
|
Minor speedup topk postprocessing (#7058)
|
2025-06-13 00:50:18 -07:00 |
|
|
fzyzcjy
|
b02df20a8d
|
Minor add utility to read expert distribution recorder output (#7134)
|
2025-06-12 16:50:05 -07:00 |
|
|
fzyzcjy
|
fe55947acd
|
Add sanity checks when a test file is not added to CI (#6947)
|
2025-06-10 12:34:57 -07:00 |
|
|
fzyzcjy
|
19995dd78e
|
Tiny fix cutlass_mla_get_workspace_size stub incorrect signature (#7057)
|
2025-06-10 12:27:57 -07:00 |
|
|
fzyzcjy
|
f6ebba537a
|
Support both approximate and exact expert distribution collection (#6964)
|
2025-06-09 20:56:17 -07:00 |
|
|
fzyzcjy
|
e58423b2b9
|
Fix cutlass MLA gets almost zero accuracy (#6998)
|
2025-06-09 10:16:29 -07:00 |
|
|
fzyzcjy
|
de1350ea20
|
Minor remove one kernel for DeepSeek (#6977)
|
2025-06-08 17:41:35 -07:00 |
|
|
fzyzcjy
|
86fe943bc3
|
Fix expert distribution dumping causes OOM (#6967)
|
2025-06-08 17:41:14 -07:00 |
|
|
fzyzcjy
|
2fc1299562
|
Remove unnecessary kernels of num_token_non_padded (#6965)
|
2025-06-08 05:09:17 -07:00 |
|
|
fzyzcjy
|
f5599ef124
|
Refactor global_server_args_dict (#6866)
|
2025-06-07 03:10:35 -07:00 |
|
|
fzyzcjy
|
c499591ac8
|
Add canary for EPLB rebalancing (#6895)
|
2025-06-07 03:09:33 -07:00 |
|
|
fzyzcjy
|
d5c097a2f9
|
Tiny re-introduce profile id logging (#6912)
|
2025-06-07 02:32:50 -07:00 |
|
|
fzyzcjy
|
2f715f51cc
|
Minor compile fused topk (#6944)
|
2025-06-07 01:40:38 -07:00 |
|
|
fzyzcjy
|
61ce91ed28
|
Tiny support customize DeepEP max dispatch tokens per rank (#6934)
|
2025-06-06 17:18:35 -07:00 |
|
|
fzyzcjy
|
35b65cf0ca
|
Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa (#6890)
|
2025-06-05 11:37:05 -07:00 |
|
|
fzyzcjy
|
0166403c20
|
Support Blackwell DeepEP docker images (#6868)
|
2025-06-05 00:07:53 -07:00 |
|
|
fzyzcjy
|
bcf66ef3e1
|
Tiny allow profiler API to auto create directory (#6865)
|
2025-06-05 00:07:03 -07:00 |
|
|
fzyzcjy
|
0de5e7d40f
|
Support layerwise rebalancing experts (#6851)
|
2025-06-05 00:05:52 -07:00 |
|
|
fzyzcjy
|
72a110f664
|
Tiny update error hints (#6846)
|
2025-06-05 00:05:28 -07:00 |
|
|
fzyzcjy
|
5aff1e9392
|
Fix Qwen3MoE missing token padding optimization (#6820)
|
2025-06-05 00:04:59 -07:00 |
|
|
fzyzcjy
|
b6d0ce9f78
|
Minor add metrics to expert location updater (#6816)
|
2025-06-02 23:59:11 -07:00 |
|
|
fzyzcjy
|
0ea330ca34
|
Fix wrong weight reference in dynamic EPLB (#6818)
|
2025-06-02 23:26:04 -07:00 |
|
|
fzyzcjy
|
df7f61ee7d
|
Speed up rebalancing when using non-static dispatch algorithms (#6812)
|
2025-06-02 11:18:17 -07:00 |
|
|
fzyzcjy
|
ef21729c1d
|
Fix profiles do not have consistent names (#6811)
|
2025-06-02 11:17:22 -07:00 |
|
|
fzyzcjy
|
f5159315b2
|
Add simple utility to dump tensors for debugging (#6815)
|
2025-06-02 11:15:31 -07:00 |
|
|
fzyzcjy
|
6d7b6696d4
|
Tiny fix EPLB assertion about rebalancing period and recorder window size (#6813)
|
2025-06-02 11:13:33 -07:00 |
|
|
fzyzcjy
|
6376b632eb
|
Tiny log prefill time (#6780)
|
2025-06-02 10:28:27 -07:00 |
|
|
fzyzcjy
|
e05e29d178
|
Refactor CustomOp to avoid confusing bugs (#5382)
|
2025-06-02 10:27:36 -07:00 |
|
|
fzyzcjy
|
2c3b71d678
|
Improve EPLB logical to physical dispatch map (#6727)
|
2025-05-29 19:23:54 -07:00 |
|
|
fzyzcjy
|
6df81e8a39
|
Support tuning DeepEP configs (#6742)
|
2025-05-29 08:12:22 -07:00 |
|
|
fzyzcjy
|
3ab7d9b55e
|
Support picking variants of EPLB algorithms (#6728)
|
2025-05-29 08:12:01 -07:00 |
|
|
fzyzcjy
|
7e5071c92a
|
Super tiny enable sole usage of expert distribution metrics and update doc (#6680)
|
2025-05-29 08:11:38 -07:00 |
|
|
fzyzcjyandCheng Wan
|
31589e177e
|
Speed up when having padding tokens two-batch overlap (#6668)
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
|
2025-05-28 16:00:58 -07:00 |
|
|
fzyzcjy
|
ae6a5b2950
|
Minor refactor two-batch overlap (#6682)
|
2025-05-28 15:54:17 -07:00 |
|
|
fzyzcjy
|
4839999b76
|
Overlap two kernels in DeepSeek with communication (#6711)
|
2025-05-28 15:53:51 -07:00 |
|
|
fzyzcjy
|
541a985f85
|
Fuse routed_scaling_factor in DeepSeek (#6710)
|
2025-05-28 15:53:37 -07:00 |
|
|
fzyzcjy
|
87068b5cc7
|
Support gathering expert distribution details (#6665)
|
2025-05-27 15:32:59 -07:00 |
|
|
fzyzcjy
|
a564e001b5
|
Fix DeepEP error in Qwen 3 MoE models (#6673)
|
2025-05-27 15:12:54 -07:00 |
|
|
fzyzcjy
|
673ff668f7
|
Speed up expert location update (#6661)
|
2025-05-27 10:00:09 -07:00 |
|
|
fzyzcjy
|
447be24228
|
Fix OOM when updating expert locations (#6660)
|
2025-05-27 09:59:53 -07:00 |
|
|
fzyzcjy
|
1a8f5f6836
|
Super tiny rename environment variable (#6648)
|
2025-05-26 21:01:16 -07:00 |
|
|
fzyzcjy
|
32cd707002
|
Support TP in attention for two batch overlap (#6634)
|
2025-05-26 20:28:12 -07:00 |
|
|
fzyzcjy
|
ebd1ed49d4
|
Tiny refactor communicator (#6646)
|
2025-05-26 20:24:17 -07:00 |
|
|
fzyzcjy
|
ca95556c76
|
Tiny fix sampler error when prob is not contiguous (#6639)
|
2025-05-26 19:19:08 -07:00 |
|
|
fzyzcjy
|
0ca3e56802
|
Tiny fix missing expert location dispatch info (#6620)
|
2025-05-26 08:58:31 -07:00 |
|
|
fzyzcjy
|
5c7aa00976
|
Fix EPLB algorithm fail to run when using 3 nodes for prefill (#6629)
|
2025-05-26 08:43:24 -07:00 |
|
|
fzyzcjy
|
fe386acae6
|
Automatically configure for EPLB-related args (#6628)
|
2025-05-26 08:42:49 -07:00 |
|
|
fzyzcjy
|
501efc3d36
|
Tiny fix CI (#6611)
|
2025-05-25 23:36:34 -07:00 |
|
|
fzyzcjy
|
84147254c9
|
Tiny fix lint CI does not trigger on master (#6609)
|
2025-05-25 22:47:03 -07:00 |
|
|
fzyzcjy
|
6bebef60a7
|
Support accurate length control for bench serving (#6594)
|
2025-05-25 22:46:23 -07:00 |
|
|
fzyzcjyandyizhang2077
|
25be63d0b2
|
Auto handle PD disaggregation in bench_serving (#6587)
Co-authored-by: yizhang2077 <1109276519@qq.com>
|
2025-05-25 22:41:27 -07:00 |
|
|
fzyzcjy
|
d502dae0f0
|
Tiny change killall_sglang.sh (#6596)
|
2025-05-25 22:36:51 -07:00 |
|
|
fzyzcjy
|
93e53f6e0b
|
Logging and minor fixes to two batch overlap and EPLB (#6595)
|
2025-05-25 22:36:40 -07:00 |
|
|
fzyzcjy
|
a191a0e47c
|
Improve performance of two batch overlap in some imbalanced cases (#6593)
|
2025-05-25 22:36:18 -07:00 |
|
|
fzyzcjy
|
8c7279c24e
|
Fix profiling will crash the server when using num_steps (#6586)
|
2025-05-25 22:36:02 -07:00 |
|
|
fzyzcjy
|
0ca1811715
|
Support fake perfectly balanced EP dispatch algorithm (#6571)
|
2025-05-25 22:35:51 -07:00 |
|
|
fzyzcjy
|
2c3a6fe1de
|
Fix bench_serving does not support changing warmup requests (#6439)
|
2025-05-25 22:35:36 -07:00 |
|
|
fzyzcjy
|
5ccf8fe1a0
|
Hint users when weight update timeouts (#6570)
|
2025-05-25 09:13:17 -07:00 |
|
|
fzyzcjy
|
0d47788025
|
Support overlapping two batches (#4068)
|
2025-05-24 17:39:07 -07:00 |
|
|
fzyzcjy
|
f456037396
|
Utilize static dispatching for communicator (#6577)
|
2025-05-24 17:34:35 -07:00 |
|
|
fzyzcjy
|
b2388433be
|
Add back DeepSeek non-TBO branches (#6578)
|
2025-05-24 17:34:00 -07:00 |
|
|
fzyzcjy
|
a38376fa99
|
Refactor attention into multiple stages (#6477)
|
2025-05-24 17:33:25 -07:00 |
|
|
fzyzcjy
|
c4831e2fcf
|
Fix accuracy is zero when enabling moe-dense-tp-size as in large scale EP (#6567)
|
2025-05-24 00:27:10 -07:00 |
|
|
fzyzcjy
|
7a80f56513
|
Support dynamically rebalancing experts using EPLB (#6469)
|
2025-05-21 23:13:21 -07:00 |
|
|
fzyzcjy
|
9484eba4ad
|
Support logging expert balancedness metrics (#6482)
|
2025-05-21 23:05:33 -07:00 |
|
|
fzyzcjy
|
fc992a09f9
|
Support updating expert locations dynamically (#6388)
|
2025-05-21 21:59:33 -07:00 |
|
|
fzyzcjy
|
969660c762
|
Recover from corrupted cache file in bench serving (#6510)
|
2025-05-21 17:13:54 -07:00 |
|
|
fzyzcjy
|
55f6005f53
|
Fix bench_one_batch_server (#6503)
|
2025-05-21 11:08:17 -07:00 |
|
|
fzyzcjy
|
7222e1dacc
|
Let bench_one_batch_server use sharegpt data to make expert distribution more natural (#5573)
|
2025-05-21 02:08:43 -07:00 |
|