minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity
18,691 Commits 2 Branches 0 Tags
dsv41-pd
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Byron Hsu 50a7de47d5 [Benchmark] Add agentic rollout simulator and offline explorer (#40034) 2026-09-18 10:06:13 -07:00
Byron HsuandByron Hsu a98d921658 [DP Attn] Fix crash for no token all-gather case (#39899) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
 2026-09-17 09:21:09 -07:00
Byron HsuandByron Hsu c9fbe5f655 [MM] Add flag to force Kimi image preprocessing onto CPU (#39148) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
 2026-09-14 21:19:39 -07:00
Byron HsuandByron Hsu 5c2de3f355 [PD] Preserve the prefill rank during rebootstrap (#39357) 
Co-authored-by: Byron Hsu <byronhsu@users.noreply.github.com>
 2026-09-14 09:20:37 -07:00
Byron HsuByron HsuManik Singhal
 		39e147443b [Session] Fix image append positions and parent metadata (#39145) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: Manik Singhal <manikvsinghal.pub@gmail.com>
 2026-09-13 21:06:47 -07:00
Byron HsuByron HsuManik Singhal
 		34b2904741 [Session] Work with PD and Fix empty continuations (#39038) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: Manik Singhal <3400497+Manikvsin@users.noreply.github.com>
 2026-09-12 22:24:32 -07:00
Byron HsuByron HsuManik Singhal
 		25a5641cf2 [Session + MM] Fix text positions in session continuations (#39144) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: Manik Singhal <3400497+Manikvsin@users.noreply.github.com>
 2026-09-12 10:31:55 -07:00
Byron HsuByron HsuManik Singhal
 		7d9c57da6e [Session] Fix session idle timeout after rejected requests (#39035) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: Manik Singhal <3400497+Manikvsin@users.noreply.github.com>
 2026-09-11 16:59:43 -07:00
Byron HsuByronHsurootByron Hsu
 		fd7743e0e1 [Sampling] Support sampling masks with overlap scheduling (#36631) 
Co-authored-by: ByronHsu <ByronHsu@users.noreply.github.com>
Co-authored-by: root <root@slurm-h200-208-179.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
 2026-09-10 14:52:44 -07:00
Byron HsuandByron Hsu a88e852fab [Sampling] Allow sampling-mask replay with DisallowedTokensLogitsProcessor (#38279) 
Co-authored-by: Byron Hsu <24364830+ByronHsu@users.noreply.github.com>
 2026-09-06 23:56:16 -07:00
Byron HsuandByronHsu 046cdaabaa [Sampling] Capture masks from sampler support (#36630) 
Co-authored-by: ByronHsu <ByronHsu@users.noreply.github.com>
 2026-09-02 17:13:28 -07:00
Byron HsuByron HsurootCursorMick
 		f6a5a1b59c [RL+VLM] Avoid retokenization drift for pre-tokenized (token-id) VLM requests (#26555) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
Co-authored-by: root <root@slurm-h200-209-231.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-06-01 09:58:14 -07:00
Byron HsurootCursorCheng Wan
 		6f1c9fc77b [RL] Fix crash when the reqs in a batch have a mix of return_routed_experts = True and False. (#26423) 
Co-authored-by: root <root@slurm-h200-209-231.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2026-05-29 20:46:10 -07:00
Byron HsuandByron Hsu 6ea69efb7f [RL] Forward Kimi K2.5 weight hooks to language model (#26744) 
Co-authored-by: Byron Hsu <24364830+ByronHsu@users.noreply.github.com>
 2026-05-29 15:08:58 -07:00
Byron HsuByron HsuCursorroot
 		cf66693b35 [Model] Add Qwen3-MoE MTP (#26468) 
Co-authored-by: Byron Hsu <byronhsu@Byrons-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: root <root@slurm-h200-209-231.slurm-compute.tenant-slurm.svc.cluster.local>
 2026-05-29 14:31:05 -07:00
Byron HsuByron HsuCursor
 		d96e593fd0 [Benchmark] Add SGLANG_SIMULATE_UNIFORM_EXPERTS for balanced expert routing with dummy weights (#25571) 
Co-authored-by: Byron Hsu <byronhsu@Byrons-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-18 09:16:12 -07:00
Byron HsuByron HsuCursorzyzshishuiYuzhen Zhou
 		d82e339ce2 [Session R3] Add routed_experts_start_len for absolute routing slice control (#24851) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: zyzshishui <zyzshishui@gmail.com>
Co-authored-by: Yuzhen Zhou <82826991+zyzshishui@users.noreply.github.com>
 2026-05-10 10:04:43 -07:00
Byron HsuByron HsuCursor
 		cfd3fd00d0 [RL] Call torch.cuda.empty_cache() for in-place pause mode to avoid OOM (#24854) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-09 23:36:52 -07:00
Byron HsuByron HsuCursor
 		1e6c6d1f07 [Utils] Make request dump robust to unpicklable server_args and large meta_info (#24767) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-09 21:41:41 -07:00
Byron HsuByron HsuCursor
 		47483001b6 [PrefillDelayer] support NCCL all-gather for cross-DP info sync (#24768) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-09 21:20:03 -07:00
Byron HsuByron HsuCursor
 		7edb4c3cea [NUMA+Ray] Fix NUMA NVML handle resolution under shuffled CUDA_VISIBLE_DEVICES (#24766) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-09 21:18:39 -07:00
Byron HsuByron HsuCursor
 		f9c315e85d docs: clarify how /tag-and-rerun-ci kicks off CI on the current commit (#24774) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-10 11:28:41 +08:00
Byron Hsuandroot 27659ea8c8 [PD+Pause] Remove redundant post processing (#23886) 
Co-authored-by: root <root@slurm-h200-206-011.slurm-compute.tenant-slurm.svc.cluster.local>
 2026-04-27 17:01:26 -07:00
Byron HsuByron HsuClaude Opus 4.7
 		cb0429f253 [Disagg] Finalize routed_experts_output in process_batch_result_disagg_prefill (#23885) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 16:40:05 -07:00
Byron HsuByron HsuClaude Opus 4.7
 		85376a6119 refactor(moe): centralize post-experts all-reduce skip predicate (#23748) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-26 20:29:59 -07:00
Byron HsuByron HsuClaude Opus 4.7Kangyan-Zhou
 		ba4e9d2ac2 Apply should_use_dp_reduce_scatterv guard to remaining MoE models (follow-up to #23731) (#23732) 
Co-authored-by: Byron Hsu <byronhsu@noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Kangyan-Zhou <zky314343421@gmail.com>
 2026-04-25 20:36:16 -07:00
Byron HsuByron HsuClaude Opus 4.7
 		71029abd64 Fix Qwen3 MoE: also guard EP all-reduce with not use_reduce_scatter (follow-up to #23731) (#23734) 
Co-authored-by: Byron Hsu <byron@periodiclabs.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-25 20:35:52 -07:00
Byron HsuandByron Hsu 99b59b279c Fix Qwen3 MoE double-reduce when DP attention + EP + reduce_scatterv (#23729) (#23731) 
Co-authored-by: Byron Hsu <byronhsu@noreply.github.com>
 2026-04-25 15:28:28 -07:00
Byron Hsu 17210350fd [PD+DP] Allow PrefillDelayer in disaggregated-prefill mode (#23588) 2026-04-23 14:51:16 -07:00
Byron Hsu 0b77284587 [minor] Make DEFAULT_FORCE_STREAM_INTERVAL configurable via SGLANG_FORCE_STREAM_INTERVAL (#23215) 2026-04-22 13:05:40 -07:00
Byron Hsu c090f71bf2 feat: enable SGLANG_PATCH_TOKENIZER by default (#23409) 2026-04-21 17:53:43 -07:00
Byron HsuByron HsuCheng Wan
 		1cff871c67 [Bugfix] Fix DeepEP timeout when compiling DeepGeMM in EP+DP+TP (#23185) 
Co-authored-by: Byron Hsu <byronhsu@Byrons-MacBook-Pro.local>
Co-authored-by: Cheng Wan <ch-wan@users.noreply.github.com>
 2026-04-19 17:36:11 -07:00
Byron HsuandByron Hsu cf9845f8e3 [Bug Fix] Ensure prefill_info_table is populated before honoring disagg_prefill_dp_rank (#22990) 
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
 2026-04-17 11:10:31 +08:00
Byron Hsu 3600465e81 [Bug Fix] Remove follow_bootstrap_room fast path in PD disaggregation DP rank resolution (#22901) 2026-04-15 22:53:29 -07:00
Byron Hsu 5636d16dec [CI] Add logs to debug TestOpenAIServer.test_completion_stream (#17471) 2026-02-02 02:38:08 -08:00
Byron Hsuandroot 339915ce2b [logprob] Fix logprob + streaming for long concurrent decode by caching already processed logprob (#17005) 
Co-authored-by: root <root@memx-cge-29-sr1.xpop.twttr.net>
 2026-01-13 12:39:14 -08:00
Byron Hsu edbeaf3b88 [MM][style] rename inputs_embeds to input_embeds for consistency (#14240) 2025-12-01 11:36:51 -08:00
Byron Hsu 0825d7f4c6 [piecewise] Refactor VLM to support input embed buffer and remove external embedder hack (#14155) 2025-11-30 21:43:09 -08:00
Byron Hsu 96be97bfff Minor PD style fix (#7215) 2025-06-15 16:12:12 -07:00
Byron Hsu 88f9c347b2 [PD] use int32 for kv indices & get num_reserved_decode_tokens from server_args (#7214) 2025-06-15 11:51:03 -07:00
Byron Hsu db0cc57e75 [PD] Support decode retract and update decode.py (#7196) 2025-06-14 19:48:05 -07:00
Byron Hsu 7d316991b2 [PD] Update prefill.py (#7190) 2025-06-14 15:59:54 -07:00
Byron Hsu c2b16795b5 Add decode req pool (#6980) 2025-06-09 21:23:36 -07:00
Byron Hsu 2d831c6ef9 [PD] Support structured output (#6560) 2025-05-23 21:49:00 -07:00
Byron Hsu 8233cc10fd [PD] Support logprob & Add failure test (#6558) 2025-05-23 14:29:20 -07:00
Byron HsuandSangBin Cho d2e0881a34 [PD] support spec decode (#6507) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-05-23 12:03:05 -07:00
Byron Hsu 0a4fc73b48 [PD] Fix failure abort (#6535) 2025-05-22 20:32:03 -07:00
Byron Hsu 3bde101099 [PD] Abort request if transfer fails (#6504) 2025-05-21 21:44:25 -07:00
Byron Hsu 7513558074 [PD] Add doc and simplify sender.send (#6019) 2025-05-21 21:22:21 -07:00
Byron Hsu bf98d2e377 [PD] Support prefill overlap + Ensure no race condition (#5609) 2025-04-21 12:12:56 -07:00
Byron Hsu e65b9f21e3 [PD] Support decode overlap schedule (#5608) 2025-04-21 12:06:16 -07:00
Byron Hsu deded17f38 [PD] Fix edge case and simplify large page size + chunked prefill (#5589) 2025-04-21 10:27:02 -07:00
Byron Hsu c951d312ed [PD] Fix large page size + chunk prefill (#5588) 2025-04-20 17:21:54 -07:00
Byron Hsu ab4b5606e4 [PD] Support page size > 1 (#5561) 2025-04-19 21:54:27 -07:00
Byron Hsu a9499885e9 [PD] Add transfer backend abstraction (#5328) 2025-04-14 01:39:39 +08:00
Byron HsuandLiangsheng Yin 6d3b35fae9 [PD] Simplify mini LB (#4911) 
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
 2025-04-08 09:42:34 -07:00
Byron HsuSangBin ChoYing1123merrymercymakrodhou-xai
 		c7c7dbebbe [PD] Release initial code (#4654) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Ying1123 <sqy1415@gmail.com>
Co-authored-by: merrymercy <lianminzheng@gmail.com>
Co-authored-by: makro
Co-authored-by: dhou-xai
 2025-03-21 14:47:47 -07:00
Byron Hsu 8cc300f536 Fix router test (#4483) 2025-03-16 22:49:47 -07:00
Byron Hsu 734daedd8f [fix] Clamp logprob with dtype min to prevent -inf (#3224) 2025-01-31 17:04:04 +08:00
Byron Hsu 20453cef62 [test] Lower number of top logprobs to get rid of -inf (#3212) 2025-01-30 18:01:23 +08:00
Byron Hsuandzhyncs 988d0a4bfc [kernel] Use sgl_kernel rope (#3169) 
Co-authored-by: zhyncs <me@zhyncs.com>
 2025-01-28 14:33:11 +08:00
Byron Hsu 27aeb4b7d8 [test] deduplicate test_session_control (#3183) 2025-01-28 13:17:06 +08:00
Byron Hsu 514f37c32b [kernel] Fix position ids in rope (#3173) 2025-01-27 17:09:51 +08:00
Byron Hsu 741fccd7bf Bump sgl kernel to 0.0.2.post19 (#3167) 2025-01-27 15:36:07 +08:00
Byron Hsu fb11a43981 [kernel] Integrate flashinfer's rope with higher precision and better perf (#3134) 2025-01-27 15:28:00 +08:00
Byron Hsu 3ed0a547b2 [router] Fix twine uploading (#3095) 2025-01-23 21:01:01 -08:00
Byron Hsu 8d8ef8497e bump router to 0.1.4 (#3094) 2025-01-23 20:32:43 -08:00
Byron Hsu 9a0cc2e90e [router] Forward all request headers from router to workers (#3070) 2025-01-23 20:30:31 -08:00
Byron Hsu 5de50653cd [router] make error actionable (#3063) 2025-01-22 17:56:21 -08:00
Byron Hsu c0bf9bf15c [devcontainer] add non-root user (#2989) 2025-01-22 17:47:54 -08:00
Byron Hsu 3ad4cd4915 bump router to 0.1.3 (#3020) 2025-01-20 14:38:06 -08:00
Byron Hsu 3a8428ecaa [router] Expose worker startup interval (#3019) 2025-01-20 14:36:54 -08:00
Byron Hsu 0311ce8e1c [router] Expose worker startup secs & Return error instead of panic for router init (#3016) 2025-01-20 12:45:13 -08:00
Byron HsuandYineng Zhang b5caa22dfb [kernel] port rope cuda kernel to sgl-kernel (#2993) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2025-01-20 20:58:51 +08:00
Byron Hsu 4719c1d04a [router] Fix sgl router path for release (#2980) 2025-01-19 01:11:06 -08:00
Byron Hsu ef18b0eda2 [router] Allow empty worker list for sglang.launch_router (#2979) 2025-01-19 01:05:23 -08:00
Byron Hsu 53cc91e504 [devcontainer] Fix mount and GPU & Support rust dev (#2978) 2025-01-19 16:34:01 +08:00
Byron Hsu 2d60a5ee75 Update v0.1.0.md 2024-12-11 13:48:18 -08:00
Byron Hsu 2e4a5907c9 [router] Release router 0.1.0 with dynamic scaling and fault tolerance (#2455) 2024-12-11 13:42:35 -08:00
Byron Hsu c0ee46fe10 [router] Update doc for dynamic scaling and fault tolerance (#2454) 2024-12-11 13:11:42 -08:00
Byron Hsu 864bf2ba00 [router] remove main.rs because only lib.rs is used for py binding (#2453) 2024-12-11 12:13:19 -08:00
Byron Hsu a4cca7fc53 [router] Add retries based fault tolerance (#2452) 2024-12-11 12:13:08 -08:00
Byron Hsu 0fb88aaa77 [router] Use borrow if possible to save cost (#2441) 2024-12-11 01:38:50 -08:00
Byron Hsu d4de9a6235 [router] Refactor: decouple select and send stage (#2440) 2024-12-11 00:51:21 -08:00
Byron Hsu 27f7bed7a7 reduce watchdog interval to 5s (#2410) 2024-12-08 21:17:31 -08:00
Byron Hsu 6387098f5f [router] add health checking in router init (#2393) 2024-12-08 17:17:37 -08:00
Byron Hsu 2a717c5078 [Router] fix interrupt from terminal (#2413) 2024-12-08 16:58:41 -08:00
Byron Hsu a1e697b25b [router] Improve cleanup logic (#2411) 2024-12-08 15:24:02 -08:00
Byron Hsu ef995dae1e [router] Health check on worker before adding to the router (#2392) 2024-12-07 15:39:54 -08:00
Byron Hsu c36736c841 [router] Add remove worker api (#2380) 2024-12-06 17:16:03 -08:00
Byron Hsu 1bf9e34745 [router] add remove tenant method in the radix tree (#2379) 2024-12-06 11:53:15 -08:00
Byron Hsu 499c85f131 [Router] remove duplicate char count (#2378) 2024-12-06 11:26:07 -08:00
Byron Hsu 67b657945a [router] support /add_worker api (#2369) 2024-12-06 01:17:04 -08:00
Byron Hsu 64fceab8af [router] use 2-gpu-runner (#2368) 2024-12-06 14:13:57 +08:00
Byron Hsu 0495796517 [router] Copy license when publishing & bump version (#2339) 2024-12-03 10:27:43 -08:00
Byron Hsu 92d1253e58 Bump sglang-router to 0.0.10 for env name change (#2226) 2024-11-27 11:23:32 -08:00
Byron Hsu 0b46b951ae Fix rust warning (#2208) 2024-11-26 15:00:41 -08:00
Byron Hsu 2763c0a73a Bump router to 0.0.9 with better logging (#2207) 2024-11-26 13:30:28 -08:00
Byron Hsu 1f76fc6e3f [router] Rust e2e test (#2184) 2024-11-25 16:02:03 -08:00
Byron Hsu 4d62bca542 [router] Replace print with logger (#2183) 2024-11-25 13:36:02 -08:00