Compare commits
529
Commits
g9i-est-td
...
B300261102
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ee9170e7bb | ||
|
|
17505e6cf6 | ||
|
|
81f7398855 | ||
|
|
0a56a5f6e7 | ||
|
|
7f6fe22445 | ||
|
|
f6fc6ac3e7 | ||
|
|
0a3c370451 | ||
|
|
b1b303c9f2 | ||
|
|
7d996300b9 | ||
|
|
54df642fe6 | ||
|
|
564aec61c8 | ||
|
|
a7183a8246 | ||
|
|
af3f9e4c8f | ||
|
|
f09cec65e9 | ||
|
|
f4ecaae888 | ||
|
|
cdab6c1bcc | ||
|
|
f2c634961b | ||
|
|
3b95120f00 | ||
|
|
55f23a8b59 | ||
|
|
3c6a6c8068 | ||
|
|
17d92ea671 | ||
|
|
7cefa60765 | ||
|
|
25ba24e9a7 | ||
|
|
42ac81c655 | ||
|
|
de7055f596 | ||
|
|
88ac7c02a6 | ||
|
|
a77998e157 | ||
|
|
79701a2549 | ||
|
|
327c906764 | ||
|
|
5bef70c0a2 | ||
|
|
1d4dcc81cc | ||
|
|
47c0791ad1 | ||
|
|
bbc77e7db4 | ||
|
|
4e2d5b17ad | ||
|
|
be8600d02d | ||
|
|
a08bb0ec9e | ||
|
|
e66c9e981d | ||
|
|
2e6d87e2ca | ||
|
|
386fa9ecfd | ||
|
|
5b8955e70f | ||
|
|
738e4535ea | ||
|
|
3712b5944d | ||
|
|
2d5ad3243f | ||
|
|
e22a530859 | ||
|
|
49a2fbf817 | ||
|
|
ee2dd727a3 | ||
|
|
79e5c4bb9b | ||
|
|
5994c5597d | ||
|
|
6a319e7b34 | ||
|
|
0d592c475b | ||
|
|
855c092deb | ||
|
|
197be2203a | ||
|
|
f05ad5ec01 | ||
|
|
15dc838f9b | ||
|
|
ea226cf782 | ||
|
|
5f7542d762 | ||
|
|
27ac614f05 | ||
|
|
4542fc8f59 | ||
|
|
194abb8e54 | ||
|
|
1d605b7ece | ||
|
|
005333d029 | ||
|
|
d0022e1df3 | ||
|
|
ac3db731b3 | ||
|
|
8b9b992e41 | ||
|
|
0931dfedc9 | ||
|
|
e6451c1a95 | ||
|
|
1340044e97 | ||
|
|
9d7ad308bd | ||
|
|
baf80c61b7 | ||
|
|
1e894eb92d | ||
|
|
16aa816060 | ||
|
|
74c08f3dc5 | ||
|
|
30dc2f9430 | ||
|
|
5a86d6ee17 | ||
|
|
eb8ef80842 | ||
|
|
efd2ce8a9d | ||
|
|
c677171c1d | ||
|
|
02b983603d | ||
|
|
0e34962884 | ||
|
|
82ff16dad0 | ||
|
|
3e5f5505ed | ||
|
|
2fbd094a8c | ||
|
|
9b46f27090 | ||
|
|
2a46e32fbc | ||
|
|
86933b1f33 | ||
|
|
0347b2bb4e | ||
|
|
4b1941e889 | ||
|
|
f967418a35 | ||
|
|
535aeae2a7 | ||
|
|
69f830756e | ||
|
|
611a703061 | ||
|
|
494eeecb6c | ||
|
|
f9129915a3 | ||
|
|
8b9f72417f | ||
|
|
bb72b47daf | ||
|
|
77bc8cab18 | ||
|
|
7d1a16d3eb | ||
|
|
2b8e4cde95 | ||
|
|
4b273537de | ||
|
|
02d2445138 | ||
|
|
472208fec0 | ||
|
|
6d6c9cd4ae | ||
|
|
6f466f655a | ||
|
|
8708d2ee99 | ||
|
|
4f47c48b33 | ||
|
|
1cd0ebb3d3 | ||
|
|
4da6f585f3 | ||
|
|
79ab8f6d7c | ||
|
|
6e667c4310 | ||
|
|
6c844eb12e | ||
|
|
6dd0dfabf0 | ||
|
|
2be6577ba9 | ||
|
|
76c382e80f | ||
|
|
904691e0b5 | ||
|
|
3f36fc0833 | ||
|
|
0e338777b0 | ||
|
|
7c680a0d45 | ||
|
|
7baacd1d1b | ||
|
|
c7994e7076 | ||
|
|
5bdbec0d48 | ||
|
|
e83f585ae6 | ||
|
|
49933325cc | ||
|
|
f4a2df3dac | ||
|
|
fa628c5359 | ||
|
|
79eb10e182 | ||
|
|
f2873f1613 | ||
|
|
698651acd0 | ||
|
|
ce2261863f | ||
|
|
a4838160b3 | ||
|
|
c4fa35b829 | ||
|
|
2da6c0c237 | ||
|
|
363e6c6b4d | ||
|
|
5daa1352ba | ||
|
|
e5dc0c363a | ||
|
|
c82f7141ad | ||
|
|
94e4611eac | ||
|
|
92e57d5de7 | ||
|
|
b588aa39e9 | ||
|
|
f36ee5de93 | ||
|
|
26396bf566 | ||
|
|
a95ae5f1c5 | ||
|
|
97b79f0796 | ||
|
|
9c7dd3c45f | ||
|
|
ae4258715c | ||
|
|
77f90ade97 | ||
|
|
bb9e9d5632 | ||
|
|
5849ff5337 | ||
|
|
8d47d515e4 | ||
|
|
035ed73b9d | ||
|
|
c394a90c2b | ||
|
|
f9b7225f2e | ||
|
|
9c0f6d6a61 | ||
|
|
818d7932d7 | ||
|
|
3b0003dac1 | ||
|
|
a904e4d31a | ||
|
|
840a6313c9 | ||
|
|
25ad569ff9 | ||
|
|
e5db41b329 | ||
|
|
786cdfc040 | ||
|
|
13999673ef | ||
|
|
154e29c1ec | ||
|
|
ba83415eea | ||
|
|
bf0e193bfd | ||
|
|
aae2145eb2 | ||
|
|
357323e912 | ||
|
|
4f96807ea8 | ||
|
|
d67d1ae02c | ||
|
|
0d6b283d7e | ||
|
|
8d8916815b | ||
|
|
913e91ed51 | ||
|
|
c80285ca9a | ||
|
|
cc811ff0c3 | ||
|
|
8cd3c494a3 | ||
|
|
afd0d7b47b | ||
|
|
0f7d3a0451 | ||
|
|
dcf05724f7 | ||
|
|
698fa949ec | ||
|
|
635836129b | ||
|
|
a9902f47e3 | ||
|
|
851419717a | ||
|
|
91fc46a017 | ||
|
|
0ec3fee429 | ||
|
|
e43f6128a0 | ||
|
|
f4ccfd4183 | ||
|
|
178febef5d | ||
|
|
81841d6bdc | ||
|
|
8a6770d566 | ||
|
|
96f013b452 | ||
|
|
d0a7565436 | ||
|
|
3f6c313389 | ||
|
|
5c2cb60790 | ||
|
|
0a87f163dc | ||
|
|
1c967f5043 | ||
|
|
b5137693a5 | ||
|
|
24e45961c6 | ||
|
|
c64886f9e3 | ||
|
|
f727076ed0 | ||
|
|
9d5386fd13 | ||
|
|
33a0734ca1 | ||
|
|
569dda1e61 | ||
|
|
779adb4b82 | ||
|
|
f68e82e7bc | ||
|
|
82f7249ce6 | ||
|
|
a56024c267 | ||
|
|
9569a36561 | ||
|
|
ce66b3ab59 | ||
|
|
12370da115 | ||
|
|
a6a6beffac | ||
|
|
d9ce511646 | ||
|
|
3472e16f96 | ||
|
|
80263b5580 | ||
|
|
f91b81b82a | ||
|
|
8d7df07a56 | ||
|
|
17babd67b4 | ||
|
|
45de4adde9 | ||
|
|
9ee9858805 | ||
|
|
f1e26dc658 | ||
|
|
1962cbeebe | ||
|
|
5f62e23d30 | ||
|
|
6929afb528 | ||
|
|
8dc7f98005 | ||
|
|
3468a65b51 | ||
|
|
780a5b1acc | ||
|
|
b057d5b3f1 | ||
|
|
b453996b7d | ||
|
|
3d508d614f | ||
|
|
67bbfcee3c | ||
|
|
d61f618276 | ||
|
|
4049941a82 | ||
|
|
aa10927271 | ||
|
|
59149553b9 | ||
|
|
9ec74a53cc | ||
|
|
9301ca37a0 | ||
|
|
5fff58a169 | ||
|
|
eacd0a07d7 | ||
|
|
ed89693c4a | ||
|
|
da7b97a737 | ||
|
|
3438ed660d | ||
|
|
ed1f56fead | ||
|
|
f4035cab0b | ||
|
|
42db2db4a8 | ||
|
|
991b846227 | ||
|
|
4b9936b5e3 | ||
|
|
99cf403c9e | ||
|
|
aa5880e2b8 | ||
|
|
51baa1f6e9 | ||
|
|
f3f0f7b68c | ||
|
|
f5eca05b05 | ||
|
|
29e78045fb | ||
|
|
68623cae92 | ||
|
|
f7d4f06242 | ||
|
|
bc412213c1 | ||
|
|
620718c8ea | ||
|
|
97945831f9 | ||
|
|
2c7a9e3b50 | ||
|
|
028de2c5d4 | ||
|
|
a2e4ffbcea | ||
|
|
7e0e259435 | ||
|
|
a3d00eff58 | ||
|
|
7e5256d1ba | ||
|
|
f06d1bb1f4 | ||
|
|
740d5f9f9e | ||
|
|
5e79333c95 | ||
|
|
b57b5e2286 | ||
|
|
ed268dd320 | ||
|
|
6e98c84c60 | ||
|
|
4b1a44bc77 | ||
|
|
75878f27c8 | ||
|
|
cdbd478422 | ||
|
|
af49ada16d | ||
|
|
7163e97462 | ||
|
|
8e3e2980db | ||
|
|
7e5d3780ef | ||
|
|
dd9ea06c00 | ||
|
|
e109bc1832 | ||
|
|
2815eed49c | ||
|
|
fda84f80b7 | ||
|
|
a175e39b84 | ||
|
|
62b4127668 | ||
|
|
43a3f956cb | ||
|
|
43728298fe | ||
|
|
ca72dad953 | ||
|
|
5bf10d4c79 | ||
|
|
2e7c9d851c | ||
|
|
9c10e0dd61 | ||
|
|
7554be5a69 | ||
|
|
131c42ba33 | ||
|
|
be65391abd | ||
|
|
1278087336 | ||
|
|
2c9d919071 | ||
|
|
21f775cd97 | ||
|
|
7a632cf998 | ||
|
|
8bd6aa9a6c | ||
|
|
dbe0a9a7c9 | ||
|
|
6efdd33d07 | ||
|
|
6cac4dc3fe | ||
|
|
fee32a5c1f | ||
|
|
a361f0c410 | ||
|
|
f7b8c6fe40 | ||
|
|
8acfabbd67 | ||
|
|
cd61db49c7 | ||
|
|
132836b369 | ||
|
|
4449d7ecb0 | ||
|
|
844d2b5dec | ||
|
|
8599b7251f | ||
|
|
fb6e6a09ce | ||
|
|
c3b2b55bbb | ||
|
|
da7e3f1c12 | ||
|
|
bd5a445bf7 | ||
|
|
574f9cdb2e | ||
|
|
956b9a24ee | ||
|
|
1689cb2d52 | ||
|
|
4b5c9c4b5e | ||
|
|
32c26788f3 | ||
|
|
573b5716f5 | ||
|
|
1c2f42470c | ||
|
|
6bd1c76735 | ||
|
|
aba892fbdf | ||
|
|
eb91a41f38 | ||
|
|
f1ba723c64 | ||
|
|
93088901fb | ||
|
|
c0d18c0254 | ||
|
|
c1c287a429 | ||
|
|
b326e8349d | ||
|
|
5b1dbc2c80 | ||
|
|
9686edcf80 | ||
|
|
027208ae02 | ||
|
|
299a75ac5d | ||
|
|
533020b500 | ||
|
|
4f8f06faf3 | ||
|
|
7b1d5cd02d | ||
|
|
bf145750a6 | ||
|
|
d5e83ea4f6 | ||
|
|
c1e43b95f4 | ||
|
|
f1eec1232d | ||
|
|
e2e1a97fc3 | ||
|
|
b008ec3de6 | ||
|
|
1dd0f361b8 | ||
|
|
583cfc4d55 | ||
|
|
f0b6590722 | ||
|
|
7d9bc10493 | ||
|
|
cc7ae372ac | ||
|
|
0b536ca112 | ||
|
|
468e26f253 | ||
|
|
bab4a808d8 | ||
|
|
0d7638b035 | ||
|
|
dc62c3effd | ||
|
|
194876bea0 | ||
|
|
afe3e1df31 | ||
|
|
d69966f9c0 | ||
|
|
cfec1b91bb | ||
|
|
29274b41c6 | ||
|
|
82ac83a8d0 | ||
|
|
f82465718b | ||
|
|
c2e5742464 | ||
|
|
89860fbd2e | ||
|
|
7591118794 | ||
|
|
eb4b167d27 | ||
|
|
7656071309 | ||
|
|
e262b44eef | ||
|
|
952c63b1bb | ||
|
|
10d5293546 | ||
|
|
3bf4cbc6cf | ||
|
|
aa91108f64 | ||
|
|
ee2f95a522 | ||
|
|
df13d5dce3 | ||
|
|
eae134d61f | ||
|
|
07eb671b6a | ||
|
|
ca473ef3e4 | ||
|
|
800e60e345 | ||
|
|
306ccf6f57 | ||
|
|
9db58ab547 | ||
|
|
a70b1b73d3 | ||
|
|
16eb87c06f | ||
|
|
2229345dff | ||
|
|
f8fd41c36e | ||
|
|
06314dc2e8 | ||
|
|
0788f1fdd0 | ||
|
|
fdd89d182a | ||
|
|
78185f4b3f | ||
|
|
1739d962a4 | ||
|
|
15e8e687db | ||
|
|
9e23107772 | ||
|
|
1ec902e00f | ||
|
|
407bfc94f9 | ||
|
|
4b83f215f4 | ||
|
|
4a651146de | ||
|
|
36978d6c4d | ||
|
|
50fa3727eb | ||
|
|
472f0d540b | ||
|
|
476fe86a12 | ||
|
|
35c873cff4 | ||
|
|
c7687a0e7c | ||
|
|
602c92ccd8 | ||
|
|
553836c164 | ||
|
|
43a896933e | ||
|
|
d1c3346f74 | ||
|
|
72d886efb8 | ||
|
|
fdd53418a0 | ||
|
|
c53b98d1fe | ||
|
|
5b683d6b72 | ||
|
|
4a0c6f2fe0 | ||
|
|
e3205080ce | ||
|
|
1140a8dfd9 | ||
|
|
3c07ff2654 | ||
|
|
3eb4bd598f | ||
|
|
ec1fdc22a9 | ||
|
|
8683f8ad4a | ||
|
|
5e630fa571 | ||
|
|
af856717c2 | ||
|
|
55b2aef7ad | ||
|
|
2c00b5d129 | ||
|
|
67d9ee156c | ||
|
|
ce329c85fa | ||
|
|
79ed13fc2c | ||
|
|
1897575ce1 | ||
|
|
ebc71e650e | ||
|
|
71ee2dc950 | ||
|
|
693d333306 | ||
|
|
ab580b2e06 | ||
|
|
6b90e5716d | ||
|
|
964ab2b280 | ||
|
|
48ea486928 | ||
|
|
3a059de94c | ||
|
|
28448a86d4 | ||
|
|
1238edd954 | ||
|
|
9a949403f1 | ||
|
|
ab258b6453 | ||
|
|
05d7f7a166 | ||
|
|
49766a74f9 | ||
|
|
80ea5ee638 | ||
|
|
fdc2dd0d05 | ||
|
|
81ac15391c | ||
|
|
7656ea0e74 | ||
|
|
b20cb3665d | ||
|
|
642a671c30 | ||
|
|
8b1d1e0d74 | ||
|
|
aaa9388c91 | ||
|
|
f1a2ee763d | ||
|
|
5385866108 | ||
|
|
9ab74b8517 | ||
|
|
0ce41c5a06 | ||
|
|
722669e606 | ||
|
|
678913a0f0 | ||
|
|
80b86dbecb | ||
|
|
5ed306bd69 | ||
|
|
4ef9765e76 | ||
|
|
716c9151f2 | ||
|
|
8e7601ae4a | ||
|
|
806e7423af | ||
|
|
c04f9a61f0 | ||
|
|
7c4cc5a51c | ||
|
|
96ba108072 | ||
|
|
417606527d | ||
|
|
8c4680cb65 | ||
|
|
62cee1b15c | ||
|
|
ed0b8f433f | ||
|
|
0025f93944 | ||
|
|
e45cbce599 | ||
|
|
a90e9c5dd8 | ||
|
|
8111841103 | ||
|
|
33dd8f6a2e | ||
|
|
5d9c956f8e | ||
|
|
af9bb2a32d | ||
|
|
3d588779e1 | ||
|
|
f86a04a331 | ||
|
|
a5117beb11 | ||
|
|
b6d3b7e163 | ||
|
|
97857f9f96 | ||
|
|
dc81f17b90 | ||
|
|
a3330237db | ||
|
|
04c8c9703f | ||
|
|
7c9a78efbe | ||
|
|
f726461287 | ||
|
|
9a93a17c08 | ||
|
|
2b75048e2a | ||
|
|
58ad0748a6 | ||
|
|
266550048a | ||
|
|
82c9a8b037 | ||
|
|
834bfe4ef0 | ||
|
|
8d3eee46a7 | ||
|
|
06342acad2 | ||
|
|
f02458fff1 | ||
|
|
617c8b67b1 | ||
|
|
7b1710324c | ||
|
|
7f96dc6874 | ||
|
|
642f43ce59 | ||
|
|
a7fd2e1158 | ||
|
|
69076b9f9e | ||
|
|
fea48754de | ||
|
|
377cd867fd | ||
|
|
df078cc729 | ||
|
|
10ec53ac2f | ||
|
|
0dc6de2e74 | ||
|
|
cff88d6eac | ||
|
|
299da60131 | ||
|
|
329072d99a | ||
|
|
59fec011dd | ||
|
|
3c560089e1 | ||
|
|
1637a4e305 | ||
|
|
6708426728 | ||
|
|
3f13f5d966 | ||
|
|
1461152cb0 | ||
|
|
fe89a7c890 | ||
|
|
da414adda7 | ||
|
|
0f4aedfecc | ||
|
|
4ec19ca27f | ||
|
|
bb1d4cddd9 | ||
|
|
d4e3c163c9 | ||
|
|
e9b22a0818 | ||
|
|
57db5089c2 | ||
|
|
2e9b2cacda | ||
|
|
2a83471676 | ||
|
|
d318a546da | ||
|
|
a5b21145e1 | ||
|
|
3edb156dec | ||
|
|
19cb24f1e1 | ||
|
|
1d17066bad | ||
|
|
ba50ea3d77 | ||
|
|
47eeddcd18 | ||
|
|
fe1851c3c2 | ||
|
|
e17fed6ea7 | ||
|
|
680a2ff27e | ||
|
|
1fd46ddd56 | ||
|
|
8dee3f6bdb | ||
|
|
f8e32706fc | ||
|
|
a3abea3560 | ||
|
|
3b932cde26 | ||
|
|
5a930cfd88 |
@@ -742,7 +742,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
|
||||
- `docs/docs/developer_guide/development_jit_kernel_guide.mdx`
|
||||
- `test/run_suite.py` — suite names, discovery of `test/registered/`, execution entrypoint for CI
|
||||
- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
|
||||
- `python/sglang/kernels/jit/utils/compile.py` — `load_jit`, `make_cpp_args`
|
||||
- `python/sglang/kernels/jit/utils/compile/` — `load_jit` (`loader.py`), `make_cpp_args` (`cpp_args.py`)
|
||||
- `python/sglang/kernels/jit/utils/common.py` — `cache_once`, `should_run_full_tests`, `get_ci_test_range`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`, `is_type`
|
||||
- `python/sglang/kernels/jit/include/sgl_kernel/ffi.h` — `ffi::empty`, `ffi::empty_like`, `ffi::from_blob`
|
||||
|
||||
@@ -398,6 +398,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
|
||||
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||
| `/run-cpu-ci` | Dispatches `run-cpu-ci.yml` on main: only the `base-a-test-cpu` suite, at the PR head SHA, no caches saved. Same gate as `/rerun-test`. Alias: `/rerun-cpu-ci` |
|
||||
|
||||
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ These are investigation hints, not a list of changes to apply.
|
||||
| `No platform detected. Using base SRTPlatform with defaults.` | Platform detection | Decide whether fallback is expected on this machine or indicates a missing plugin. |
|
||||
| `Unexpected error during package walk` in CUTE_DSL | Installed cutlass package, its handler, root propagation, and `CUTE_DSL_LOG_LEVEL` | Inspect the actual exception and duplicate-handler path; do not assume the current log level or hide all warnings. |
|
||||
| Repeated dtype, hybrid-model, or tokenizer fallback messages | Model-config and tokenizer construction in launcher, scheduler, and detokenizer | Multiple constructors may explain repetition. Retain meaningful per-process information and distinguish these from handler duplication. |
|
||||
| Repeated template detection messages | `managers/template_detection.py` and `managers/template_manager.py` | Check whether a summary already contains the same information. |
|
||||
| Repeated template detection messages | `parser/template_detection.py` and `parser/template_manager.py` | Check whether a summary already contains the same information. |
|
||||
| KV cache dtype logged separately from allocation | Model runner and memory pool | Check whether the allocation summary already carries dtype; retain useful allocation and SWA details. |
|
||||
| CUTLASS backend disabled during graph capture | Attention backend selection | Expected fallback may be useful information; retain the reason and selected behavior. |
|
||||
| `Ignore import error when loading ...` | Model, multimodal, or algorithm registries | Distinguish an unused optional model dependency from failure to load the requested model. |
|
||||
|
||||
@@ -205,7 +205,7 @@ than restating.
|
||||
`python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
|
||||
prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
|
||||
factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
|
||||
(`python/sglang/srt/managers/template_detection.py`) actually resolve THIS model's
|
||||
(`python/sglang/srt/parser/template_detection.py`) actually resolve THIS model's
|
||||
chat template to the right parser — no rule match means auto silently disables the
|
||||
parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
|
||||
|
||||
|
||||
@@ -17,25 +17,47 @@ logical grouping and remove the old `_jit_` filename prefix.
|
||||
The `sgl_kernel` wheel retains its own public API and packaging tests.
|
||||
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
|
||||
routing in `moe`, attention index selection in `attention`, sampling in
|
||||
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
|
||||
quantization alone belongs in `quantization`.
|
||||
`sampling`, normalization in `layernorm`.
|
||||
- Classify a fused operator by its complete contract; do not split a fused
|
||||
kernel into separate launches just to separate its parts. Split unrelated
|
||||
public entry points that happen to share a source file. Decided rules:
|
||||
- Any cross-rank collective or peer-memory access (all-reduce, all-gather,
|
||||
reduce-scatter, a2a, multicast/NVLS) puts the op in `communication`,
|
||||
whatever compute it fuses.
|
||||
- Producer + output quantization stays with the producer (norm + quant in
|
||||
`layernorm`, activation + quant in `activation`, GEMM with a quant epilogue
|
||||
in `gemm`, MoE dispatch/permute + quant in `moe`). `quantization` holds only
|
||||
pure format conversion: quant, dequant, scale-layout conversion, repack.
|
||||
- RoPE stays in `attention` (standalone RoPE, position ids, QK-norm + RoPE).
|
||||
- `kvcache` owns ops that read/write KV-cache contents; `attention` owns the
|
||||
index/shape/schedule tables (page tables, `cu_seqlens`, `kv_indices`, split
|
||||
metadata) and index selection. A fused compute + store stays with its
|
||||
compute group.
|
||||
- Model-specific files/subpackages and tuning data are allowed **inside** a
|
||||
logical group. Do not add a model bundle or an implementation file directly
|
||||
under `ops/`. Propose a new logical group only for a distinct responsibility,
|
||||
and update `ops._GROUPS`, documentation, and tests together.
|
||||
- Classify a fused operator by its complete contract. Do not split a fused
|
||||
kernel into separate launches just to separate norm, RoPE, or quantization.
|
||||
Split unrelated public entry points that happen to share a source file.
|
||||
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
|
||||
wrappers call it from their logical group. Keep process groups, communicator
|
||||
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
|
||||
`srt/layers/communication/` need not pretend to be generic interfaces.
|
||||
and update `ops._GROUPS`, the README group list, and tests together
|
||||
(`test_kernels_namespace.py` checks the README against `_GROUPS` and the
|
||||
README layout against the directories under `kernels/`).
|
||||
- Agent-generated implementations live in `kernels/kda_kernels/` (Kernel Design
|
||||
Agents, `KernelBackend.KDA`) and `kernels/cake_kernels/` (Cake adapters over
|
||||
FlashInfer). They are backends, not groups: register their public ops in the
|
||||
logical group with a `KernelSpec` whose target points into them, and do not
|
||||
reorganize those directories.
|
||||
- Keep shared build/runtime infrastructure (including the Helion runtime
|
||||
helper) in `kernels/jit`; operator wrappers call it from their logical group.
|
||||
Keep process groups, communicator state, workspace classes, model dispatch,
|
||||
and buffer ownership in `srt` (`srt/layers/communication/` for the K3 and Muse
|
||||
Spark adapters); kernels must not import srt runtime modules to inspect
|
||||
model objects, so pass flags in. Adapters there need not pretend to be generic
|
||||
interfaces.
|
||||
|
||||
## Register the public entry point
|
||||
|
||||
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
|
||||
existing `BaseFusedOp` registration when the operation has interchangeable
|
||||
backends. Group imports must not import GPU implementations or compile kernels.
|
||||
backends (a `BaseFusedOp` built per layer registers `module:Class.forward_<backend>`
|
||||
targets). Group imports must not import GPU implementations or compile kernels.
|
||||
|
||||
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
|
||||
existing backend dispatch and describe actual device/architecture restrictions
|
||||
@@ -53,8 +75,9 @@ factories, reference helpers, and runtime classes are not separate kernel APIs.
|
||||
runtime module. A CUDA allocation does not make a runtime test a kernel test.
|
||||
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
|
||||
standalone test/benchmark entry points in production operator modules.
|
||||
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
|
||||
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
|
||||
- Shared test helpers and reference oracles: `sglang.test.kernels`. Preserve
|
||||
vendored upstream trees and AOT wheel packaging boundaries instead of
|
||||
reorganizing them incidentally.
|
||||
|
||||
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
|
||||
execution entry points, and CI stages/runners. Apportion existing time estimates
|
||||
|
||||
@@ -49,15 +49,15 @@ in-flight row as shipped.
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/kernels/ops/layernorm/fused_rmsnorm.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
||||
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
|
||||
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
|
||||
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/kernels/ops/attention/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/kernels/ops/attention/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
|
||||
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
|
||||
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
|
||||
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
|
||||
@@ -69,18 +69,18 @@ in-flight row as shipped.
|
||||
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
||||
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
||||
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
||||
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/srt/layers/attention/fla/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/kernels/ops/attention/linear/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
|
||||
| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
|
||||
| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/srt/layers/attention/fla/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/srt/layers/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/kernels/ops/attention/linear/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/kernels/ops/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
|
||||
| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
|
||||
|
||||
@@ -146,7 +146,7 @@ Stable entries should be folded into the mainline family rows above.
|
||||
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/kernels/jit/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/kernels/jit/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
||||
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
|
||||
@@ -18,7 +18,7 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
||||
|
||||
## Scheduler-side trace writing
|
||||
|
||||
- `python/sglang/srt/managers/scheduler_profiler_mixin.py`
|
||||
- `python/sglang/srt/managers/scheduler_components/profiler_manager.py`
|
||||
- actual trace start/stop behavior
|
||||
- filename pattern for `TP/DP/PP/EP` and optional stage suffixes
|
||||
- `CUDA_PROFILER` and torch profiler handling
|
||||
@@ -37,6 +37,3 @@ Use these upstream files when the workflow or behavior needs to be justified fro
|
||||
|
||||
- `test/registered/profiling/test_start_profile.py`
|
||||
- validates `/start_profile` behavior, including `CUDA_PROFILER`
|
||||
|
||||
- `test/registered/profiling/test_profile_v2.py`
|
||||
- validates stage-scoped trace outputs under `SGLANG_PROFILE_V2`
|
||||
|
||||
@@ -482,7 +482,9 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="Fused QK RoPE reshape + KV cache write",
|
||||
candidate_path="python/sglang/srt/layers/attention/utils.py",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py"
|
||||
),
|
||||
active_keywords=("fused_qk_rope_reshape_and_cache",),
|
||||
split_groups=(
|
||||
("rotary", "rope", "mrope"),
|
||||
@@ -533,7 +535,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused metadata copy for graph replay",
|
||||
candidate_path="python/sglang/kernels/ops/attention/fused_metadata_copy.py",
|
||||
candidate_path="python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py",
|
||||
active_keywords=(
|
||||
"fused_metadata_copy",
|
||||
"fused_metadata_copy_multi",
|
||||
@@ -700,7 +702,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
pattern="Fused MoE activation + quant / re-quant",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/moe/ep_moe/kernels.py"
|
||||
"<br>python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/srt/layers/moe/cutlass_w4a8_moe.py"
|
||||
),
|
||||
active_keywords=(
|
||||
@@ -761,7 +763,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused quantize + indexed K-cache store",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/attention/fused_store_index_cache.py"
|
||||
"python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py"
|
||||
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
|
||||
),
|
||||
active_keywords=("fused_store_index_k_cache",),
|
||||
|
||||
@@ -15,7 +15,7 @@ One container owns process-static runtime state: `sglang.srt.runtime_context.Run
|
||||
| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
|
||||
| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
|
||||
| per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults |
|
||||
| parallel | `get_parallel()` | one spelling per name: ranks and group handles are the live topology (`@property`, read-through); every other name, sizes included, is a leaf of the parallel config bag | ranks/groups: after dist init; leaves: after publish |
|
||||
| parallel | `get_parallel()` | one spelling per name: scoped override, then permanent stamp, then published parallel config leaf | ranks: stamped at publish; group handles: stamped at distributed init; configured and derived widths: after publish |
|
||||
|
||||
`reset_context()` (unit-test teardown) drops the published config and installs fresh
|
||||
flags/resources/forward tiers.
|
||||
@@ -87,13 +87,12 @@ with what the operator typed, not with what resolution decided.**
|
||||
`ModelRunner.update_model_fields` overrides `model_path` / `load_format` for
|
||||
the target runner.
|
||||
- **Late launcher-stage resolution (pre-publish)**: a few rules cannot run inside
|
||||
`__post_init__` — LoRA normalization, and the auto-parser detection that needs a
|
||||
`ServerArgs.resolve_once` — LoRA normalization, and the auto-parser detection that needs a
|
||||
tokenizer/chat-template load. They are resolution, not mutation, and they
|
||||
**declare** via `arg_groups.overrides.declare_resolution(server_args, source,
|
||||
**fields)`, the same call the rest of the pipeline makes; there is no
|
||||
`declare_late_resolution` any more. *When* a declaration is made is not
|
||||
something the code marks — the guardrails that used to read that marker
|
||||
cover these sites through the ordinary keyword scan instead. The declaration lands
|
||||
something the code marks. The declaration lands
|
||||
in the stash on that very object, so every holder of it carries the decision —
|
||||
the HTTP server, the multi-tokenizer workers it is serialized for, the
|
||||
schedulers it forks — and each of them publishes bags projected from it. The
|
||||
@@ -135,16 +134,23 @@ bag to override at all.
|
||||
instance: **constructor arguments** (`ModelRunner(draft_attention_backend=...)`,
|
||||
`MMEncoder(gpu_id=...)`) and **runner attributes holding the resolved value**
|
||||
(`model_runner.kv_cache_dtype_str`, `prefill_attention_backend_str`,
|
||||
`num_fused_shared_experts`, `linear_attn_backends`) — threaded to consumers as
|
||||
`linear_attn_backends`) — threaded to consumers as
|
||||
arguments, never backfilled onto a shared object. A per-runner choice also stays
|
||||
*out* of the bags: recording it there is how a second runner inherits the first
|
||||
one's answer, which is exactly the bug `linear_attn_backends` replaced. The one sanctioned bend in that rule is
|
||||
*scoped*: `ModelRunner._load_format_scope` exposes the draft's
|
||||
`--speculative-draft-load-format` through `get_model().override(load_format=...)`
|
||||
for exactly the duration of the draft build, because model construction
|
||||
reads that bag leaf — the override restores on exit, so nothing outlives
|
||||
the scope. When there is a runner in hand, read its
|
||||
stamp; that is a different rule from "read the instance".
|
||||
one's answer, which is exactly the bug `linear_attn_backends` replaced. The two
|
||||
sanctioned bends in that rule are *scoped*. `ModelRunner._load_format_scope`
|
||||
exposes the draft's `--speculative-draft-load-format` through
|
||||
`get_model().override(load_format=...)` for exactly the duration of the draft
|
||||
build, because model construction reads that bag leaf — the override restores
|
||||
on exit, so nothing outlives the scope. The shared-expert layout is another scoped view:
|
||||
`get_model().shared_experts` selects the immutable `SharedExpertsConfig` owned
|
||||
by the loaded model (`model.shared_experts_config`). The loader binds it for
|
||||
construction, and runners bind it for execution, warmup and graph capture.
|
||||
Both the fused count and ordinary TopK slot count belong to this object;
|
||||
neither is a separately mutable flag. A scope selects a built layout and
|
||||
restores the previous reference on exit, including exceptions. It cannot
|
||||
resize weights, buffers or a captured graph. When the model is in hand, read
|
||||
its owned config directly (as the routed-experts capturer does).
|
||||
- **Per-instance boundaries** — the tokenizer-manager family, everything under
|
||||
`entrypoints/`, and the tokenizer-process multimodal processors read the bags.
|
||||
The old justification for keeping them on `self.server_args` ("several
|
||||
@@ -178,8 +184,8 @@ bag to override at all.
|
||||
|
||||
### Four ways a config sweep breaks something no test runs
|
||||
|
||||
Each of these shipped in a review round and cost a real defect; each now has a
|
||||
guard, named here so the next sweep checks the same four things by hand first.
|
||||
Each of these shipped in a review round and cost a real defect. Where a test
|
||||
guards one it is named; the next sweep checks all four by hand first.
|
||||
|
||||
1. **The other implementations of an interface.** Dropping a parameter means
|
||||
auditing implementers, not just callers: `CustomSpecAlgo` is the plugin
|
||||
@@ -191,7 +197,8 @@ guard, named here so the next sweep checks the same four things by hand first.
|
||||
`publish` says nothing about whether a given read runs before it. Spawned
|
||||
workers (`MMEncoder` for encoder DP/TP, the Ray scheduler actor) start with
|
||||
an empty context, so a bag read above the publish raises only there.
|
||||
Guard: `test_publish_precedes_bag_reads.py`.
|
||||
Guarded on the launch path only (`test_pre_publish_readers.py`); other
|
||||
entries, spawned workers included, are checked by review.
|
||||
3. **The role namespace a process publishes under.** `ROLE_NAMESPACE_SETS`
|
||||
narrows what each role may read; the DP controller is audited for `exec`
|
||||
alone. A helper that reaches for another namespace passes every default-mode
|
||||
@@ -201,7 +208,9 @@ guard, named here so the next sweep checks the same four things by hand first.
|
||||
means enumerating the others: HTTP, gRPC and in-process `Engine` each have
|
||||
their own server-info and model-info, and each passes its own tests while
|
||||
its users lose the field.
|
||||
Guard: `test_effective_state_surfaces.py`.
|
||||
Guarded for server info across HTTP, both gRPC roles and the Rust frontend
|
||||
(`TestServerInfoTransportParity` in `entrypoints/test_server_info.py`);
|
||||
`Engine.get_server_info` and model info are checked by review.
|
||||
|
||||
A fifth, from the same rounds: the accessor **name itself**. Called as an
|
||||
object member (`manager.get_disagg()`), or shadowed by a same-named import
|
||||
@@ -223,12 +232,13 @@ slack (`>= len(...) - 1`) or compared key names instead of value sources.
|
||||
|
||||
### `get_parallel()`: one spelling per name
|
||||
|
||||
**There is no `.config` hop.** Ranks and group handles are `@property`
|
||||
read-through over the canonical getters, so they answer with the live process
|
||||
groups. Everything else — `tp_size`, `pp_size`, `attn_cp_size`, `dcp_size`,
|
||||
`moe_dp_size` included, alongside config-only leaves like `nccl_port`,
|
||||
`dp_size`, `ep_size`, `dwdp_size` — is answered from the
|
||||
published `parallel` bag. Reading a leaf before publish raises a `ValueError`
|
||||
**There is no `.config` hop.** `ParallelContext._read` answers every name in
|
||||
one order: a scoped override, then a permanent stamp, then the published
|
||||
`parallel` bag leaf. `publish` stamps ranks from the launcher; distributed init
|
||||
stamps group handles and validates their widths. Configured widths such as
|
||||
`tp_size`, `pp_size`, `attn_cp_size`, `dcp_size` and `moe_dp_size`, and config-only
|
||||
leaves such as `nccl_port`, `dp_size`, `ep_size` and `dwdp_size`, fall back to the
|
||||
published bag. Reading a config leaf before publish raises a `ValueError`
|
||||
naming the namespace; an unknown name is an `AttributeError`.
|
||||
|
||||
A size reads from the configuration because the groups are built at exactly the
|
||||
@@ -245,38 +255,33 @@ configured widths — checked at every assignment to `_TP` / `_PP` / `_ATTN_CP`
|
||||
coordinators keep their construction width. Those are different names, not two
|
||||
answers to one name.
|
||||
|
||||
DCP keeps its own pair: `get_parallel().attn_dcp_size` / `.dcp_enabled` answer the
|
||||
*effective* topology (`1` / `False` with no group installed), while `dcp_size` is
|
||||
DCP keeps its own pair: `get_parallel().attn_dcp_size` / `.dcp_enabled` are derived
|
||||
from the published configuration unless explicitly overridden; `dcp_size` is
|
||||
what the launch requested.
|
||||
|
||||
A process-global seed field-read of one of these sizes
|
||||
(`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure. A
|
||||
`server_args` the object was *handed* is a different thing and not a ratchet
|
||||
matter — see "Reads that legitimately stay on a ServerArgs instance".
|
||||
Fail-loud is narrower: before dist init, a live *rank/group* read raises. The six
|
||||
Fail-loud is narrower: an unstamped rank or group handle with no declared default
|
||||
raises; ranks are available after publish, before dist init. The six
|
||||
parallel quotients are not live reads at all — `attn_tp_size`, `attn_dp_size`,
|
||||
`attn_dcp_size`, `moe_ep_size`, `moe_tp_size`, `dcp_enabled` are a function of the
|
||||
configured leaves, computed once at publish into bag leaves, and answered
|
||||
override → stamp → published leaf. So `dcp_enabled` means "the launch configured
|
||||
DCP" (`dcp_size > 1`), not "a DCP group is installed here"; in a scheduler the
|
||||
stamp makes the two identical, in a process that publishes without dist init they
|
||||
differ. `test_a_topology_is_stated_by_naming_the_width` and its neighbours in
|
||||
`test_runtime_context.py` pin this; they replaced
|
||||
DCP" (`dcp_size > 1`), not "a DCP group is installed here". Normal distributed
|
||||
initialization builds the groups to match the configured widths; it does not
|
||||
restamp these DCP leaves. `test_a_topology_is_stated_by_naming_the_width` and its
|
||||
neighbours in `test_runtime_context.py` pin this; they replaced
|
||||
`test_attn_dcp_defaults_when_group_is_uninitialized`. One consequence for tests:
|
||||
overriding a leaf no longer moves its quotient — state a topology by publishing a
|
||||
config, or by naming the width. After init,
|
||||
only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and
|
||||
moe-DP always install, as size-1 aliases if unused). The `config` hop is
|
||||
deliberately dynamo-traceable (a plain property over a slot, no
|
||||
`object.__getattribute__`); gate helpers like `enable_moe_dense_fully_dp()` run inside
|
||||
moe-DP always install, as size-1 aliases if unused). Bare parallel leaf reads are
|
||||
dynamo-traceable; gate helpers like `enable_moe_dense_fully_dp()` run inside
|
||||
compiled model forwards (`test_parallel_config_leaves_trace_under_torch_compile` pins
|
||||
this).
|
||||
|
||||
A third surface carries the same names: `ParallelState` (`self.ps` / `mr.ps`), the
|
||||
frozen per-process snapshot built once in `Scheduler.__init__` from these configured
|
||||
sizes plus this process's ranks, and handed down (draft runners included). Prefer it
|
||||
where an object was handed one; it is not a global accessor.
|
||||
|
||||
### Reading config: the seed is off limits
|
||||
|
||||
`get_server_args().field` in business code is a ratchet failure. Read:
|
||||
@@ -322,12 +327,13 @@ where an object was handed one; it is not a global accessor.
|
||||
follows a post-publish override. Two questions are *not* that, and have their
|
||||
own spelling: the width of the MoE communicator you are about to collectively
|
||||
operate on is `get_moe_cp_size()` (the `_MOE_DP = _ATTN_CP` alias makes it
|
||||
differ), and the effective DCP topology is `get_parallel().attn_dcp_size` /
|
||||
`.dcp_enabled` (`1` / `False` when no group is installed), which does not need
|
||||
dist init to answer.
|
||||
differ), and DCP's derived width and enablement are
|
||||
`get_parallel().attn_dcp_size` / `.dcp_enabled` (published configuration
|
||||
unless explicitly overridden).
|
||||
- **this runner's resolved value** → the runner
|
||||
(`prefill_attention_backend_str`, `kv_cache_dtype_str`,
|
||||
`draft_attention_backend`, `num_fused_shared_experts` on the model).
|
||||
`draft_attention_backend`); for what the model's layers bake in at
|
||||
construction, the layer (`num_fused_shared_experts` on a MoE block).
|
||||
|
||||
`self.server_args.field` is still right for handed per-instance config (see
|
||||
"Reads that legitimately stay on a ServerArgs instance" above for the full set —
|
||||
@@ -335,10 +341,9 @@ per-instance boundaries and whole-object passes; there are no per-runner config
|
||||
copies to read any more). The allow-list is `GrammarManager` and `MMEncoder`;
|
||||
what sits beside it is residue, not a family — and not for one single reason:
|
||||
|
||||
- the tokenizer-manager family and `entrypoints/` **read the bags**; what is
|
||||
left of them in the exposure ratchet is a handful of individually-dispositioned
|
||||
pairs, not a family awaiting conversion. Read the ratchet for the current set
|
||||
rather than assuming a directory is off-limits;
|
||||
- the tokenizer-manager family and `entrypoints/` **read the bags**; review a new
|
||||
instance read there against the raw-input contract rather than assuming a
|
||||
directory is off-limits;
|
||||
- `GrammarManager` is a handed instance for its residual `self.server_args`
|
||||
reads, but backend selection is **not** on the instance any more:
|
||||
`create_grammar_backend` reads `get_exec().kernel.grammar_backend`, and
|
||||
@@ -384,15 +389,15 @@ if you do it, say so in the test.
|
||||
|
||||
### Mid-resolution reads (inside the pipeline only)
|
||||
|
||||
Resolution runs in `__post_init__` and **writes nothing onto the record**: a
|
||||
handler declares (`self._declare` / `declare_resolution`), the declaration goes
|
||||
Resolution runs in `ServerArgs.resolve_once` and **writes nothing onto the record**: a
|
||||
handler declares (`declare_resolution`), the declaration goes
|
||||
into the stash, and the fields keep what the caller passed. So a mid-resolution
|
||||
read of a field answers with the *raw input* — every reader in the pipeline goes
|
||||
through a view instead:
|
||||
|
||||
- `resolving_view(server_args)` / `self._resolved()` — the live view (walks the
|
||||
- `resolving_view(server_args)` — the live view (walks the
|
||||
stash per read). This is what handlers and hooks bind, conventionally as
|
||||
`cfg = resolving_view(self)` at the top of the handler.
|
||||
`cfg = resolving_view(server_args)` at the top of the handler.
|
||||
- `resolved_view(server_args)` — snapshots the overlay when built, which is what
|
||||
a post-process pass wants: it reads the state at *its* slot.
|
||||
|
||||
@@ -451,13 +456,18 @@ Never assign `server_args` fields from model code. Declare instead
|
||||
its checkpoint drives (shared-experts fusion) is a question the *loader* asks
|
||||
the model class — `shared_experts_fusion_disable_reason(hf_config,
|
||||
quant_config)`, a classmethod answering without an instance — at the single
|
||||
model-instantiation point, and
|
||||
`install_shared_experts_fusion_decision` writes the answer to the ACTIVE moe
|
||||
flag before that model's layers build and read it
|
||||
(`is_shared_experts_fusion_disabled`, config-intent fallback).
|
||||
`draft_model_build_scope` brackets every draft build and routes the draft's
|
||||
answer to the speculative leaf, so a draft's decision never overwrites the
|
||||
target's. A process-level load-time fact (the sm80 dtype fallback —
|
||||
model-instantiation point, together with how many shared experts it fuses
|
||||
(`num_fusable_shared_experts(hf_config)`; an entry class without both hooks
|
||||
fuses none). `resolve_shared_experts_config` returns one immutable config;
|
||||
the loader scopes it through `get_model().override(shared_experts=config)`
|
||||
while building and retains it on the resulting model. `num_shared_topk_slots`
|
||||
derives from the fused count and frozen Waterfill policy, so ordinary routing
|
||||
and Waterfill cannot both claim the shared slots. `TopK`/`HashTopK` retain the
|
||||
same config; `top_k` counts only routed experts at construction.
|
||||
`draft_model_build_scope` still selects draft boundary/MMA preferences; it
|
||||
does not save shared-expert flags. Target and draft each own their layout.
|
||||
These values have model lifetime, so they do not belong in `get_forward()`.
|
||||
A process-level load-time fact (the sm80 dtype fallback —
|
||||
device-driven, identical for every runner) records directly via
|
||||
`get_context().override`.
|
||||
|
||||
@@ -466,12 +476,12 @@ dataclass. A declaration against a non-whitelisted field fails at its slot.
|
||||
|
||||
### Load-time vs resolution-time (critical)
|
||||
|
||||
`__post_init__` runs in the launcher process before any model/platform import. Logic that
|
||||
`ServerArgs.resolve_once` runs in the launcher process before any model/platform import. Logic that
|
||||
consults an **extensible registry** (e.g. out-of-tree platforms registering attention
|
||||
backends in `init_backend()`, which runs at `model_runner` import) must stay at load time
|
||||
(ModelRunner init), writing through `get_context().override()`. Before moving any
|
||||
load-time logic into resolution, verify everything it reads is already complete at
|
||||
construction time.
|
||||
resolution time.
|
||||
|
||||
## Runtime flags (`get_flags()`)
|
||||
|
||||
@@ -499,8 +509,8 @@ Named slots + two keyed-lazy registries:
|
||||
side stream is `"lora_side"`. Two call sites may share a name only if their work belongs
|
||||
on one stream — sharing across roles serializes intended overlap.
|
||||
- `get_buffer(name, factory)` — get-or-create a named persistent buffer. Grow-only or
|
||||
per-device semantics manage their `resources.buffers` entries directly (see tokenspeed /
|
||||
SM120 split / Marlin workspace). Buffer names are per-backend today; do not silently
|
||||
per-device semantics manage their `resources.buffers` entries directly (see SM120
|
||||
split / Marlin workspace). Buffer names are per-backend today; do not silently
|
||||
share.
|
||||
- Singletons with manager semantics (EP dispatcher buffers, EPLB recorder/metadata, graph
|
||||
memory pool) keep their owning accessors/classes as facades; only the *state* lives in a
|
||||
@@ -543,8 +553,9 @@ ONE thread — do not design for TBO threads that don't exist.
|
||||
them explicitly on the mock; `MagicMock(spec=...)` raises on attributes that only
|
||||
exist post-`__init__`, which is the fastest way to find a missed stub.
|
||||
- `reset_context()` in teardown when a test publishes outside a scoped override.
|
||||
- `ServerArgs(model_path="dummy")` early-returns the pipeline (few declarations, no
|
||||
strict guard) — fine for lightweight fixtures.
|
||||
- `ServerArgs(model_path="dummy")` resolves through the pipeline's early return (few
|
||||
declarations) — fine for lightweight fixtures. The strict guard still applies once
|
||||
`resolve_once` has run, as it does at `publish`.
|
||||
- **Asserting what resolution decided reads `resolution_result(sa, "field")`**, not
|
||||
`sa.field`: the field is the raw input. Assert the field only when the point of
|
||||
the case *is* that the record stayed pristine (the FA4 page-size and waterfill
|
||||
@@ -553,7 +564,7 @@ ONE thread — do not design for TBO threads that don't exist.
|
||||
pytest run lets a context published by an earlier file mask a missing-publish bug in a
|
||||
later one.
|
||||
|
||||
## Guardrails (these fail CI; what to do when they fire)
|
||||
## Guardrails and review rules
|
||||
|
||||
1. **Strict mutation guard** (always on, and with no exception): bare
|
||||
`server_args.x = ...` after resolution raises unconditionally in
|
||||
@@ -563,10 +574,9 @@ ONE thread — do not design for TBO threads that don't exist.
|
||||
resolved config with `get_context().override`; hand a per-runner value to its
|
||||
runner as a constructor argument. Projected bags are sealed the same way (leaf
|
||||
assignment raises).
|
||||
2. **Mutation ratchet** (`test_server_args_mutation_ratchet.py`, exact pin 0 over the whole
|
||||
package minus the pipeline / multimodal_gen): textual scan for assignment forms. Never
|
||||
raise the baseline.
|
||||
3. **No-copy contract** (`test_server_args_no_instance_mutation_entry.py`): neither
|
||||
2. **No textual mutation ratchet**: the strict guard above is the only mutation
|
||||
check.
|
||||
3. **No-copy contract** (review only, no automated guard): neither
|
||||
`ServerArgs.override` nor `ServerArgs.derive` exists, and nothing in the package
|
||||
calls either form. Rerouting a writer to the bags means flipping **all its readers
|
||||
in the same commit** (no transitional dual-write).
|
||||
@@ -577,23 +587,23 @@ ONE thread — do not design for TBO threads that don't exist.
|
||||
runner's stamp — not `get_server_args().field`, which the read ratchet below pins
|
||||
at zero. `runtime_context.get_server_args()` is only for the
|
||||
whole-object shapes (dumps, provenance, a hand-off to a callee that takes a config).
|
||||
5. **Global config read ratchet** (`test_global_config_read_ratchet.py`): baselines are
|
||||
**0** for both the direct `get_server_args().field` and the alias form (function-local
|
||||
5. **Global config read check** (`test_global_config_read_ratchet.py`): no business code
|
||||
reads a field off `get_server_args()`, directly or through an alias (function-local
|
||||
— including local copies of an alias, `cfg = sa` — module-level, or parked on an
|
||||
instance attribute, plus the `getattr(..., "field")` spelling of each; a name
|
||||
computed at runtime or indirection deeper than a local name copy is census-tool
|
||||
territory, per the test's docstring). The scanner matches `get_server_args` by its
|
||||
computed at runtime or indirection deeper than a local name copy is outside the
|
||||
check, per the test's docstring). The scanner matches `get_server_args` by its
|
||||
literal name — bare or module-qualified (`ctx.get_server_args()`) — and
|
||||
`TestNoRenamedAccessorImports` in the same file *bans* `import ... as` renames of it,
|
||||
which is what makes literal-name matching sound. Exempt by owner
|
||||
module only (`runtime_context.py`, `server_args.py`, `arg_groups/`). Two classes,
|
||||
no more: `TestGlobalConfigReadRatchet` holds the two baselines and
|
||||
no more: `TestGlobalConfigReads` holds the zero check and
|
||||
`TestNoRenamedAccessorImports` holds the ban. There is no configured-size registry
|
||||
here any longer — `get_parallel()` has one spelling per name, so a size read is not a
|
||||
choice between two answers and nothing needs registering.
|
||||
6. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the
|
||||
flag-owning layers are pinned by name. A new module-level runtime global belongs on a
|
||||
flags group / resources slot instead; migrating a pinned survivor must shrink the pin.
|
||||
6. **Module-state rule** (review only, no automated guard): a new module-level
|
||||
runtime global in the flag-owning layers belongs on a flags group / resources slot
|
||||
instead.
|
||||
7. **Namespace coverage** (`test_server_args_namespaces.py`,
|
||||
`test_runtime_context_config_bags.py`): every `ServerArgs` field resolves to a
|
||||
namespace — from the `arg_groups/fields/` class that declares it — and the
|
||||
@@ -649,8 +659,7 @@ Key source files: `python/sglang/srt/runtime_context.py` (the container, every t
|
||||
`python/sglang/srt/arg_groups/overrides.py` (override registry, passes,
|
||||
`declare_resolution` and the spellings around it), `python/sglang/srt/server_args.py` (`NS` metadata,
|
||||
`Arg(..., resolvable=True)`, `__setattr__` strict guard), and the guardrail tests under
|
||||
`test/registered/unit/` (`test_server_args_mutation_ratchet.py`,
|
||||
`test_global_config_read_ratchet.py`,
|
||||
`test_module_state_ratchet.py`, `test_server_args_namespaces.py`,
|
||||
`test/registered/unit/` (`test_global_config_read_ratchet.py`,
|
||||
`test_server_args_namespaces.py`,
|
||||
`test_runtime_context.py` — the last one doubles
|
||||
as executable documentation of every tier's semantics).
|
||||
|
||||
@@ -36,4 +36,4 @@ a real `AttributeError` if the field is ever renamed:
|
||||
revision=getattr(server_args, "revision", None), # BAD
|
||||
revision=server_args.revision, # GOOD
|
||||
```
|
||||
(see `python/sglang/srt/managers/template_detection.py`)
|
||||
(see `python/sglang/srt/parser/template_detection.py`)
|
||||
|
||||
@@ -27,8 +27,8 @@ A new unit test case must fall into one of these categories:
|
||||
3. **Critical-path bookkeeping.** Defends conventions that are easy to break by
|
||||
forgetting to sync -- registry completeness, field lifecycle, serialization
|
||||
compatibility. Enumerating assertions are fine here; the guarded failure
|
||||
mode is "someone extended X without updating Y". Example: the ratchet tests
|
||||
(`test/registered/unit/test_module_state_ratchet.py`).
|
||||
mode is "someone extended X without updating Y". Example:
|
||||
`test/registered/unit/test_global_config_read_ratchet.py`.
|
||||
|
||||
Not admissible:
|
||||
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
.github @merrymercy @Fridge003 @ispobock @Kangyan-Zhou @HaiShaw @bingxche
|
||||
/docker @Fridge003 @ispobock @HaiShaw @ishandhanani @yctseng0211 @sogalin
|
||||
/docker/npu.Dockerfile @ping1jing2 @iforgetmyname @whybeyoung
|
||||
/docker/sgl-router.Dockerfile @sherlockwu @Kangyan-Zhou @ShangmingCai
|
||||
/docs @wisclmy0611 @zijiexia @sogalin
|
||||
/docs @wisclmy0611 @zijiexia @Richardczl98 @JustinTong0323 @sogalin
|
||||
/python/pyproject.toml @merrymercy @Fridge003 @ispobock
|
||||
/python/sglang/kernels @DarkSharpness @HaiShaw @BBuf @celve @HydraQYH @yuan-luo
|
||||
/python/sglang/kernels/ops/diffusion @yingluosanqian @BBuf @mickqian @triple-mu @niehen6174
|
||||
/python/sglang/kernels/ops/attention/fla @yizhang2077 @hebiao064 @yuan-luo
|
||||
/python/sglang/kernels/ops/attention/linear @yizhang2077 @hebiao064 @yuan-luo
|
||||
/python/sglang/kernels/ops/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
|
||||
/python/sglang/kernels/jit/csrc/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
|
||||
/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf @triple-mu @kevin-mii @niehen6174
|
||||
@@ -81,6 +83,11 @@
|
||||
/rust/sglang-processor @sagearc @sherlockwu
|
||||
/rust/sglang-radix-tree @Jialin @alphabetc1
|
||||
/rust/sglang-server @rainj-me @merrymercy
|
||||
/sgl-router @sherlockwu @Kangyan-Zhou @ShangmingCai @hzh0425
|
||||
/sgl-router/sgl-kv-indexer @sherlockwu @lixiufei-leo @ShangmingCai @wuyl1 @hzh0425
|
||||
/sgl-router/src/policies @sherlockwu @ShangmingCai @Bo-Vincent
|
||||
/sgl-router/src/workers @sherlockwu @AMD-yanfeiwang
|
||||
/sgl-router/src/protocol @sherlockwu @whybeyoung
|
||||
/sgl-model-gateway @slin1237 @CatherineSue
|
||||
/sgl-model-gateway/benches @slin1237
|
||||
/sgl-model-gateway/bindings/python @CatherineSue @key4ng @slin1237
|
||||
@@ -107,6 +114,7 @@
|
||||
/python/sglang/srt/speculative/cpp_ngram @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/srt/speculative/frozen_kv_mtp_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl @pyc96
|
||||
/python/sglang/kernels/ops/speculative/ngram_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/kernels/ops/embeddings/ngram_embedding.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/kernels/jit/csrc/ngram_corpus @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/srt/models/laguna.py @Jiminator
|
||||
/python/sglang/srt/configs/laguna.py @Jiminator
|
||||
|
||||
@@ -14,9 +14,9 @@ Recognized permission keys:
|
||||
| --- | --- |
|
||||
| `can_tag_run_ci_label` | `/tag-run-ci-label`, `/tag-and-rerun-ci`, `/run-full-ci`, `/run-extra-ci` |
|
||||
| `can_rerun_failed_ci` | `/rerun-failed-ci`, `/tag-and-rerun-ci`, `/run-full-ci`, `/run-extra-ci` |
|
||||
| `cooldown_interval_minutes` | rate limit in `pr-gate.yml`; `0` also grants `/rerun-test`, `/rerun-group` |
|
||||
| `cooldown_interval_minutes` | rate limit in `pr-gate.yml`; `0` also grants `/rerun-test`, `/rerun-group`, `/run-cpu-ci` |
|
||||
|
||||
`/rerun-test` and `/rerun-group` are gated on the commenter alone: either
|
||||
`/rerun-test`, `/rerun-group` and `/run-cpu-ci` are gated on the commenter alone: either
|
||||
`cooldown_interval_minutes: 0`, or `write`/`admin` permission on the repo. Where
|
||||
the PR comes from makes no difference, and authoring it grants nothing.
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ inputs:
|
||||
required: false
|
||||
default: ''
|
||||
cache_key_prefix:
|
||||
description: 'Must match what _pr-test-rust-ext-build.yml saves under.'
|
||||
description: 'Must match what _pr-test-rust-ext-build.yml saves under. ARM64 runners always restore rust-ext-aarch64-cp312.'
|
||||
required: false
|
||||
default: rust-ext-x86_64-cp312
|
||||
|
||||
@@ -53,7 +53,7 @@ runs:
|
||||
path: |
|
||||
python/sglang/srt/rust_extensions/_*.so
|
||||
python/sglang/srt/mem_cache/rust_tree_core/mem_cache*.so
|
||||
key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'proto/**', 'python/setup.py', 'python/pyproject.toml', 'python/sglang/srt/rust_extensions/torch_build.py', '.github/workflows/_pr-test-rust-ext-build.yml', 'scripts/ci/utils/stage_rust_ext_modules.sh') }}
|
||||
key: ${{ runner.arch == 'ARM64' && 'rust-ext-aarch64-cp312' || inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'proto/**', 'python/setup.py', 'python/pyproject.toml', 'python/sglang/srt/rust_extensions/torch_build.py', '.github/workflows/_pr-test-rust-ext-build.yml', 'scripts/ci/utils/stage_rust_ext_modules.sh') }}
|
||||
|
||||
# Job-wide, but only setup.py reads it, and only while building.
|
||||
# Whether the modules suit this interpreter is not decided here:
|
||||
|
||||
@@ -135,6 +135,28 @@ hicache:
|
||||
- 'python/sglang/srt/mem_cache/**/*storage*'
|
||||
- 'python/sglang/srt/mem_cache/l2_transfer.py'
|
||||
|
||||
# A *mooncake* filename glob also matches benchmark/datasets/mooncake.py.
|
||||
mooncake:
|
||||
- changed-files:
|
||||
- any-glob-to-any-file:
|
||||
# PD KV transfer.
|
||||
- 'python/sglang/srt/disaggregation/mooncake/**/*'
|
||||
- 'python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py'
|
||||
- 'python/sglang/srt/observability/mooncake_trace.py'
|
||||
- 'test/registered/unit/disaggregation/**/*mooncake*'
|
||||
- 'test/manual/kv_transfer/**/*mooncake*'
|
||||
# HiCache store.
|
||||
- 'python/sglang/srt/mem_cache/storage/mooncake_store/**/*'
|
||||
- 'python/sglang/test/mooncake_utils.py'
|
||||
- 'test/registered/unit/mem_cache/**/*mooncake*'
|
||||
- 'test/registered/hicache/**/*mooncake*'
|
||||
# MoE EP. elastic_ep.py also implements nixl.
|
||||
- 'python/sglang/srt/layers/moe/token_dispatcher/mooncake.py'
|
||||
- 'python/sglang/srt/elastic_ep/expert_backup_client.py'
|
||||
- 'python/sglang/srt/elastic_ep/expert_backup_manager.py'
|
||||
- 'test/registered/ep/**/*mooncake*'
|
||||
- 'test/manual/ep/**/*mooncake*'
|
||||
|
||||
# Deterministic
|
||||
deterministic:
|
||||
- changed-files:
|
||||
|
||||
@@ -1,133 +0,0 @@
|
||||
# Reusable workflow: cross-reference CI test failures with coverage-based
|
||||
# test recommendations. Called by pr-test-npu.yml's analyze-failure-report job.
|
||||
name: Analyze Failure Report
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
log_artifact_pattern:
|
||||
type: string
|
||||
required: true
|
||||
description: 'Glob pattern for test log artifacts (e.g. selected-test-logs-*)'
|
||||
diffusion_artifact_pattern:
|
||||
type: string
|
||||
required: false
|
||||
default: 'diffusion-failures-*'
|
||||
description: 'Glob pattern for diffusion failure artifacts (consistency failures)'
|
||||
recommendations_content:
|
||||
type: string
|
||||
required: false
|
||||
default: ''
|
||||
description: 'Content of recommendations from upstream job output'
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash -el {0}
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
name: cross-reference failures with recommendations
|
||||
continue-on-error: true # Non-blocking on parse/read failure.
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout sglang repo
|
||||
uses: actions/checkout@v4
|
||||
|
||||
# ----- Step 1: Locate recommendations -----
|
||||
- name: Locate recommendations file
|
||||
id: locate-recs
|
||||
run: |
|
||||
mkdir -p ./test-logs
|
||||
# Always create the file so the analyzer can run on diffusion-only failures.
|
||||
touch ./test-logs/_recommended.txt
|
||||
echo "RECS_FILE=./test-logs/_recommended.txt" >> $GITHUB_ENV
|
||||
|
||||
if [ -n "${{ inputs.recommendations_content }}" ]; then
|
||||
echo "has_recommendations=true" >> $GITHUB_OUTPUT
|
||||
echo "Using workflow output recommendations"
|
||||
echo "${{ inputs.recommendations_content }}" > ./test-logs/_recommended.txt
|
||||
echo "RECS_SOURCE=output" >> $GITHUB_ENV
|
||||
COUNT=$(wc -l < ./test-logs/_recommended.txt)
|
||||
echo "Recommendations count: ${COUNT}"
|
||||
else
|
||||
echo "has_recommendations=false" >> $GITHUB_OUTPUT
|
||||
echo "RECS_SOURCE=none" >> $GITHUB_ENV
|
||||
echo "::notice::No recommended test cases detected; analyzing log and diffusion failures only."
|
||||
fi
|
||||
|
||||
# ----- Step 2: Download test logs -----
|
||||
# continue-on-error is a step keyword, not a download-artifact input.
|
||||
- name: Download test logs
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: ${{ inputs.log_artifact_pattern }}
|
||||
path: ./test-logs
|
||||
merge-multiple: true
|
||||
continue-on-error: true
|
||||
|
||||
# ----- Step 2b: Download diffusion failure artifacts -----
|
||||
# Keep merge-multiple off so each partition's consistency_failures/summary.json
|
||||
# lands in its own <artifact-name>/ directory instead of clobbering the others.
|
||||
- name: Download diffusion failure artifacts
|
||||
uses: actions/download-artifact@v7
|
||||
with:
|
||||
pattern: ${{ inputs.diffusion_artifact_pattern }}
|
||||
path: ./diffusion-failures
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check for diffusion failure artifacts
|
||||
id: check-diffusion
|
||||
run: |
|
||||
if compgen -G "./diffusion-failures/*/consistency_failures/summary.json" > /dev/null; then
|
||||
echo "found=true" >> $GITHUB_OUTPUT
|
||||
echo "Diffusion consistency failure artifacts found"
|
||||
else
|
||||
echo "found=false" >> $GITHUB_OUTPUT
|
||||
echo "No diffusion consistency failure artifacts"
|
||||
fi
|
||||
|
||||
- name: Show downloaded artifacts structure
|
||||
run: |
|
||||
echo "Downloaded log files:"
|
||||
find ./test-logs -type f | head -50 || echo " (no logs found)"
|
||||
echo "Downloaded diffusion failure artifacts:"
|
||||
if compgen -G "./diffusion-failures/*" > /dev/null; then
|
||||
find ./diffusion-failures -type f | head -50
|
||||
else
|
||||
echo " (no diffusion failure artifacts found)"
|
||||
fi
|
||||
|
||||
# ----- Step 3: Run analysis -----
|
||||
- name: Ensure regex is available
|
||||
if: ${{ steps.locate-recs.outputs.has_recommendations != 'false' || steps.check-diffusion.outputs.found == 'true' }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if ! python3 -c "import regex" >/dev/null 2>&1; then
|
||||
echo "regex not found for $(python3 -V); bootstrapping pip and installing regex"
|
||||
curl -fsSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
|
||||
python3 get-pip.py --break-system-packages
|
||||
python3 -m pip install --break-system-packages regex
|
||||
fi
|
||||
python3 -c "import regex; print('regex ok:', regex.__file__)"
|
||||
|
||||
- name: Run failure analysis
|
||||
if: ${{ steps.locate-recs.outputs.has_recommendations != 'false' || steps.check-diffusion.outputs.found == 'true' }}
|
||||
id: analysis
|
||||
continue-on-error: true
|
||||
run: |
|
||||
python3 scripts/ci/npu/precise-test/analyze_failure_report.py \
|
||||
--log-dir ./test-logs \
|
||||
--diffusion-dir ./diffusion-failures \
|
||||
--recommendations-file "${RECS_FILE}" \
|
||||
--recommendations-source "${RECS_SOURCE:-none}" \
|
||||
--output ./test-logs/failure_report.md
|
||||
|
||||
# ----- Step 4: Upload report -----
|
||||
- name: Upload failure analysis report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: failure-analysis-report
|
||||
path: ./test-logs/failure_report.md
|
||||
if-no-files-found: ignore
|
||||
retention-days: 14
|
||||
@@ -50,10 +50,10 @@ on:
|
||||
description: 'If true, delegate test execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh instead of running via run_suite.py directly'
|
||||
type: boolean
|
||||
default: false
|
||||
upload_test_logs:
|
||||
description: 'If true, upload the test logs as a GitHub Actions artifact (only enabled by pr-test-npu)'
|
||||
type: boolean
|
||||
default: false
|
||||
only_tests:
|
||||
description: 'Coverage-recommended test files for this suite (one repo-relative path per line, from recommend-tests-from-coverage). Empty = run the full suite.'
|
||||
type: string
|
||||
default: ''
|
||||
github-token:
|
||||
description: 'GitHub token for API calls'
|
||||
type: string
|
||||
@@ -70,6 +70,7 @@ env:
|
||||
|
||||
jobs:
|
||||
run:
|
||||
name: ${{ inputs.use_coverage_runner && inputs.self_name || 'run' }} (${{ matrix.partition }})
|
||||
runs-on: ${{ inputs.runner_config }}
|
||||
container:
|
||||
image: ${{ inputs.image }}
|
||||
@@ -241,6 +242,7 @@ jobs:
|
||||
env:
|
||||
CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
|
||||
TEST_LOG_DIR: /tmp/test-logs
|
||||
ONLY_TESTS: ${{ inputs.only_tests }}
|
||||
shell: bash
|
||||
run: |
|
||||
# Fail fast on any command error, undefined variable, or pipe failure.
|
||||
@@ -284,6 +286,16 @@ jobs:
|
||||
NIGHTLY_FLAG="--nightly --continue-on-error"
|
||||
fi
|
||||
|
||||
# Precision-test mode: run only the coverage-recommended files for
|
||||
# this suite. The bucketed list comes from recommend-tests-from-
|
||||
# coverage; empty input keeps the full-suite behavior.
|
||||
ONLY_FILE_ARGS=""
|
||||
if [ -n "${ONLY_TESTS}" ]; then
|
||||
printf '%s\n' "${ONLY_TESTS}" > /tmp/precision_tests.txt
|
||||
ONLY_FILE_ARGS="--only-file-list /tmp/precision_tests.txt"
|
||||
echo "Precision mode: running $(wc -l < /tmp/precision_tests.txt) recommended file(s)"
|
||||
fi
|
||||
|
||||
# Optional: delegate execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh when the
|
||||
# caller sets use_coverage_runner=true. list_tests.py (same directory) enumerates the
|
||||
# selected test files; the original run_suite.py execution below is preserved
|
||||
@@ -308,13 +320,4 @@ jobs:
|
||||
--auto-partition-id ${{ matrix.partition }} \
|
||||
--auto-partition-size ${{ fromJson(inputs.partitions).size }} \
|
||||
${{ inputs.timeout_per_file && format('--timeout-per-file {0}', inputs.timeout_per_file) || '' }} \
|
||||
$NIGHTLY_FLAG $CONTINUE_ON_ERROR_FLAG 2>&1 | tee "${LOG_FILE}"
|
||||
|
||||
- name: Upload test logs
|
||||
if: always() && inputs.use_coverage_runner != true && inputs.upload_test_logs == true
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: selected-test-logs-${{ inputs.self_name }}-part${{ matrix.partition }}
|
||||
path: /tmp/test-logs/
|
||||
if-no-files-found: ignore
|
||||
retention-days: 14
|
||||
$ONLY_FILE_ARGS $NIGHTLY_FLAG $CONTINUE_ON_ERROR_FLAG 2>&1 | tee "${LOG_FILE}"
|
||||
|
||||
@@ -71,10 +71,10 @@ on:
|
||||
description: 'If true, delegate test execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh instead of running via run_suite.py directly'
|
||||
type: boolean
|
||||
default: false
|
||||
upload_test_logs:
|
||||
description: 'If true, upload the test logs as a GitHub Actions artifact (only enabled by pr-test-npu)'
|
||||
type: boolean
|
||||
default: false
|
||||
only_tests:
|
||||
description: 'Coverage-recommended test files for this suite (one repo-relative path per line, from recommend-tests-from-coverage). Empty = run the full suite.'
|
||||
type: string
|
||||
default: ''
|
||||
|
||||
env:
|
||||
SKIP_PR_TEST_HEALTH_CHECK: ${{ inputs.skip_pr_test_health_check }}
|
||||
@@ -85,7 +85,7 @@ concurrency:
|
||||
|
||||
jobs:
|
||||
e2e:
|
||||
name: ${{ inputs.test_suite }}
|
||||
name: ${{ inputs.test_suite }}${{ inputs.use_coverage_runner && format(' ({0})', matrix.partition) || '' }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -238,6 +238,7 @@ jobs:
|
||||
SGLANG_IS_IN_CI: true
|
||||
TRANSFORMERS_VERBOSITY: "error"
|
||||
GDN_ATTN_BACKEND_TRITON: 1
|
||||
ONLY_TESTS: ${{ inputs.only_tests }}
|
||||
shell: bash
|
||||
run: |
|
||||
# Fail fast on any command error, undefined variable, or pipe failure.
|
||||
@@ -401,6 +402,16 @@ jobs:
|
||||
PARTITION_ARGS="--auto-partition-id ${PARTITION_ID} --auto-partition-size ${PARTITION_SIZE}"
|
||||
fi
|
||||
|
||||
# Precision-test mode: run only the coverage-recommended files for
|
||||
# this suite. The bucketed list comes from recommend-tests-from-
|
||||
# coverage; empty input keeps the full-suite behavior.
|
||||
ONLY_FILE_ARGS=""
|
||||
if [ -n "${ONLY_TESTS}" ]; then
|
||||
printf '%s\n' "${ONLY_TESTS}" > /tmp/precision_tests.txt
|
||||
ONLY_FILE_ARGS="--only-file-list /tmp/precision_tests.txt"
|
||||
echo "Precision mode: running $(wc -l < /tmp/precision_tests.txt) recommended file(s)"
|
||||
fi
|
||||
|
||||
# Append `--timeout-from-est‑time` when the job belongs to the nightly pipeline.
|
||||
# per‑case timeout is derived from each case's est_time.
|
||||
TIMEOUT_FLAG=""
|
||||
@@ -435,7 +446,7 @@ jobs:
|
||||
# Run NPU test suite with run_suite.py.
|
||||
# Capture mode (--enable-retry --max-attempts 1) keeps test subprocesses off the
|
||||
# tee pipe, so leftover server processes cannot block the pipeline on exit.
|
||||
${PYTHON_FOR_SGLANG} -u run_suite.py --hw npu --suite ${test_suite} ${NIGHTLY_FLAG} ${PARTITION_ARGS} ${TIMEOUT_FLAG} --enable-retry --max-attempts 1 --timeout-per-file 7200 2>&1 | tee "${LOG_TEE_TARGETS[@]}" || test_exit_code=$?
|
||||
${PYTHON_FOR_SGLANG} -u run_suite.py --hw npu --suite ${test_suite} ${NIGHTLY_FLAG} ${PARTITION_ARGS} ${ONLY_FILE_ARGS} ${TIMEOUT_FLAG} --enable-retry --max-attempts 1 --timeout-per-file 7200 2>&1 | tee "${LOG_TEE_TARGETS[@]}" || test_exit_code=$?
|
||||
|
||||
echo "Finished test: ${tc_name}"
|
||||
|
||||
@@ -467,12 +478,3 @@ jobs:
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
exit ${test_exit_code}
|
||||
|
||||
- name: Upload test logs
|
||||
if: always() && inputs.use_coverage_runner != true && inputs.upload_test_logs == true
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: selected-test-logs-${{ inputs.test_suite }}-part${{ matrix.partition }}
|
||||
path: /tmp/test-logs/
|
||||
if-no-files-found: ignore
|
||||
retention-days: 14
|
||||
|
||||
@@ -95,7 +95,7 @@ jobs:
|
||||
- "python/setup.py"
|
||||
- "python/sglang/!(multimodal_gen|kernels)/**/!(*.md)"
|
||||
# Diffusion-only kernels and tests are left to multimodal_gen and
|
||||
# jit_kernel; ops/diffusion/common stays since srt imports it via kda_kernels.
|
||||
# jit_kernel; ops/diffusion/common stays since kda_kernels (an srt path) imports it.
|
||||
- "python/sglang/kernels/{,ops/,jit/,jit/csrc/}!(*.md)"
|
||||
- "python/sglang/kernels/!(aot|ops|jit)/**/!(*.md)"
|
||||
- "python/sglang/kernels/jit/!(csrc)/**/!(*.md)"
|
||||
@@ -130,6 +130,7 @@ jobs:
|
||||
simulator:
|
||||
- ".github/workflows/pr-test-extra.yml"
|
||||
- ".github/workflows/_pr-test-simulator-cpu.yml"
|
||||
- ".github/workflows/_pr-test-simulator-gpu.yml"
|
||||
- "benchmark/simulator/**/!(*.md)"
|
||||
- "tools/sglang-simulator/**/!(*.md)"
|
||||
multimodal_gen:
|
||||
|
||||
@@ -50,7 +50,7 @@ jobs:
|
||||
strategy:
|
||||
matrix:
|
||||
include:
|
||||
- python-version: "3.10"
|
||||
- python-version: "3.11"
|
||||
cuda-version: "13.0"
|
||||
name: ${{ inputs.job_display_name }}
|
||||
steps:
|
||||
|
||||
@@ -70,7 +70,7 @@ jobs:
|
||||
run: |
|
||||
uv pip install -e "python" --index-strategy unsafe-best-match --prerelease allow
|
||||
uv pip install pytest
|
||||
uv pip install -e "tools/sglang-simulator[aic]"
|
||||
uv pip install "./tools/sglang-simulator[aic]"
|
||||
|
||||
- name: Prebuild HiCache native hash extension
|
||||
timeout-minutes: 5
|
||||
@@ -91,6 +91,7 @@ jobs:
|
||||
PYTEST_ADDOPTS: "-s"
|
||||
TORCH_EXTENSIONS_DIR: ${{ runner.temp }}/torch-extensions
|
||||
run: |
|
||||
OMP_NUM_THREADS=2 python3 -m pytest -q tools/sglang-simulator/test/test_train_latency_model.py
|
||||
python3 -m pytest -q tools/sglang-simulator/test/test_simulation_sglang_runner.py
|
||||
python3 -m pytest -q tools/sglang-simulator/test/test_simulation_sglang_serving.py
|
||||
python3 -m pytest -q tools/sglang-simulator/test/test_simulation_cache_hit_ratio.py
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
name: PR Test SGLang Simulator (GPU)
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
check_changes:
|
||||
description: 'toJson(needs.check-changes.outputs).'
|
||||
type: string
|
||||
required: true
|
||||
caller_inputs:
|
||||
description: 'toJson(inputs) from the caller workflow.'
|
||||
type: string
|
||||
required: true
|
||||
rust_ext_artifact:
|
||||
description: 'Artifact of prebuilt Rust extension modules.'
|
||||
type: string
|
||||
default: ''
|
||||
|
||||
env:
|
||||
SGLANG_IS_IN_CI: true
|
||||
SKIP_PR_TEST_HEALTH_CHECK: ${{ (fromJson(inputs.caller_inputs).skip_pr_test_health_check || fromJson(inputs.caller_inputs).test_parallel_dispatch || fromJson(inputs.caller_inputs).run_all_tests) && 'true' || 'false' }}
|
||||
PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
|
||||
USE_VENV: "1"
|
||||
|
||||
jobs:
|
||||
run:
|
||||
name: simulator-test-gpu
|
||||
if: fromJson(inputs.check_changes).main_package == 'true' || fromJson(inputs.check_changes).simulator == 'true' || fromJson(inputs.check_changes).sgl_kernel == 'true'
|
||||
runs-on: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)['1-gpu-small'] }}
|
||||
timeout-minutes: 40
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ fromJson(inputs.caller_inputs).git_ref || github.sha }}
|
||||
|
||||
- uses: ./.github/actions/check-pr-test-health
|
||||
|
||||
- uses: ./.github/actions/check-maintenance
|
||||
|
||||
- name: Download kernel wheels
|
||||
if: fromJson(inputs.check_changes).sgl_kernel == 'true'
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- uses: ./.github/actions/download-rust-ext
|
||||
with:
|
||||
artifact_name: ${{ inputs.rust_ext_artifact }}
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
CUSTOM_BUILD_SGL_KERNEL: ${{ fromJson(inputs.check_changes).sgl_kernel }}
|
||||
run: bash scripts/ci/cuda/ci_install_dependency.sh
|
||||
|
||||
- name: Test baseline collection
|
||||
timeout-minutes: 10
|
||||
env:
|
||||
PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1"
|
||||
run: |
|
||||
uv pip install "./tools/sglang-simulator"
|
||||
python3 -m pytest -q tools/sglang-simulator/test/test_baseline_collection_sglang.py
|
||||
|
||||
- name: Cleanup venv
|
||||
if: always()
|
||||
run: bash scripts/ci/cuda/ci_cleanup_venv.sh
|
||||
@@ -17,7 +17,7 @@ on:
|
||||
type: string
|
||||
required: true
|
||||
caller_inputs:
|
||||
description: 'toJson(inputs) from pr-test.yml. Read via fromJson(...).git_ref / skip_pr_test_health_check / test_parallel_dispatch / run_all_tests.'
|
||||
description: 'toJson(inputs) from pr-test.yml. Read via fromJson(...).git_ref / skip_pr_test_health_check / test_parallel_dispatch / run_all_tests / skip_cache_save.'
|
||||
type: string
|
||||
required: true
|
||||
partitions:
|
||||
@@ -110,7 +110,7 @@ jobs:
|
||||
with:
|
||||
workspaces: rust
|
||||
shared-key: "sglang-grpc-cpu"
|
||||
save-if: ${{ matrix.partition == 0 }}
|
||||
save-if: ${{ matrix.partition == 0 && !fromJson(inputs.caller_inputs).skip_cache_save }}
|
||||
|
||||
# uv pip targets a venv by default; setup-python has no venv — install into that interpreter (see UV_SYSTEM_PYTHON in https://docs.astral.sh/uv/guides/integration/github/)
|
||||
- name: Install dependencies
|
||||
@@ -156,9 +156,12 @@ jobs:
|
||||
python3 run_suite.py --hw cpu --suite ${{ inputs.self_name }} --auto-partition-id ${{ matrix.partition }} --auto-partition-size ${{ fromJson(inputs.partitions)[inputs.self_name].size }} --partition-model-file /tmp/partition-model.json --fork-worker-batch-size 20 $CONTINUE_ON_ERROR_FLAG
|
||||
|
||||
# PRs save only what the base lacks; the implicit if: success() keeps a
|
||||
# half-downloaded cache unpublished.
|
||||
# half-downloaded cache unpublished. skip_cache_save is for callers that
|
||||
# dispatch on main but test PR code (run-cpu-ci.yml): a save there would
|
||||
# publish PR-written files under main's cache scope.
|
||||
- name: Decide whether to save the HF cache
|
||||
id: hf_cache_save
|
||||
if: ${{ !fromJson(inputs.caller_inputs).skip_cache_save }}
|
||||
run: |
|
||||
size=$(du -sb "$HF_HOME" 2>/dev/null | cut -f1)
|
||||
after=${size:-0}
|
||||
|
||||
@@ -147,7 +147,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Download artifacts (v6)
|
||||
if: ${{ fromJson(inputs.check_changes).sgl_kernel == 'true' && steps.rc.outputs.artifact_version == 'v6' }}
|
||||
@@ -155,7 +155,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- uses: ./.github/actions/download-rust-ext
|
||||
if: ${{ !inputs.skip_prebuilt_rust_ext }}
|
||||
@@ -216,7 +216,7 @@ jobs:
|
||||
CONTINUE_ON_ERROR_FLAG: ${{ fromJson(inputs.check_changes).continue_on_error == 'true' && '--continue-on-error' || '' }}
|
||||
RUNNER_LABELS: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)[inputs.runner_config] }}
|
||||
GPU_CONFIG: ${{ inputs.runner_config }}
|
||||
HF_MUSE_SPARK_TOKEN: ${{ inputs.self_name == 'base-c-test-8-gpu-b300' && secrets.HF_MUSE_SPARK_TOKEN || '' }}
|
||||
HF_MUSE_SPARK_TOKEN: ${{ (inputs.self_name == 'base-c-test-8-gpu-b300' || inputs.self_name == 'extra-b-test-8-gpu-b300') && secrets.HF_MUSE_SPARK_TOKEN || '' }}
|
||||
# Read only by test_runai_model_loader.py (nightly 1-gpu); inert elsewhere.
|
||||
# Left unconditional because an `env:` key cannot be omitted, and an empty
|
||||
# value would reach the runai streamer as a malformed limit.
|
||||
|
||||
@@ -58,11 +58,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install tomli
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Configure Git and branch
|
||||
run: |
|
||||
|
||||
@@ -24,11 +24,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install tomli
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Configure Git and branch
|
||||
run: |
|
||||
|
||||
@@ -22,11 +22,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install tomli
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Check if sync is needed
|
||||
id: check_sync
|
||||
|
||||
@@ -24,11 +24,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install tomli
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Configure Git and branch
|
||||
run: |
|
||||
|
||||
@@ -26,11 +26,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
|
||||
- name: Install Python dependencies
|
||||
run: |
|
||||
pip install tomli
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Configure Git and branch
|
||||
id: set_output
|
||||
|
||||
@@ -25,7 +25,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Generate Summary Report
|
||||
run: |
|
||||
@@ -41,7 +41,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Generate Tests by Folder Report
|
||||
run: |
|
||||
@@ -57,7 +57,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Generate Tests by Suite Report
|
||||
run: |
|
||||
@@ -73,7 +73,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Generate Tests by GPU Count Report
|
||||
run: |
|
||||
@@ -184,7 +184,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Generate JSON Report
|
||||
run: |
|
||||
|
||||
@@ -1,15 +1,17 @@
|
||||
# Coverage data collection pipeline for the NPU precision test selection.
|
||||
# Coverage baseline collection for the NPU precision test selection.
|
||||
# Delegates to pr-test-npu.yml with coverage_mode=true, which switches:
|
||||
# - Runners to the coverage pool (shared /root/.cache with setup-covstub)
|
||||
# - use_coverage_runner=true on all test jobs
|
||||
# - Skips pr-gate, multimodal-gen, recommend, analyze
|
||||
# - Runs setup-covstub to assemble coverage data and publish it to the
|
||||
# shared disk at a fixed path (/root/.cache/tests/precise-test/coverage)
|
||||
# pr-test-npu-finish is the final gate in the called workflow.
|
||||
#
|
||||
# Manually triggered only (workflow_dispatch): PRs run the standard
|
||||
# pr-test-npu.yml instead; this pipeline builds the full coverage baseline
|
||||
# on demand.
|
||||
# - Test jobs to the coverage pool with use_coverage_runner=true
|
||||
# - base-a to a hosted runner (base-a-test-npu-cov, data merged via artifact)
|
||||
# - Runs the multimodal-gen suites too. Their runner is asked for this
|
||||
# shard's targets (case-level pytest node ids) and each target is run in
|
||||
# its own coverage invocation, so diffusion coverage is attributed per
|
||||
# test case instead of per shard.
|
||||
# - Skips pr-gate, recommend
|
||||
# - setup-covstub assembles and publishes the baseline to the shared disk
|
||||
# at /root/.cache/tests/precise-test/coverage
|
||||
# pr-test-npu-finish is the final gate in the called workflow. Manually
|
||||
# triggered only (workflow_dispatch): PRs run the standard pr-test-npu.yml
|
||||
# instead; this pipeline builds the full coverage baseline on demand.
|
||||
name: Coverage Collection (NPU)
|
||||
|
||||
on:
|
||||
|
||||
@@ -24,7 +24,7 @@ on:
|
||||
default: ''
|
||||
type: string
|
||||
kernel_artifact_run_id:
|
||||
description: "Optional sgl-kernel wheel source: GitHub Actions run ID of a pr-test.yml run on the SAME commit as 'ref' that uploaded wheel-python3.10-cuda13.0. Use only when this branch's torch/ABI change makes the PyPI sglang-kernel wheel incompatible. Find it under Actions > pr-test; artifacts expire after 90 days."
|
||||
description: "Optional sgl-kernel wheel source: GitHub Actions run ID of a pr-test.yml run on the SAME commit as 'ref' that uploaded wheel-python3.11-cuda13.0. Use only when this branch's torch/ABI change makes the PyPI sglang-kernel wheel incompatible. Find it under Actions > pr-test; artifacts expire after 90 days."
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
@@ -204,7 +204,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
name: wheel-python3.10-cuda13.0
|
||||
name: wheel-python3.11-cuda13.0
|
||||
run-id: ${{ inputs.kernel_artifact_run_id }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -351,7 +351,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Verify write access to sgl-project/ci-data-diffusion
|
||||
if: ${{ inputs.publish }}
|
||||
@@ -403,7 +403,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
name: wheel-python3.10-cuda13.0
|
||||
name: wheel-python3.11-cuda13.0
|
||||
run-id: ${{ inputs.kernel_artifact_run_id }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -479,7 +479,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
name: wheel-python3.10-cuda13.0
|
||||
name: wheel-python3.11-cuda13.0
|
||||
run-id: ${{ inputs.kernel_artifact_run_id }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -557,7 +557,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
name: wheel-python3.10-cuda13.0
|
||||
name: wheel-python3.11-cuda13.0
|
||||
run-id: ${{ inputs.kernel_artifact_run_id }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -630,7 +630,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
name: wheel-python3.10-cuda13.0
|
||||
name: wheel-python3.11-cuda13.0
|
||||
run-id: ${{ inputs.kernel_artifact_run_id }}
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
name: Nightly Experimental sgl-router Docker Image
|
||||
name: Nightly sgl-router Docker Image
|
||||
|
||||
on:
|
||||
schedule:
|
||||
@@ -34,15 +34,18 @@ jobs:
|
||||
- suite: nightly-stage-c-2-gpu-mi350
|
||||
runner: linux-mi300-2gpu-sglang
|
||||
timeout: 180
|
||||
# Single-request decode of Qwen3.8-27B-FP8 runs at ~20 tok/s on MI300X, so long
|
||||
# generations outlast the test client's 180 s request timeout.
|
||||
skip_tests: tests/e2e/sglang/test_session_server_multi_role/test_qwen38small.py
|
||||
# test_qwen38small: single-request decode of Qwen3.8-27B-FP8 runs at ~20 tok/s on
|
||||
# MI300X, so long generations outlast the test client's 180 s request timeout.
|
||||
# test_glm5_2_744b_a40b_5layer_ci: its tilelang indexer kernel needs 98 KiB of LDS
|
||||
# per block; gfx942 has 64 KiB.
|
||||
skip_tests: >-
|
||||
tests/e2e/sglang/test_session_server_multi_role/test_qwen38small.py
|
||||
tests/e2e/megatron/model_scripts/test_glm5_2_744b_a40b_5layer_ci.py
|
||||
# The 4- and 8-GPU suites run past 180 min on MI300X; they take ~155 and ~112 min on MI355X.
|
||||
- suite: nightly-stage-c-4-gpu-mi350
|
||||
runner: linux-mi300-4gpu-sglang
|
||||
timeout: 300
|
||||
# Its tilelang indexer kernel needs 98 KiB of LDS per block; gfx942 has 64 KiB.
|
||||
skip_tests: tests/e2e/megatron/model_scripts/test_glm5_2_744b_a40b_5layer_ci.py
|
||||
skip_tests: ""
|
||||
- suite: nightly-stage-c-8-gpu-mi350
|
||||
runner: linux-mi300-8gpu-sglang
|
||||
timeout: 300
|
||||
|
||||
@@ -24,15 +24,28 @@ concurrency:
|
||||
jobs:
|
||||
test-miles-mi35x:
|
||||
if: github.repository == 'sgl-project/sglang'
|
||||
name: test-miles-mi35x (${{ matrix.suite }}, ${{ matrix.runner }}${{ matrix.partition_size && format(', part {0}', matrix.part) || '' }})
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# All suites use the 8-GPU MI35X pool. The 2-GPU suite includes the
|
||||
# GLM-5.2 5-layer test, which takes the linux-mi35x-gpu-2 runner down
|
||||
# ("runner lost communication") but passes on an 8-GPU runner.
|
||||
include:
|
||||
- suite: nightly-stage-c-2-gpu-mi350
|
||||
runner: linux-mi35x-gpu-2
|
||||
# Use 8-GPU MI35X pool for 4-GPU suite.
|
||||
runner: linux-mi35x-gpu-8
|
||||
# The 4-GPU suite is ~6 h of estimated test time; split it so each
|
||||
# half fits in the step timeout.
|
||||
- suite: nightly-stage-c-4-gpu-mi350
|
||||
runner: linux-mi35x-gpu-8
|
||||
partition_id: 0
|
||||
partition_size: 2
|
||||
part: 1/2
|
||||
- suite: nightly-stage-c-4-gpu-mi350
|
||||
runner: linux-mi35x-gpu-8
|
||||
partition_id: 1
|
||||
partition_size: 2
|
||||
part: 2/2
|
||||
- suite: nightly-stage-c-8-gpu-mi350
|
||||
runner: linux-mi35x-gpu-8
|
||||
runs-on: ${{ matrix.runner }}
|
||||
@@ -88,6 +101,7 @@ jobs:
|
||||
-e PYTHONPATH=/root/miles:/opt/tilelang \
|
||||
-e GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \
|
||||
python3 tests/ci/run_suite.py --hw rocm --suite ${{ matrix.suite }} --cadence nightly \
|
||||
${{ matrix.partition_size && format('--auto-partition-id {0} --auto-partition-size {1}', matrix.partition_id, matrix.partition_size) || '' }} \
|
||||
${{ (github.event_name != 'workflow_dispatch' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$?
|
||||
cat "${GITHUB_WORKSPACE}/github_summary.md" >> "$GITHUB_STEP_SUMMARY" || true
|
||||
exit ${TEST_EXIT_CODE}
|
||||
|
||||
@@ -1000,6 +1000,11 @@ jobs:
|
||||
# HF download env vars:
|
||||
# - HF_HUB_ENABLE_HF_TRANSFER=1: Use faster hf_transfer for downloads (if available)
|
||||
# - HF_HUB_DISABLE_SYMLINKS_WARNING=1: Suppress symlink warnings
|
||||
#
|
||||
# Skips ltx_2_5_diffusion_decoder_2gpus: on mi300 its server has never
|
||||
# finished warmup inside the 1200s startup deadline, and the harness
|
||||
# retries that TimeoutError six times, running the shard into the
|
||||
# step timeout.
|
||||
docker exec \
|
||||
-e HF_TOKEN \
|
||||
-e HUGGING_FACE_HUB_TOKEN \
|
||||
@@ -1019,6 +1024,7 @@ jobs:
|
||||
--suite 2-gpu \
|
||||
--partition-id ${{ matrix.part }} \
|
||||
--total-partitions 2 \
|
||||
-k "not ltx_2_5_diffusion_decoder" \
|
||||
${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
|
||||
|
||||
# Post-test diagnostics
|
||||
@@ -1456,9 +1462,13 @@ jobs:
|
||||
run: |
|
||||
docker exec ci_sglang rocm-smi --showtopotype || true
|
||||
docker exec ci_sglang bash -c 'strings $(python3 -c "import mooncake,os;print(os.path.dirname(mooncake.__file__))")/engine*.so | grep -iE "rdma,hip|Unsupported multi-protocol"' || true
|
||||
# With Mooncake v0.3.14-rc1, registering the KV cache through HIP dmabuf
|
||||
# leaves the server's next HIP stream creation failing (out of memory on
|
||||
# rocm10, segfault on rocm720). Register through ibv_reg_mr instead.
|
||||
bash scripts/ci/amd/amd_ci_exec.sh \
|
||||
-e SGLANG_TEST_RDMA_DEVICE="${{ env.SGLANG_TEST_RDMA_DEVICE }}" \
|
||||
-e MC_LOG_LEVEL=TRACE \
|
||||
-e MOONCAKE_DISABLE_HIP_DMABUF=1 \
|
||||
-e GLOG_logtostderr=1 \
|
||||
-w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-large-8-gpu-mi35x-disaggregation-amd --timeout-per-file 1800 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
|
||||
|
||||
|
||||
@@ -150,6 +150,15 @@ jobs:
|
||||
rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }}
|
||||
# No `secrets: inherit`: this hosted CPU job has no secret consumer.
|
||||
|
||||
simulator-test-gpu:
|
||||
needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build]
|
||||
if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }}
|
||||
uses: ./.github/workflows/_pr-test-simulator-gpu.yml
|
||||
with:
|
||||
check_changes: ${{ toJson(needs.check-changes.outputs) }}
|
||||
caller_inputs: ${{ toJson(inputs) }}
|
||||
rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }}
|
||||
|
||||
# =============================================== extra-a / extra-b ===============================================
|
||||
# `name:` keeps the job names wait-for-jobs and check-pr-test-health match on.
|
||||
extra-test:
|
||||
@@ -191,6 +200,7 @@ jobs:
|
||||
sgl-kernel-build-wheels,
|
||||
rust-ext-build,
|
||||
simulator-test-cpu,
|
||||
simulator-test-gpu,
|
||||
extra-test,
|
||||
]
|
||||
if: always()
|
||||
|
||||
@@ -99,7 +99,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda13.0
|
||||
pattern: wheel-python3.11-cuda13.0
|
||||
|
||||
- uses: ./.github/actions/download-rust-ext
|
||||
with:
|
||||
|
||||
@@ -65,7 +65,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Compute partitions
|
||||
id: compute
|
||||
@@ -101,7 +101,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
@@ -168,13 +168,19 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
run: |
|
||||
CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
|
||||
|
||||
- name: Run SM120-only diffusion unit tests
|
||||
timeout-minutes: 10
|
||||
working-directory: python
|
||||
run: |
|
||||
python3 -m pytest -v sglang/multimodal_gen/test/unit/test_fp8_fa_sm120_attn.py
|
||||
|
||||
- name: Run 5090 diffusion canary tests
|
||||
timeout-minutes: 120
|
||||
env:
|
||||
@@ -233,7 +239,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
@@ -298,7 +304,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
@@ -368,7 +374,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
@@ -413,7 +419,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
@@ -427,6 +433,22 @@ jobs:
|
||||
run: |
|
||||
python3 -m pytest -v python/sglang/multimodal_gen/test/single_test_file/test_wan_vae_parallel_groups_4_gpu.py
|
||||
|
||||
- name: Run Blackwell-only diffusion unit tests
|
||||
timeout-minutes: 10
|
||||
working-directory: python
|
||||
run: |
|
||||
python3 -m pytest -v \
|
||||
sglang/multimodal_gen/test/unit/test_mxfp8_online_gemm.py \
|
||||
sglang/multimodal_gen/test/unit/test_wan_gelu_mlp.py
|
||||
|
||||
- name: Run 4-GPU diffusion unit tests
|
||||
timeout-minutes: 15
|
||||
working-directory: python
|
||||
run: |
|
||||
python3 -m pytest -v \
|
||||
sglang/multimodal_gen/test/unit/test_kandinsky6_sequence_parallel.py::test_joint_video_audio_tensor_sequence_parallel \
|
||||
sglang/multimodal_gen/test/unit/test_kandinsky6_sr_sequence_parallel.py::test_sr_four_gpus
|
||||
|
||||
- name: Run diffusion server tests
|
||||
timeout-minutes: 240
|
||||
env:
|
||||
@@ -483,13 +505,20 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 20
|
||||
run: |
|
||||
CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
|
||||
|
||||
- name: Install Quant-VideoGen
|
||||
timeout-minutes: 5
|
||||
run: |
|
||||
# quant-videogen pins an incompatible torch, so install it without deps
|
||||
# plus the import-time deps of the diffusion-qvg extra.
|
||||
uv pip install --no-deps quant-videogen==0.1.0 "loguru>=0.7" "termcolor>=2.3"
|
||||
|
||||
- name: Run diffusion unit tests
|
||||
timeout-minutes: 60
|
||||
run: |
|
||||
@@ -519,7 +548,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Download all execution reports
|
||||
uses: actions/download-artifact@v4
|
||||
|
||||
+596
-108
@@ -1,11 +1,13 @@
|
||||
name: PR Test (NPU)
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ main ]
|
||||
schedule:
|
||||
- cron: '0 12 * * *' # Run daily at 12:00 UTC
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
- 'npu/**'
|
||||
- 'npu-*'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
coverage_mode:
|
||||
@@ -169,9 +171,84 @@ jobs:
|
||||
cd test
|
||||
python3 run_suite.py --hw npu --suite base-a-test-npu
|
||||
|
||||
# Coverage-mode-only counterpart of the base-a PR gate above. Stays on
|
||||
# ubuntu-latest like the gate (its UTs mock the NPU stack and need a CPU
|
||||
# triton; the NPU image's triton-ascend breaks those mocks), so the shared
|
||||
# template (unconditional CANN container:) cannot be used and coverage
|
||||
# data reaches setup-covstub via artifact instead of the shared disk.
|
||||
base-a-test-npu-cov:
|
||||
needs: [set-image-config]
|
||||
if: ${{ inputs.coverage_mode == true }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
env:
|
||||
SGLANG_IS_IN_CI: true
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# Pin to the event SHA so this job tests the same commit as the rest of the run.
|
||||
ref: ${{ inputs.ref || github.sha }}
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: 'python/pyproject_npu.toml'
|
||||
|
||||
- name: Install dependencies
|
||||
timeout-minutes: 30
|
||||
env:
|
||||
# uv pip targets a venv by default; setup-python has no venv.
|
||||
UV_SYSTEM_PYTHON: "1"
|
||||
run: |
|
||||
# Same dependency set as the base-a PR gate above (CPU torch + CPU
|
||||
# triton, no Ascend wheels); keep the two blocks in sync.
|
||||
rm -f python/pyproject.toml
|
||||
mv python/pyproject_npu.toml python/pyproject.toml
|
||||
uv pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 triton \
|
||||
--index-url https://download.pytorch.org/whl/cpu \
|
||||
--extra-index-url https://pypi.org/simple/ \
|
||||
--index-strategy unsafe-best-match
|
||||
uv pip install -e "python[dev_npu]" --index-strategy unsafe-best-match
|
||||
pip install coverage==7.8.*
|
||||
|
||||
- name: Run test with coverage
|
||||
timeout-minutes: 15
|
||||
run: |
|
||||
# Hosted runners cannot write the shared disk, so stage locally;
|
||||
# run_tests_with_coverage.sh derives RUN_DIR/date tag from run-level
|
||||
# env vars, so the artifact merges 1:1 into the shared-disk layout.
|
||||
export PRECISE_TEST_CACHE_ROOT=/tmp/precise-test
|
||||
cd test
|
||||
python3 ../scripts/ci/npu/precise-test/list_tests.py --hw npu --suite base-a-test-npu \
|
||||
-o /tmp/selected_tests.txt
|
||||
echo "Selected test files:"
|
||||
cat /tmp/selected_tests.txt
|
||||
if [ -s /tmp/selected_tests.txt ]; then
|
||||
bash ../scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt)
|
||||
fi
|
||||
|
||||
- name: Upload coverage outputs
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: coverage-outputs-base-a-test-npu
|
||||
path: /tmp/precise-test/
|
||||
retention-days: 1
|
||||
if-no-files-found: warn
|
||||
|
||||
base-b-test-1-npu-a3:
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_1_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-pr-test-stage.yml
|
||||
with:
|
||||
self_name: base-b-test-1-npu-a3
|
||||
@@ -179,13 +256,16 @@ jobs:
|
||||
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
|
||||
run_timeout_minutes: '60'
|
||||
timeout_per_file: '3600'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_1_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
secrets: inherit
|
||||
|
||||
base-b-test-2-npu-a3:
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_2_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-pr-test-stage.yml
|
||||
with:
|
||||
self_name: base-b-test-2-npu-a3
|
||||
@@ -193,13 +273,16 @@ jobs:
|
||||
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
|
||||
run_timeout_minutes: '60'
|
||||
timeout_per_file: '3600'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_2_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
secrets: inherit
|
||||
|
||||
base-b-test-4-npu-a3:
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_4_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-pr-test-stage.yml
|
||||
with:
|
||||
self_name: base-b-test-4-npu-a3
|
||||
@@ -208,13 +291,16 @@ jobs:
|
||||
run_timeout_minutes: '120'
|
||||
timeout_per_file: '3600'
|
||||
partitions: '{"size":2,"arr":[0, 1]}'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_4_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
secrets: inherit
|
||||
|
||||
base-b-test-8-npu-a3:
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_8_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-pr-test-stage.yml
|
||||
with:
|
||||
self_name: base-b-test-8-npu-a3
|
||||
@@ -222,13 +308,16 @@ jobs:
|
||||
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
|
||||
run_timeout_minutes: '60'
|
||||
timeout_per_file: '3600'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_8_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
secrets: inherit
|
||||
|
||||
base-b-test-16-npu-a3:
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_16_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-pr-test-stage.yml
|
||||
with:
|
||||
self_name: base-b-test-16-npu-a3
|
||||
@@ -236,13 +325,24 @@ jobs:
|
||||
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
|
||||
run_timeout_minutes: '120'
|
||||
timeout_per_file: '3600'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_16_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
secrets: inherit
|
||||
|
||||
multimodal-gen-test-1-npu-a3:
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
|
||||
if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }}
|
||||
needs:
|
||||
[
|
||||
check-changes,
|
||||
pr-gate,
|
||||
set-image-config,
|
||||
base-a-test-npu,
|
||||
base-a-test-npu-cov,
|
||||
recommend-tests-from-coverage,
|
||||
]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.multimodal_gen == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_multimodal_gen_test_1_npu_a3 != 'false' }}
|
||||
runs-on: linux-aarch64-a3-800t-2
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -290,7 +390,8 @@ jobs:
|
||||
run: python3 scripts/ci/utils/diffusion/prefetch_input_image.py
|
||||
|
||||
- name: Run test
|
||||
timeout-minutes: 60
|
||||
timeout-minutes: ${{ inputs.coverage_mode == true && 150 || 60 }}
|
||||
shell: bash
|
||||
env:
|
||||
SGLANG_USE_MODELSCOPE: true
|
||||
SGLANG_IS_IN_CI: true
|
||||
@@ -299,12 +400,75 @@ jobs:
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
STREAMS_PER_DEVICE: 32
|
||||
SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures
|
||||
ONLY_TESTS: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_multimodal_gen_test_1_npu_a3 }}
|
||||
run: |
|
||||
# Coverage mode: enumerate this shard's targets with an NPU-only CI
|
||||
# helper (it reuses the shared diffusion runner read-only) and run
|
||||
# each target in its own `coverage run -m pytest` invocation. The
|
||||
# runner normally batches a whole shard into one pytest call, which
|
||||
# cannot be attributed per test case by the precise-test selector.
|
||||
if [[ "${{ inputs.coverage_mode }}" == "true" ]]; then
|
||||
set -euo pipefail
|
||||
pip install coverage==7.8.*
|
||||
# Coverage tracing slows down the Python glue, which shows up in
|
||||
# wall-clock assertions. Relax only the timing tolerances for the
|
||||
# collection run; PR runs (no coverage) keep the strict profile and
|
||||
# the memory/VRAM checks are untouched.
|
||||
export SGLANG_E2E_TOLERANCE=1.0
|
||||
export SGLANG_STAGE_TIME_TOLERANCE=1.0
|
||||
export SGLANG_NON_DENOISE_STAGE_TIME_TOLERANCE=2.0
|
||||
export SGLANG_DENOISE_STEP_TOLERANCE=1.0
|
||||
export SGLANG_DENOISE_AGG_TOLERANCE=1.0
|
||||
python3 scripts/ci/npu/precise-test/list_multimodal_tests.py \
|
||||
--suite 1-npu \
|
||||
--partition-id ${{ matrix.part }} \
|
||||
--total-partitions 2 \
|
||||
-o /tmp/selected_tests.txt
|
||||
echo "Selected test targets:"
|
||||
cat /tmp/selected_tests.txt
|
||||
if [ -s /tmp/selected_tests.txt ]; then
|
||||
bash scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt)
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Precision mode: ONLY_TESTS holds one pytest -k token per line
|
||||
# ('[case_id]' for parametrized cases, a file stem for standalone
|
||||
# files). It is empty when nothing was recommended, precision mode
|
||||
# is off, or a safety gate forced the full suite; then no filter is
|
||||
# applied and the shard runs its whole assignment.
|
||||
FILTER=""
|
||||
while IFS= read -r line; do
|
||||
if [ -z "${line}" ]; then
|
||||
continue
|
||||
fi
|
||||
# Every line must be a -k token ('[case_id]' or a standalone file
|
||||
# stem). A path means the bucket contract drifted: with a path the
|
||||
# -k expression matches nothing, pytest exits 5, and the shard
|
||||
# would report success after running zero tests.
|
||||
if [[ "${line}" == */* || "${line}" == *.py ]]; then
|
||||
echo "::error::Unexpected only_tests entry '${line}' (expected a pytest -k token)"
|
||||
exit 1
|
||||
fi
|
||||
if [ -n "${FILTER}" ]; then
|
||||
FILTER="${FILTER} or "
|
||||
fi
|
||||
FILTER="${FILTER}${line}"
|
||||
done <<< "${ONLY_TESTS}"
|
||||
FILTER_ARGS=()
|
||||
if [ -n "${FILTER}" ]; then
|
||||
FILTER_ARGS+=(-k "${FILTER}")
|
||||
echo "Precision mode: filtered to ${FILTER}"
|
||||
else
|
||||
echo "Precision mode: no filter (full suite)"
|
||||
fi
|
||||
|
||||
cd python
|
||||
python3 sglang/multimodal_gen/test/run_suite.py \
|
||||
--suite 1-npu \
|
||||
--partition-id ${{ matrix.part }} \
|
||||
--total-partitions 2
|
||||
--total-partitions 2 \
|
||||
"${FILTER_ARGS[@]}"
|
||||
|
||||
- name: Upload diffusion failure artifacts
|
||||
if: always()
|
||||
@@ -316,8 +480,19 @@ jobs:
|
||||
retention-days: 7
|
||||
|
||||
multimodal-gen-test-4-npu-a3:
|
||||
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
|
||||
if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }}
|
||||
needs:
|
||||
[
|
||||
check-changes,
|
||||
pr-gate,
|
||||
set-image-config,
|
||||
base-a-test-npu,
|
||||
base-a-test-npu-cov,
|
||||
recommend-tests-from-coverage,
|
||||
]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.multimodal_gen == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_multimodal_gen_test_4_npu_a3 != 'false' }}
|
||||
runs-on: linux-aarch64-a3-800t-4
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -360,7 +535,8 @@ jobs:
|
||||
python3 scripts/ci/utils/diffusion/prefetch_consistency_gt.py
|
||||
|
||||
- name: Run test
|
||||
timeout-minutes: 60
|
||||
timeout-minutes: ${{ inputs.coverage_mode == true && 150 || 60 }}
|
||||
shell: bash
|
||||
env:
|
||||
SGLANG_USE_MODELSCOPE: true
|
||||
SGLANG_IS_IN_CI: true
|
||||
@@ -369,12 +545,75 @@ jobs:
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
STREAMS_PER_DEVICE: 32
|
||||
SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures
|
||||
ONLY_TESTS: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_multimodal_gen_test_4_npu_a3 }}
|
||||
run: |
|
||||
# Coverage mode: enumerate this shard's targets with an NPU-only CI
|
||||
# helper (it reuses the shared diffusion runner read-only) and run
|
||||
# each target in its own `coverage run -m pytest` invocation. The
|
||||
# runner normally batches a whole shard into one pytest call, which
|
||||
# cannot be attributed per test case by the precise-test selector.
|
||||
if [[ "${{ inputs.coverage_mode }}" == "true" ]]; then
|
||||
set -euo pipefail
|
||||
pip install coverage==7.8.*
|
||||
# Coverage tracing slows down the Python glue, which shows up in
|
||||
# wall-clock assertions. Relax only the timing tolerances for the
|
||||
# collection run; PR runs (no coverage) keep the strict profile and
|
||||
# the memory/VRAM checks are untouched.
|
||||
export SGLANG_E2E_TOLERANCE=1.0
|
||||
export SGLANG_STAGE_TIME_TOLERANCE=1.0
|
||||
export SGLANG_NON_DENOISE_STAGE_TIME_TOLERANCE=2.0
|
||||
export SGLANG_DENOISE_STEP_TOLERANCE=1.0
|
||||
export SGLANG_DENOISE_AGG_TOLERANCE=1.0
|
||||
python3 scripts/ci/npu/precise-test/list_multimodal_tests.py \
|
||||
--suite 2-npu \
|
||||
--partition-id ${{ matrix.part }} \
|
||||
--total-partitions 2 \
|
||||
-o /tmp/selected_tests.txt
|
||||
echo "Selected test targets:"
|
||||
cat /tmp/selected_tests.txt
|
||||
if [ -s /tmp/selected_tests.txt ]; then
|
||||
bash scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt)
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Precision mode: ONLY_TESTS holds one pytest -k token per line
|
||||
# ('[case_id]' for parametrized cases, a file stem for standalone
|
||||
# files). It is empty when nothing was recommended, precision mode
|
||||
# is off, or a safety gate forced the full suite; then no filter is
|
||||
# applied and the shard runs its whole assignment.
|
||||
FILTER=""
|
||||
while IFS= read -r line; do
|
||||
if [ -z "${line}" ]; then
|
||||
continue
|
||||
fi
|
||||
# Every line must be a -k token ('[case_id]' or a standalone file
|
||||
# stem). A path means the bucket contract drifted: with a path the
|
||||
# -k expression matches nothing, pytest exits 5, and the shard
|
||||
# would report success after running zero tests.
|
||||
if [[ "${line}" == */* || "${line}" == *.py ]]; then
|
||||
echo "::error::Unexpected only_tests entry '${line}' (expected a pytest -k token)"
|
||||
exit 1
|
||||
fi
|
||||
if [ -n "${FILTER}" ]; then
|
||||
FILTER="${FILTER} or "
|
||||
fi
|
||||
FILTER="${FILTER}${line}"
|
||||
done <<< "${ONLY_TESTS}"
|
||||
FILTER_ARGS=()
|
||||
if [ -n "${FILTER}" ]; then
|
||||
FILTER_ARGS+=(-k "${FILTER}")
|
||||
echo "Precision mode: filtered to ${FILTER}"
|
||||
else
|
||||
echo "Precision mode: no filter (full suite)"
|
||||
fi
|
||||
|
||||
cd python
|
||||
python3 sglang/multimodal_gen/test/run_suite.py \
|
||||
--suite 2-npu \
|
||||
--partition-id ${{ matrix.part }} \
|
||||
--total-partitions 2
|
||||
--total-partitions 2 \
|
||||
"${FILTER_ARGS[@]}"
|
||||
|
||||
- name: Upload diffusion failure artifacts
|
||||
if: always()
|
||||
@@ -387,8 +626,11 @@ jobs:
|
||||
|
||||
base-c-test-acc-2-npu-a3:
|
||||
name: base-c-test-acc-2-npu-a3
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, recommend-tests-from-coverage ]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_acc_2_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-single-node-test-stage.yml
|
||||
with:
|
||||
runner: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-2' || 'linux-aarch64-a3-2-' }}
|
||||
@@ -398,13 +640,16 @@ jobs:
|
||||
install_sglang_deps: true
|
||||
device_type_for_deps: 'a3'
|
||||
partitions: '{"size":2,"arr":[0,1]}'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_acc_2_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
|
||||
base-c-test-acc-16-npu-a3:
|
||||
name: base-c-test-acc-16-npu-a3
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, recommend-tests-from-coverage ]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_acc_16_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-single-node-test-stage.yml
|
||||
with:
|
||||
runner: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-16' || 'linux-aarch64-a3-16-' }}
|
||||
@@ -413,13 +658,16 @@ jobs:
|
||||
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
|
||||
install_sglang_deps: true
|
||||
device_type_for_deps: 'a3'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_acc_16_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
|
||||
base-c-test-perf-2-npu-a3:
|
||||
name: base-c-test-perf-2-npu-a3
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3 ]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3, recommend-tests-from-coverage ]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_perf_2_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-single-node-test-stage.yml
|
||||
with:
|
||||
runner: linux-aarch64-a3-800t-2
|
||||
@@ -428,13 +676,16 @@ jobs:
|
||||
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
|
||||
install_sglang_deps: true
|
||||
device_type_for_deps: 'a3'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_perf_2_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
|
||||
base-c-test-perf-16-npu-a3:
|
||||
name: base-c-test-perf-16-npu-a3
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3 ]
|
||||
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
|
||||
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3, recommend-tests-from-coverage ]
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
|
||||
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_perf_16_npu_a3 != 'false' }}
|
||||
uses: ./.github/workflows/_npu-single-node-test-stage.yml
|
||||
with:
|
||||
runner: linux-aarch64-a3-800t-16
|
||||
@@ -444,8 +695,8 @@ jobs:
|
||||
install_sglang_deps: true
|
||||
device_type_for_deps: 'a3'
|
||||
test_timeout_minutes: '180'
|
||||
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_perf_16_npu_a3 }}
|
||||
use_coverage_runner: ${{ inputs.coverage_mode == true }}
|
||||
upload_test_logs: ${{ inputs.coverage_mode != true }}
|
||||
|
||||
|
||||
# ==================== Recommend Tests from Coverage ==================== #
|
||||
@@ -453,25 +704,63 @@ jobs:
|
||||
# continue-on-error so it never blocks the PR gate.
|
||||
recommend-tests-from-coverage:
|
||||
name: Recommend tests from coverage
|
||||
needs: [check-changes, pr-gate]
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
# TEMP-DISABLED: OBS coverage baseline returns 403; re-enable once fixed.
|
||||
if: false
|
||||
# if: ${{ !cancelled() && github.event_name == 'pull_request' && inputs.coverage_mode != true }}
|
||||
if: >-
|
||||
${{ !failure() && !cancelled() &&
|
||||
github.event_name == 'pull_request' &&
|
||||
inputs.coverage_mode != true &&
|
||||
needs.check-changes.outputs.changes_exist == 'true' }}
|
||||
outputs:
|
||||
coverage_paths: ${{ steps.export-paths.outputs.coverage_paths }}
|
||||
should_run_base_b_test_1_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_1_npu_a3 }}
|
||||
only_tests_base_b_test_1_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_1_npu_a3 }}
|
||||
should_run_base_b_test_2_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_2_npu_a3 }}
|
||||
only_tests_base_b_test_2_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_2_npu_a3 }}
|
||||
should_run_base_b_test_4_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_4_npu_a3 }}
|
||||
only_tests_base_b_test_4_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_4_npu_a3 }}
|
||||
should_run_base_b_test_8_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_8_npu_a3 }}
|
||||
only_tests_base_b_test_8_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_8_npu_a3 }}
|
||||
should_run_base_b_test_16_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_16_npu_a3 }}
|
||||
only_tests_base_b_test_16_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_16_npu_a3 }}
|
||||
should_run_base_c_test_acc_2_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_acc_2_npu_a3 }}
|
||||
only_tests_base_c_test_acc_2_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_acc_2_npu_a3 }}
|
||||
should_run_base_c_test_acc_16_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_acc_16_npu_a3 }}
|
||||
only_tests_base_c_test_acc_16_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_acc_16_npu_a3 }}
|
||||
should_run_base_c_test_perf_2_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_perf_2_npu_a3 }}
|
||||
only_tests_base_c_test_perf_2_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_perf_2_npu_a3 }}
|
||||
should_run_base_c_test_perf_16_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_perf_16_npu_a3 }}
|
||||
only_tests_base_c_test_perf_16_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_perf_16_npu_a3 }}
|
||||
should_run_multimodal_gen_test_1_npu_a3: ${{ steps.bucket-tests.outputs.should_run_multimodal_gen_test_1_npu_a3 }}
|
||||
only_tests_multimodal_gen_test_1_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_multimodal_gen_test_1_npu_a3 }}
|
||||
should_run_multimodal_gen_test_4_npu_a3: ${{ steps.bucket-tests.outputs.should_run_multimodal_gen_test_4_npu_a3 }}
|
||||
only_tests_multimodal_gen_test_4_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_multimodal_gen_test_4_npu_a3 }}
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
steps:
|
||||
# Every step carries step-level continue-on-error so the job conclusion
|
||||
# is ALWAYS 'success' — the fast-fail health check in other jobs filters
|
||||
# by job conclusion, and job-level continue-on-error still reports
|
||||
# 'failure' via the API. Failed steps stay visible (red ✗ + annotation)
|
||||
# but never cascade to test jobs.
|
||||
- name: Checkout sglang scripts
|
||||
continue-on-error: true
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# test/registered + python/sglang/test/ci are needed by
|
||||
# list_tests.py to enumerate each suite's test files for bucketing.
|
||||
# The diffusion case config is read (AST only) by
|
||||
# bucket_recommended_tests.py to enumerate multimodal cases.
|
||||
sparse-checkout: |
|
||||
scripts
|
||||
test/registered
|
||||
python/sglang/test/ci
|
||||
python/sglang/multimodal_gen/test/server/ascend/testcase_configs_npu.py
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
- name: Download test case map
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euo pipefail
|
||||
curl -fsSL \
|
||||
@@ -485,6 +774,7 @@ jobs:
|
||||
du -h test_case_map.json
|
||||
|
||||
- name: Download coverage package and extract covstub
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euo pipefail
|
||||
curl -fsSL \
|
||||
@@ -502,6 +792,7 @@ jobs:
|
||||
find outputs/covstub -maxdepth 2 -type d | head -20 || true
|
||||
|
||||
- name: Recommend tests for current PR
|
||||
continue-on-error: true
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if ! python3 -c "import regex" >/dev/null 2>&1; then
|
||||
@@ -520,52 +811,42 @@ jobs:
|
||||
--source-dir "$PWD/outputs/covstub" \
|
||||
--map-file "$PWD/test_case_map.json"
|
||||
|
||||
- name: Export coverage paths output
|
||||
id: export-paths
|
||||
# Bucket the recommended tests per suite. Downstream jobs consume two
|
||||
# outputs per suite: should_run_<suite> ('false' -> skip the job, e.g.
|
||||
# empty bucket in precision mode) and only_tests_<suite> (non-empty only
|
||||
# in precision mode, gated inside the script). On failure outputs stay
|
||||
# empty -> callers run full suites.
|
||||
- name: Bucket recommended tests per suite
|
||||
id: bucket-tests
|
||||
continue-on-error: true
|
||||
if: always()
|
||||
env:
|
||||
# Precision-test switch: 'true' = the a3 suites run only the
|
||||
# recommended subset, 'false' = full suites.
|
||||
PRECISION_MODE: 'true'
|
||||
run: |
|
||||
FILE="scripts/ci/npu/precise-test/recommended_pytest_paths.txt"
|
||||
if [ -f "$FILE" ]; then
|
||||
{
|
||||
echo "coverage_paths<<EOF"
|
||||
cat "$FILE"
|
||||
echo "EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
echo "Exported $(wc -l < "$FILE") recommended paths"
|
||||
else
|
||||
echo "coverage_paths=" >> "$GITHUB_OUTPUT"
|
||||
echo "::notice::recommended_pytest_paths.txt not found"
|
||||
fi
|
||||
set -euo pipefail
|
||||
python3 scripts/ci/npu/precise-test/bucket_recommended_tests.py \
|
||||
--recommended scripts/ci/npu/precise-test/recommended_pytest_paths.txt \
|
||||
--changed-files scripts/ci/npu/precise-test/pr_changed_files.txt \
|
||||
--suites "base-b-test-1-npu-a3,base-b-test-2-npu-a3,base-b-test-4-npu-a3,base-b-test-8-npu-a3,base-b-test-16-npu-a3,base-c-test-acc-2-npu-a3,base-c-test-acc-16-npu-a3,base-c-test-perf-2-npu-a3,base-c-test-perf-16-npu-a3" \
|
||||
--diffusion-suites "multimodal-gen-test-1-npu-a3:1-npu,multimodal-gen-test-4-npu-a3:2-npu" \
|
||||
--case-map "$PWD/test_case_map.json" \
|
||||
--precision-mode "${PRECISION_MODE}" \
|
||||
--github-output "${GITHUB_OUTPUT}" \
|
||||
--github-summary "${GITHUB_STEP_SUMMARY:-}" \
|
||||
--report-file recommended-tests-report.md
|
||||
|
||||
# ==================== Analyze Failure Report ==================== #
|
||||
# Cross-references test failures with coverage recommendations.
|
||||
# !cancelled() only: skipped jobs in needs poison success().
|
||||
analyze-failure-report:
|
||||
name: Analyze failure report
|
||||
needs:
|
||||
[
|
||||
base-a-test-npu,
|
||||
base-b-test-1-npu-a3,
|
||||
base-b-test-2-npu-a3,
|
||||
base-b-test-4-npu-a3,
|
||||
base-b-test-8-npu-a3,
|
||||
base-b-test-16-npu-a3,
|
||||
|
||||
multimodal-gen-test-1-npu-a3,
|
||||
multimodal-gen-test-4-npu-a3,
|
||||
|
||||
base-c-test-acc-2-npu-a3,
|
||||
base-c-test-acc-16-npu-a3,
|
||||
base-c-test-perf-2-npu-a3,
|
||||
base-c-test-perf-16-npu-a3,
|
||||
|
||||
recommend-tests-from-coverage,
|
||||
]
|
||||
if: ${{ !cancelled() && inputs.coverage_mode != true }}
|
||||
uses: ./.github/workflows/_npu-analyze-failure.yml
|
||||
with:
|
||||
log_artifact_pattern: selected-test-logs-*
|
||||
recommendations_content: ${{ needs.recommend-tests-from-coverage.outputs.coverage_paths || '' }}
|
||||
# Same report as the step summary, downloadable from the run page.
|
||||
- name: Upload recommended tests report
|
||||
continue-on-error: true
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: recommended-tests-report
|
||||
path: recommended-tests-report.md
|
||||
if-no-files-found: warn
|
||||
retention-days: 7
|
||||
|
||||
# ==================== Coverage Assembly & Publish ==================== #
|
||||
# Assembles coverage data and publishes the baseline to the shared disk.
|
||||
@@ -573,7 +854,7 @@ jobs:
|
||||
setup-covstub:
|
||||
needs:
|
||||
[
|
||||
set-image-config,
|
||||
base-a-test-npu-cov,
|
||||
base-b-test-1-npu-a3,
|
||||
base-b-test-2-npu-a3,
|
||||
base-b-test-4-npu-a3,
|
||||
@@ -583,11 +864,13 @@ jobs:
|
||||
base-c-test-acc-16-npu-a3,
|
||||
base-c-test-perf-2-npu-a3,
|
||||
base-c-test-perf-16-npu-a3,
|
||||
multimodal-gen-test-1-npu-a3,
|
||||
multimodal-gen-test-4-npu-a3,
|
||||
]
|
||||
if: ${{ !failure() && !cancelled() && inputs.coverage_mode == true }}
|
||||
runs-on: linux-aarch64-a3-800t-2
|
||||
runs-on: linux-amd64-cpu-4
|
||||
container:
|
||||
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
|
||||
image: swr.ap-southeast-1.myhuaweicloud.com/base_image/ascend-ci/sglang:main-x86
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
@@ -596,12 +879,31 @@ jobs:
|
||||
# checks out the same commit even if the branch advances mid-run.
|
||||
ref: ${{ inputs.ref || github.sha }}
|
||||
|
||||
- name: Collect base-a coverage artifact
|
||||
# Runs on ubuntu-latest (no shared disk) — hence the artifact relay.
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: coverage-outputs-base-a-test-npu
|
||||
path: /tmp/base-a-coverage
|
||||
|
||||
- name: Merge base-a coverage into shared run dir
|
||||
run: |
|
||||
# Artifact stages {RUN_DIR}/outputs/... — merges 1:1 into the shared layout.
|
||||
CACHE_ROOT="/root/.cache/tests/precise-test"
|
||||
mkdir -p "${CACHE_ROOT}"
|
||||
cp -a /tmp/base-a-coverage/. "${CACHE_ROOT}/"
|
||||
echo "Merged base-a coverage into ${CACHE_ROOT}:"
|
||||
find "${CACHE_ROOT}" -maxdepth 3 -type d | head -20
|
||||
|
||||
- name: Copy sglang source to covstub
|
||||
run: |
|
||||
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
|
||||
COVSTUB="${RUN_DIR}/outputs/covstub"
|
||||
mkdir -p "${COVSTUB}"
|
||||
cp -r python/sglang "${COVSTUB}/"
|
||||
# _version.py is build-generated (git-ignored); the placeholder keeps
|
||||
# coverage html from failing on a missing recorded source file.
|
||||
echo '__version__ = "0.0.0"' > "${COVSTUB}/sglang/_version.py"
|
||||
|
||||
- name: Build test case map
|
||||
shell: bash
|
||||
@@ -609,11 +911,9 @@ jobs:
|
||||
set -euo pipefail
|
||||
|
||||
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
|
||||
# Date tag derivation must match scripts/ci/npu/precise-test/run_tests_with_coverage.sh.
|
||||
# The custom k8s runner only injects explicitly-set env vars, so
|
||||
# GITHUB_RUN_STARTED_AT may be absent (set -u would abort); fall back
|
||||
# to local date exactly like run_tests_with_coverage.sh does, so both
|
||||
# sides derive the same tag.
|
||||
# Must match run_tests_with_coverage.sh: GITHUB_RUN_STARTED_AT may be
|
||||
# unset on the k8s runner (only explicitly-set envs are injected), so
|
||||
# fall back to local date the same way.
|
||||
if [ -n "${GITHUB_RUN_STARTED_AT:-}" ]; then
|
||||
COV_DATE_TAG="${GITHUB_RUN_STARTED_AT:0:10}"
|
||||
COV_DATE_TAG="${COV_DATE_TAG//-/}"
|
||||
@@ -622,14 +922,14 @@ jobs:
|
||||
fi
|
||||
COV_ROOT="${RUN_DIR}/outputs/sglang@${COV_DATE_TAG}"
|
||||
COVSTUB="${RUN_DIR}/outputs/covstub"
|
||||
# Placed outside outputs/ so it is NOT included in outputs.tar.gz.
|
||||
# Standalone top-level deliverable, never packed inside outputs/.
|
||||
MAP_FILE="${RUN_DIR}/test_case_map.json"
|
||||
|
||||
echo "Coverage dir: ${COV_ROOT}"
|
||||
echo "Source dir: ${COVSTUB}"
|
||||
|
||||
# Coverage data may be absent when all test jobs were skipped
|
||||
# (e.g. by the changes filter). Skip map building in that case.
|
||||
# Coverage data may be absent when no test targets were enumerated
|
||||
# (run_tests_with_coverage.sh never ran). Skip map building then.
|
||||
if [ ! -d "${COV_ROOT}" ]; then
|
||||
echo "::warning::Coverage dir not found: ${COV_ROOT}, skip building test case map"
|
||||
exit 0
|
||||
@@ -643,11 +943,11 @@ jobs:
|
||||
fi
|
||||
|
||||
if ! python3 -c "import regex" 2>/dev/null; then
|
||||
pip install regex \
|
||||
python3 -m pip install regex \
|
||||
-i http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
|
||||
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
|
||||
--retries 3 --timeout 60 \
|
||||
|| pip install regex -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
|
||||
|| python3 -m pip install regex -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
|
||||
fi
|
||||
|
||||
python3 scripts/ci/npu/precise-test/test_selector.py \
|
||||
@@ -658,10 +958,182 @@ jobs:
|
||||
test -s "${MAP_FILE}"
|
||||
du -h "${MAP_FILE}"
|
||||
|
||||
- name: Generate HTML coverage report
|
||||
# Best-effort add-on: continue-on-error keeps the job green and later
|
||||
# steps running, while a failed generation still shows a red ✗ in the UI.
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
|
||||
# Date tag derivation must match run_tests_with_coverage.sh.
|
||||
if [ -n "${GITHUB_RUN_STARTED_AT:-}" ]; then
|
||||
COV_DATE_TAG="${GITHUB_RUN_STARTED_AT:0:10}"
|
||||
COV_DATE_TAG="${COV_DATE_TAG//-/}"
|
||||
else
|
||||
COV_DATE_TAG="$(date +%Y%m%d)"
|
||||
fi
|
||||
COV_ROOT="${RUN_DIR}/outputs/sglang@${COV_DATE_TAG}"
|
||||
COVSTUB="${RUN_DIR}/outputs/covstub"
|
||||
# Standalone top-level dir, packaged into htmlcov.tar.gz at publish.
|
||||
REPORT_DIR="${RUN_DIR}/htmlcov"
|
||||
|
||||
# Same skip conditions as map building: no data -> nothing to report.
|
||||
if [ ! -d "${COV_ROOT}" ] \
|
||||
|| [ "$(find "${COV_ROOT}" -name 'coverage.*' -type f | wc -l)" -eq 0 ]; then
|
||||
echo "::warning::No coverage data found, skip HTML report"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Failures below exit non-zero (errexit) — red ✗, job stays green.
|
||||
# Do NOT convert failures to exit 0 behind an `if ! ( set -e; ... )`
|
||||
# guard: bash ignores -e inside a subshell in a condition context,
|
||||
# which silently swallowed three failures in run 38713909169.
|
||||
|
||||
# Pin matches the test jobs' coverage==7.8.* so combine can read
|
||||
# their SQLite data.
|
||||
if ! python3 -c "import coverage" 2>/dev/null; then
|
||||
python3 -m pip install coverage==7.8.* \
|
||||
-i http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
|
||||
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
|
||||
--retries 3 --timeout 60 \
|
||||
|| python3 -m pip install coverage==7.8.* -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
|
||||
fi
|
||||
|
||||
# Alias runner-specific source paths onto covstub; first entry must
|
||||
# be a real local path (no wildcards).
|
||||
{
|
||||
echo "[run]"
|
||||
echo "branch = True"
|
||||
echo "[paths]"
|
||||
echo "sglang ="
|
||||
echo " ${COVSTUB}/sglang/"
|
||||
echo " */python/sglang/"
|
||||
} > /tmp/report-rc
|
||||
|
||||
# --keep preserves the per-test raw coverage files: coverage-data.tar.gz
|
||||
# archives them after this step for manual map rebuilds.
|
||||
COVERAGE_FILE=/tmp/.coverage-all \
|
||||
python3 -m coverage combine --rcfile=/tmp/report-rc --keep \
|
||||
$(find "${COV_ROOT}" -name 'coverage.*' -type f)
|
||||
|
||||
rm -rf "${REPORT_DIR}"
|
||||
COVERAGE_FILE=/tmp/.coverage-all \
|
||||
python3 -m coverage html --rcfile=/tmp/report-rc \
|
||||
--directory="${REPORT_DIR}"
|
||||
|
||||
# NPU-only view: [run] source does not filter at report time (a
|
||||
# source-based rcfile yields the full report), so instead touch the
|
||||
# unexecuted npu .py files into a data copy (os.walk, so the PEP 420
|
||||
# namespace tree counts) and filter with --include, which works.
|
||||
cp /tmp/.coverage-all /tmp/.coverage-npu
|
||||
python3 - "${COVSTUB}" <<'PYEOF'
|
||||
import os
|
||||
import sys
|
||||
from coverage import CoverageData
|
||||
|
||||
npu_root = os.path.join(sys.argv[1], "sglang", "srt", "hardware_backend", "npu")
|
||||
data = CoverageData(basename="/tmp/.coverage-npu")
|
||||
data.read()
|
||||
measured = set(data.measured_files())
|
||||
added = 0
|
||||
for root, _, files in os.walk(npu_root):
|
||||
for f in files:
|
||||
if f.endswith(".py"):
|
||||
p = os.path.join(root, f)
|
||||
if p not in measured:
|
||||
data.touch_file(p)
|
||||
added += 1
|
||||
data.write()
|
||||
print(f"NPU view: {len(measured)} measured, touched {added} unexecuted files")
|
||||
PYEOF
|
||||
rm -rf "${REPORT_DIR}-npu"
|
||||
COVERAGE_FILE=/tmp/.coverage-npu \
|
||||
python3 -m coverage html --rcfile=/tmp/report-rc \
|
||||
--include='*/sglang/srt/hardware_backend/npu/*' \
|
||||
--directory="${REPORT_DIR}-npu"
|
||||
|
||||
# Surface the totals directly on the run page.
|
||||
# GITHUB_STEP_SUMMARY may be unset on the k8s runner pool.
|
||||
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
|
||||
{
|
||||
echo "### Coverage"
|
||||
echo '```'
|
||||
echo "python/sglang (measured files only):"
|
||||
COVERAGE_FILE=/tmp/.coverage-all \
|
||||
python3 -m coverage report --rcfile=/tmp/report-rc | tail -n 1
|
||||
echo "python/sglang/srt/hardware_backend/npu (full-tree denominator):"
|
||||
COVERAGE_FILE=/tmp/.coverage-npu \
|
||||
python3 -m coverage report --rcfile=/tmp/report-rc \
|
||||
--include='*/sglang/srt/hardware_backend/npu/*' | tail -n 1
|
||||
echo '```'
|
||||
echo "Full line-by-line report: download the \`coverage-html-report\` artifact and open \`htmlcov/index.html\` (NPU-only view: \`htmlcov-npu/index.html\`)."
|
||||
} >> "${GITHUB_STEP_SUMMARY}"
|
||||
fi
|
||||
|
||||
echo "HTML coverage report: ${REPORT_DIR}"
|
||||
echo "NPU-only HTML coverage report: ${REPORT_DIR}-npu"
|
||||
|
||||
- name: Upload HTML coverage report
|
||||
# Report upload must never block the baseline publish pipeline.
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: coverage-html-report
|
||||
# Two paths -> artifact root becomes the run dir, so the archive
|
||||
# holds htmlcov/ and htmlcov-npu/ side by side.
|
||||
path: |
|
||||
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/htmlcov
|
||||
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/htmlcov-npu
|
||||
if-no-files-found: ignore
|
||||
retention-days: 14
|
||||
|
||||
- name: Archive outputs
|
||||
run: |
|
||||
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
|
||||
tar -czf "${RUN_DIR}/outputs.tar.gz" -C "${RUN_DIR}" outputs
|
||||
# Source-only baseline: recommend mode needs covstub only, and it
|
||||
# shrinks the package from ~300MB to a few MB.
|
||||
tar -czf "${RUN_DIR}/outputs.tar.gz" -C "${RUN_DIR}" outputs/covstub
|
||||
# Raw coverage data for manual map rebuilds: extract both archives
|
||||
# under one root to reconstruct outputs/. Absent when no data.
|
||||
# Date tag derivation must match run_tests_with_coverage.sh.
|
||||
if [ -n "${GITHUB_RUN_STARTED_AT:-}" ]; then
|
||||
COV_DATE_TAG="${GITHUB_RUN_STARTED_AT:0:10}"
|
||||
COV_DATE_TAG="${COV_DATE_TAG//-/}"
|
||||
else
|
||||
COV_DATE_TAG="$(date +%Y%m%d)"
|
||||
fi
|
||||
if [ -d "${RUN_DIR}/outputs/sglang@${COV_DATE_TAG}" ]; then
|
||||
tar -czf "${RUN_DIR}/coverage-data.tar.gz" -C "${RUN_DIR}" "outputs/sglang@${COV_DATE_TAG}"
|
||||
fi
|
||||
|
||||
- name: Upload outputs and test case map
|
||||
# Never blocks the publish pipeline; must run before publish mv's
|
||||
# the files to the fixed PUBLISH_DIR (races with concurrent runs).
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: precise-test-outputs
|
||||
# Two paths -> artifact root becomes RUN_DIR, so the archive
|
||||
# holds outputs.tar.gz and test_case_map.json side by side.
|
||||
path: |
|
||||
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/outputs.tar.gz
|
||||
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/test_case_map.json
|
||||
if-no-files-found: ignore
|
||||
retention-days: 14
|
||||
|
||||
- name: Upload raw coverage data
|
||||
# Separate artifact keeps the ~300MB bulk out of the baseline package;
|
||||
# must run before publish mv's the file to the fixed PUBLISH_DIR.
|
||||
continue-on-error: true
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: precise-test-coverage-data
|
||||
path: /root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/coverage-data.tar.gz
|
||||
# Absent when all test jobs were skipped (no coverage data).
|
||||
if-no-files-found: ignore
|
||||
retention-days: 14
|
||||
|
||||
- name: Publish outputs and map to shared disk
|
||||
shell: bash
|
||||
@@ -669,14 +1141,14 @@ jobs:
|
||||
set -euo pipefail
|
||||
|
||||
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
|
||||
# Fixed publish dir with no RUN_DIR layer: other pipelines have a
|
||||
# different GITHUB_RUN_ID and must be able to read the baseline.
|
||||
# Must match PUBLISH_DIR in the recommend-tests-from-coverage job.
|
||||
# Fixed-name copy for manual use; the recommend job's baseline lives
|
||||
# in OBS (upload outputs.tar.gz + test_case_map.json manually after
|
||||
# each run). coverage-data.tar.gz has no automatic consumer — no OBS
|
||||
# copy, only this dir and the precise-test-coverage-data artifact.
|
||||
PUBLISH_DIR="/root/.cache/tests/precise-test/coverage"
|
||||
|
||||
# Skip publish when no coverage data was collected this run (map
|
||||
# build was skipped), so we never overwrite the historical
|
||||
# baseline on the shared disk with an incomplete one.
|
||||
# Skip publish when no coverage data this run — never overwrite
|
||||
# the historical baseline with an incomplete one.
|
||||
if [ ! -s "${RUN_DIR}/test_case_map.json" ]; then
|
||||
echo "::warning::test_case_map.json missing (no coverage data this run), skip publish"
|
||||
exit 0
|
||||
@@ -684,14 +1156,29 @@ jobs:
|
||||
|
||||
mkdir -p "${PUBLISH_DIR}"
|
||||
|
||||
# mv (rename) within the same shared filesystem is atomic, so a
|
||||
# concurrent reader sees either the old or the new baseline,
|
||||
# never a partial file.
|
||||
# Same-filesystem mv is atomic per file, so readers never see a
|
||||
# partial file (files from two concurrent publishes can interleave).
|
||||
mv "${RUN_DIR}/outputs.tar.gz" "${PUBLISH_DIR}/outputs.tar.gz"
|
||||
mv "${RUN_DIR}/test_case_map.json" "${PUBLISH_DIR}/test_case_map.json"
|
||||
# Guarded: absent in the unlikely case this run collected no
|
||||
# coverage data while still producing a map.
|
||||
if [ -f "${RUN_DIR}/coverage-data.tar.gz" ]; then
|
||||
mv "${RUN_DIR}/coverage-data.tar.gz" "${PUBLISH_DIR}/coverage-data.tar.gz"
|
||||
fi
|
||||
|
||||
# HTML reports too, fixed names — latest wins, like the baseline.
|
||||
# [ -d ] guards the continue-on-error generation step. A tar failure
|
||||
# aborts red after the baseline mv completed; Cleanup stays skipped,
|
||||
# keeping RUN_DIR for recovery.
|
||||
if [ -d "${RUN_DIR}/htmlcov" ]; then
|
||||
tar -czf "${PUBLISH_DIR}/htmlcov.tar.gz" -C "${RUN_DIR}" htmlcov
|
||||
fi
|
||||
if [ -d "${RUN_DIR}/htmlcov-npu" ]; then
|
||||
tar -czf "${PUBLISH_DIR}/htmlcov-npu.tar.gz" -C "${RUN_DIR}" htmlcov-npu
|
||||
fi
|
||||
|
||||
echo "Published baseline to ${PUBLISH_DIR}"
|
||||
du -h "${PUBLISH_DIR}/outputs.tar.gz" "${PUBLISH_DIR}/test_case_map.json"
|
||||
ls -lh "${PUBLISH_DIR}" | tail -n +2
|
||||
|
||||
- name: Clean up per-run directory
|
||||
# Skipped on publish failure, keeping RUN_DIR for manual recovery.
|
||||
@@ -708,6 +1195,7 @@ jobs:
|
||||
check-changes,
|
||||
|
||||
base-a-test-npu,
|
||||
base-a-test-npu-cov,
|
||||
base-b-test-1-npu-a3,
|
||||
base-b-test-2-npu-a3,
|
||||
base-b-test-4-npu-a3,
|
||||
|
||||
@@ -9,14 +9,14 @@ on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
run_all_tests:
|
||||
description: "Run the preflight even if the last commit touched no PPU path"
|
||||
description: "Build and run both sanity files even without PPU path changes"
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
workflow_call:
|
||||
inputs:
|
||||
ref:
|
||||
description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
|
||||
description: "Git ref to test; defaults to the triggering event SHA"
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
@@ -27,8 +27,7 @@ on:
|
||||
default: false
|
||||
|
||||
concurrency:
|
||||
# Newer runs cancel older runs for the same ref; unrelated label events
|
||||
# use isolated groups to avoid interfering with active runs.
|
||||
# Unrelated label events must not cancel or replace the active PPU checks.
|
||||
group: >-
|
||||
pr-test-ppu-${{
|
||||
(github.event_name == 'pull_request' &&
|
||||
@@ -40,11 +39,15 @@ concurrency:
|
||||
}}
|
||||
cancel-in-progress: true
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
env:
|
||||
GIT_CONFIG_NOSYSTEM: "1"
|
||||
|
||||
jobs:
|
||||
# ==================== Check Changes ==================== #
|
||||
check-changes:
|
||||
# Give skipped checks from unrelated labels distinct names so they cannot
|
||||
# replace the corresponding checks from an active run.
|
||||
name: >-
|
||||
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
|
||||
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
|
||||
@@ -59,25 +62,28 @@ jobs:
|
||||
contents: read
|
||||
pull-requests: read
|
||||
outputs:
|
||||
# Scope only; the shared pr-gate checks both required labels.
|
||||
ppu: ${{ steps.filter.outputs.ppu == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
|
||||
source-sha: ${{ steps.source.outputs.sha }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# github.sha (not github.ref) so a runner that dequeues after main
|
||||
# advances still validates the SHA that triggered the run.
|
||||
ref: ${{ inputs.ref || github.sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Pin the source revision for every job
|
||||
id: source
|
||||
run: printf 'sha=%s\n' "$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Determine run mode
|
||||
id: run-mode
|
||||
env:
|
||||
RUN_ALL_TESTS: ${{ inputs.run_all_tests }}
|
||||
run: |
|
||||
if [[ "${{ inputs.run_all_tests }}" == "true" ]]; then
|
||||
if [[ "$RUN_ALL_TESTS" == "true" ]]; then
|
||||
echo "run_all_tests=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Run mode: ALL TESTS"
|
||||
else
|
||||
echo "run_all_tests=false" >> "$GITHUB_OUTPUT"
|
||||
echo "Run mode: FILTERED (triggered by ${{ github.event_name }})"
|
||||
fi
|
||||
|
||||
- name: Detect file changes
|
||||
@@ -85,16 +91,18 @@ jobs:
|
||||
uses: dorny/paths-filter@v3
|
||||
if: steps.run-mode.outputs.run_all_tests != 'true'
|
||||
with:
|
||||
# PPU-specific paths only.
|
||||
filters: |
|
||||
ppu:
|
||||
- ".github/workflows/pr-test-ppu.yml"
|
||||
- "scripts/ci/ppu/**"
|
||||
- "python/sglang/srt/platforms/ppu.py"
|
||||
- "python/sglang/srt/hardware_backend/ppu/**"
|
||||
- "python/sglang/kernels/aot/csrc/ppu/**"
|
||||
- "python/sglang/srt/platforms/sail.py"
|
||||
- "python/sglang/kernels/aot/setup_sail.py"
|
||||
- "python/sglang/kernels/aot/pyproject_sail.toml"
|
||||
- "python/sglang/kernels/aot/csrc/common_extension_sail.cc"
|
||||
- "python/sglang/kernels/jit/utils/compile/sailify.py"
|
||||
- "test/registered/core/test_basic_sanity_eagle3.py"
|
||||
- "test/registered/core/test_basic_sanity_dflash.py"
|
||||
|
||||
# ==================== PR Gate ==================== #
|
||||
pr-gate:
|
||||
name: >-
|
||||
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
|
||||
@@ -108,62 +116,168 @@ jobs:
|
||||
require-label: run-ci-ppu
|
||||
secrets: inherit
|
||||
|
||||
# ==================== PPU Runner Preflight ==================== #
|
||||
# Validates the runner contract only; the SRT platform and AOT kernels land
|
||||
# in later PRs and build their first real test on top of this job.
|
||||
ppu-preflight:
|
||||
ppu-build:
|
||||
name: >-
|
||||
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
|
||||
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
|
||||
'ignored-label-ppu-preflight' || 'ppu-preflight' }}
|
||||
'ignored-label-ppu-build' || 'ppu-build' }}
|
||||
needs: [check-changes, pr-gate]
|
||||
if: needs.check-changes.outputs.ppu == 'true'
|
||||
runs-on: 1-ppu-m890p
|
||||
timeout-minutes: 60
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 120
|
||||
permissions:
|
||||
contents: read
|
||||
container:
|
||||
image: download.t-head.cn/docker_release/pytorch:sdk2.2.0-2.13.0-ubuntu24.04-hggcrt3-py312-20261005
|
||||
env:
|
||||
SGLANG_IS_IN_CI: true
|
||||
PPU_CI_SOURCE_SHA: ${{ needs.check-changes.outputs.source-sha }}
|
||||
MAX_JOBS: "2"
|
||||
SGL_KERNEL_COMPILE_THREADS: "2"
|
||||
outputs:
|
||||
wheel-artifact-id: ${{ steps.wheels.outputs.artifact-id }}
|
||||
steps:
|
||||
- name: Checkout code
|
||||
- name: Initialize private build directories and Git configuration
|
||||
run: |
|
||||
work="$(mktemp -d "${RUNNER_TEMP}/ppu-build.XXXXXX")"
|
||||
touch "$work/gitconfig"
|
||||
mkdir "$work/wheels" "$work/artifacts"
|
||||
{
|
||||
printf 'GIT_CONFIG_GLOBAL=%s\n' "$work/gitconfig"
|
||||
printf 'PPU_CI_WHEEL_DIR=%s\n' "$work/wheels"
|
||||
printf 'PPU_CI_ARTIFACT_DIR=%s\n' "$work/artifacts"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
# Same ref as check-changes. build_and_test.sh rejects any HEAD other than
|
||||
# PPU_CI_SOURCE_SHA, so a workflow_call branch that moves after
|
||||
# check-changes fails the run instead of building different code.
|
||||
- name: Checkout the triggering source revision
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Build native SAIL wheels
|
||||
run: |
|
||||
mkdir -p "$PPU_CI_WHEEL_DIR"
|
||||
bash scripts/ci/ppu/build_and_test.sh build
|
||||
|
||||
- name: Upload PPU wheels
|
||||
id: wheels
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ppu-wheels-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: ${{ env.PPU_CI_WHEEL_DIR }}
|
||||
if-no-files-found: error
|
||||
|
||||
- name: Upload build diagnostics
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ppu-build-logs-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: ${{ env.PPU_CI_ARTIFACT_DIR }}
|
||||
if-no-files-found: warn
|
||||
|
||||
ppu-sanity:
|
||||
name: >-
|
||||
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
|
||||
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
|
||||
format('ignored-label-ppu-sanity ({0})', matrix.runner) ||
|
||||
format('ppu-sanity ({0})', matrix.runner) }}
|
||||
needs: [check-changes, pr-gate, ppu-build]
|
||||
if: needs.check-changes.outputs.ppu == 'true'
|
||||
# Both PPU generations reuse the same wheels; each host supplies its own
|
||||
# runtime image and device, while checkpoints come from the shared NAS.
|
||||
strategy:
|
||||
# A failure on one generation must not hide the other generation's result.
|
||||
fail-fast: false
|
||||
matrix:
|
||||
runner: [8-ppu-m890p, 16-ppu-zw810e]
|
||||
runs-on: ${{ matrix.runner }}
|
||||
timeout-minutes: 60
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
env:
|
||||
PPU_CI_SOURCE_SHA: ${{ needs.check-changes.outputs.source-sha }}
|
||||
steps:
|
||||
- name: Initialize private runtime directories and Git configuration
|
||||
run: |
|
||||
work="$(mktemp -d "${RUNNER_TEMP}/ppu-runtime.XXXXXX")"
|
||||
touch "$work/gitconfig"
|
||||
mkdir "$work/wheels" "$work/artifacts"
|
||||
{
|
||||
printf 'GIT_CONFIG_GLOBAL=%s\n' "$work/gitconfig"
|
||||
printf 'PPU_CI_WHEEL_DIR=%s\n' "$work/wheels"
|
||||
printf 'PPU_CI_ARTIFACT_DIR=%s\n' "$work/artifacts"
|
||||
} >> "$GITHUB_ENV"
|
||||
|
||||
# Same ref as check-changes; run_test.sh rejects any HEAD other than
|
||||
# PPU_CI_SOURCE_SHA.
|
||||
- name: Checkout the triggering source revision
|
||||
timeout-minutes: 10
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Validate PPU runner contract
|
||||
timeout-minutes: 10
|
||||
run: python3 scripts/ci/ppu/check_ppu_environment.py
|
||||
- name: Download the matching PPU wheel artifact
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
artifact-ids: ${{ needs.ppu-build.outputs.wheel-artifact-id }}
|
||||
merge-multiple: true
|
||||
path: ${{ env.PPU_CI_WHEEL_DIR }}
|
||||
|
||||
- name: Run both speculative sanity files in isolated Docker
|
||||
env:
|
||||
CONFIG_PPU_CI_DEVICE: ${{ vars.PPU_CI_DEVICE }}
|
||||
CONFIG_PPU_CI_TARGET_MODEL_PATH: ${{ vars.PPU_CI_TARGET_MODEL_PATH }}
|
||||
CONFIG_PPU_CI_EAGLE3_MODEL_PATH: ${{ vars.PPU_CI_EAGLE3_MODEL_PATH }}
|
||||
CONFIG_PPU_CI_DFLASH_MODEL_PATH: ${{ vars.PPU_CI_DFLASH_MODEL_PATH }}
|
||||
CONFIG_PPU_CI_EVAL_CACHE: ${{ vars.PPU_CI_EVAL_CACHE }}
|
||||
run: |
|
||||
for key in PPU_CI_DEVICE PPU_CI_TARGET_MODEL_PATH PPU_CI_EAGLE3_MODEL_PATH PPU_CI_DFLASH_MODEL_PATH PPU_CI_EVAL_CACHE; do
|
||||
override="CONFIG_$key"
|
||||
if [[ -n "${!override}" ]]; then
|
||||
export "$key=${!override}"
|
||||
fi
|
||||
done
|
||||
exec bash scripts/ci/ppu/run_test.sh
|
||||
|
||||
- name: Upload runtime diagnostics
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: ppu-runtime-logs-${{ matrix.runner }}-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: ${{ env.PPU_CI_ARTIFACT_DIR }}
|
||||
if-no-files-found: warn
|
||||
|
||||
# ==================== Finish ==================== #
|
||||
pr-test-ppu-finish:
|
||||
name: >-
|
||||
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
|
||||
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
|
||||
'ignored-label-pr-test-ppu-finish' || 'pr-test-ppu-finish' }}
|
||||
needs: [pr-gate, check-changes, ppu-preflight]
|
||||
needs: [pr-gate, check-changes, ppu-build, ppu-sanity]
|
||||
if: always() && needs.check-changes.result != 'skipped'
|
||||
runs-on: ubuntu-latest
|
||||
permissions: {}
|
||||
steps:
|
||||
- name: Check all dependent job statuses
|
||||
- name: Require successful native build and both PPU sanity files when requested
|
||||
env:
|
||||
NEEDS_JSON: ${{ toJson(needs) }}
|
||||
PPU_IN_SCOPE: ${{ needs.check-changes.outputs.ppu }}
|
||||
BUILD_RESULT: ${{ needs.ppu-build.result }}
|
||||
SANITY_RESULT: ${{ needs.ppu-sanity.result }}
|
||||
run: |
|
||||
json_needs='${{ toJson(needs) }}'
|
||||
for job in $(echo "$json_needs" | jq -r 'keys_unsorted[]'); do
|
||||
result=$(echo "$json_needs" | jq -r --arg j "$job" '.[$j].result')
|
||||
echo "$job: $result"
|
||||
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
|
||||
echo "The above jobs failed."
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
echo "All jobs completed successfully"
|
||||
|
||||
- name: Require the preflight when PPU is in scope
|
||||
run: |
|
||||
in_scope='${{ needs.check-changes.outputs.ppu }}'
|
||||
preflight='${{ needs.ppu-preflight.result }}'
|
||||
echo "PPU in scope: '$in_scope' / ppu-preflight: '$preflight'"
|
||||
|
||||
if [[ "$in_scope" == "true" && "$preflight" != "success" ]]; then
|
||||
echo "PPU was in scope but ppu-preflight did not succeed."
|
||||
if jq -e 'any(.[]; .result == "failure" or .result == "cancelled")' <<< "$NEEDS_JSON" >/dev/null; then
|
||||
printf '%s\n' '::error::A PPU CI dependency failed or was cancelled.'
|
||||
exit 1
|
||||
fi
|
||||
echo "Preflight status is consistent with the requested scope."
|
||||
if [[ "$PPU_IN_SCOPE" == "true" ]]; then
|
||||
if [[ "$BUILD_RESULT" != "success" || "$SANITY_RESULT" != "success" ]]; then
|
||||
printf '%s\n' '::error::PPU scope requires a successful native build and both PPU sanity files.'
|
||||
exit 1
|
||||
fi
|
||||
printf '%s\n' 'Native build and both PPU sanity files passed.'
|
||||
else
|
||||
printf '%s\n' 'PPU is outside this run scope; no PPU tests were requested.'
|
||||
fi
|
||||
|
||||
@@ -56,7 +56,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- uses: ./.github/actions/download-rust-ext
|
||||
with:
|
||||
@@ -95,7 +95,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- uses: ./.github/actions/download-rust-ext
|
||||
with:
|
||||
@@ -146,7 +146,7 @@ jobs:
|
||||
with:
|
||||
path: python/sglang/kernels/aot/dist/
|
||||
merge-multiple: true
|
||||
pattern: wheel-python3.10-cuda*
|
||||
pattern: wheel-python3.11-cuda*
|
||||
|
||||
- uses: ./.github/actions/download-rust-ext
|
||||
with:
|
||||
@@ -181,7 +181,7 @@ jobs:
|
||||
# with:
|
||||
# path: python/sglang/kernels/aot/dist/
|
||||
# merge-multiple: true
|
||||
# pattern: wheel-python3.10-cuda*
|
||||
# pattern: wheel-python3.11-cuda*
|
||||
|
||||
# - name: Install dependencies
|
||||
# run: |
|
||||
|
||||
@@ -6,10 +6,10 @@ name: PR Test (sgl-router)
|
||||
# fires on every PR synchronize / push, so the run record always
|
||||
# appears as a check on the PR (no more "workflow silently didn't
|
||||
# fire" mystery debugging). Path-based skip decisions are made
|
||||
# inside the `sgl-router-gate` job below, where we can log the
|
||||
# inside the `sgl-router-changes` job below, where we can log the
|
||||
# reason. Same for the `run-ci` label gate — moved off the
|
||||
# workflow trigger and into the gate job so its outcome is
|
||||
# observable.
|
||||
# observable, and now fails (rather than skips) when it is missing.
|
||||
#
|
||||
# * `push` on `main` keeps firing so post-merge runs still record
|
||||
# against the default branch.
|
||||
@@ -29,21 +29,17 @@ env:
|
||||
SGLANG_IS_IN_CI: true
|
||||
|
||||
jobs:
|
||||
# Stage 0 — decide whether to run the heavy tiers. Always runs
|
||||
# (cheap, ubuntu-latest), and emits a single `should_run` output
|
||||
# consumed by every tier below. Reasons it might evict downstream
|
||||
# work:
|
||||
# * `pull_request` event without the `run-ci` label (budget gate)
|
||||
# * No files matching the sgl-router paths changed in this PR
|
||||
# Either outcome is logged on the gate job's page, so operators can
|
||||
# see *why* the tiers were skipped instead of guessing.
|
||||
sgl-router-gate:
|
||||
name: gate
|
||||
# Stage 0a — path detection. Always runs (cheap, ubuntu-latest) and
|
||||
# emits a single `should_run` output consumed by every tier below.
|
||||
# It only answers "does this PR touch sgl-router?"; the `run-ci`
|
||||
# label is checked by the separate gate job so that a missing label
|
||||
# fails that job instead of silently skipping the tiers (see below).
|
||||
sgl-router-changes:
|
||||
name: changes
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
should_run: ${{ steps.decide.outputs.should_run }}
|
||||
paths_changed: ${{ steps.paths.outputs.sgl_router }}
|
||||
has_run_ci_label: ${{ steps.label.outputs.has_run_ci }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
# `dorny/paths-filter` computes the diff against the PR base
|
||||
@@ -53,53 +49,62 @@ jobs:
|
||||
# the gate is observable.
|
||||
- name: Detect sgl-router path changes
|
||||
id: paths
|
||||
if: github.event_name != 'workflow_dispatch'
|
||||
uses: dorny/paths-filter@v3
|
||||
with:
|
||||
filters: |
|
||||
sgl_router:
|
||||
- 'experimental/sgl-router/**'
|
||||
- 'sgl-router/**'
|
||||
- 'rust/Cargo.toml'
|
||||
- 'rust/sglang-processor/**'
|
||||
- 'rust/sglang-api-types/**'
|
||||
- 'proto/**'
|
||||
- '.github/workflows/pr-test-sgl-router.yml'
|
||||
- 'docker/sgl-router.Dockerfile'
|
||||
- 'scripts/ci/cuda/ci_install_dependency.sh'
|
||||
# `run-ci` label gate. Cheap workflow runs (e.g. docs-only PRs
|
||||
# that happen to touch the sgl-router dir) still need the
|
||||
# opt-in label before consuming the H100 / kind tiers below.
|
||||
# `push` events on main and `workflow_dispatch` skip this gate.
|
||||
- name: Check run-ci label
|
||||
id: label
|
||||
run: |
|
||||
if [[ "${{ github.event_name }}" != "pull_request" ]]; then
|
||||
echo "has_run_ci=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Non-PR event (${{ github.event_name }}); label gate bypassed."
|
||||
exit 0
|
||||
fi
|
||||
if [[ "${{ contains(github.event.pull_request.labels.*.name, 'run-ci') }}" == "true" ]]; then
|
||||
echo "has_run_ci=true" >> "$GITHUB_OUTPUT"
|
||||
echo "PR has run-ci label; downstream tiers will run."
|
||||
else
|
||||
echo "has_run_ci=false" >> "$GITHUB_OUTPUT"
|
||||
echo "::warning::PR is missing the 'run-ci' label; skipping sgl-router tiers. Apply the label to opt in."
|
||||
fi
|
||||
- name: Decide
|
||||
id: decide
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
PATHS_CHANGED: ${{ steps.paths.outputs.sgl_router }}
|
||||
run: |
|
||||
paths='${{ steps.paths.outputs.sgl_router }}'
|
||||
label='${{ steps.label.outputs.has_run_ci }}'
|
||||
if [[ "$paths" == "true" && "$label" == "true" ]]; then
|
||||
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
|
||||
echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
echo "Both path filter and run-ci label match; running tiers."
|
||||
else
|
||||
echo "Manual dispatch; running sgl-router tiers."
|
||||
elif [[ "$PATHS_CHANGED" != "true" ]]; then
|
||||
echo "should_run=false" >> "$GITHUB_OUTPUT"
|
||||
echo "Skipping tiers (paths_changed=$paths, has_run_ci=$label)."
|
||||
echo "No sgl-router-related changes; skipping tiers."
|
||||
else
|
||||
echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
echo "sgl-router-related changes detected; running tiers."
|
||||
fi
|
||||
|
||||
# Stage 0b — `run-ci` label gate, shared with `pr-test.yml`. Cheap
|
||||
# workflow runs (e.g. docs-only PRs that happen to touch the
|
||||
# sgl-router dir) still need the opt-in label before consuming the
|
||||
# H100 / kind tiers below. `push` on main and `workflow_dispatch`
|
||||
# pass straight through.
|
||||
#
|
||||
# `pr-gate.yml` fetches the PR's labels live and FAILS when `run-ci`
|
||||
# is missing. Both matter for `/tag-and-rerun-ci`: the slash handler
|
||||
# adds the label with GITHUB_TOKEN, which fires no `labeled` event,
|
||||
# then reruns the PR's unsuccessful runs. A rerun replays the original
|
||||
# event payload, so a gate reading `github.event.pull_request.labels`
|
||||
# would skip again, and a run that concluded `success` with every
|
||||
# tier skipped is never rerun at all. Failing here on the live label
|
||||
# set makes the rerun both happen and pick the label up.
|
||||
sgl-router-gate:
|
||||
name: gate
|
||||
needs: sgl-router-changes
|
||||
if: needs.sgl-router-changes.outputs.should_run == 'true'
|
||||
uses: ./.github/workflows/pr-gate.yml
|
||||
with:
|
||||
require-run-ci: true
|
||||
|
||||
sgl-router-lint:
|
||||
name: tier-1 — lint
|
||||
needs: sgl-router-gate
|
||||
if: needs.sgl-router-gate.outputs.should_run == 'true'
|
||||
needs: [sgl-router-changes, sgl-router-gate]
|
||||
if: needs.sgl-router-changes.outputs.should_run == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
# Scoped per-job (not workflow-wide) to avoid SMG's documented breakage
|
||||
@@ -133,22 +138,22 @@ jobs:
|
||||
- name: Rust cache
|
||||
uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: experimental/sgl-router
|
||||
workspaces: sgl-router
|
||||
shared-key: "sgl-router-cache"
|
||||
cache-all-crates: true
|
||||
cache-on-failure: true
|
||||
save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: cargo check
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
run: cargo check --locked --workspace --all-targets
|
||||
|
||||
- name: cargo clippy
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
run: cargo clippy --locked --workspace --all-targets -- -D warnings
|
||||
|
||||
- name: cargo fmt
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
run: |
|
||||
rustup toolchain install nightly --profile minimal --component rustfmt
|
||||
cargo +nightly fmt --all -- --check
|
||||
@@ -158,7 +163,7 @@ jobs:
|
||||
run: cargo install --locked cargo-deny@0.19.6
|
||||
|
||||
- name: cargo deny check
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
run: cargo deny check licenses bans sources advisories
|
||||
|
||||
- name: Show sccache stats
|
||||
@@ -167,8 +172,8 @@ jobs:
|
||||
|
||||
sgl-router-build-and-test:
|
||||
name: tier-2 — build + test
|
||||
needs: [sgl-router-gate, sgl-router-lint]
|
||||
if: needs.sgl-router-gate.outputs.should_run == 'true'
|
||||
needs: [sgl-router-changes, sgl-router-gate, sgl-router-lint]
|
||||
if: needs.sgl-router-changes.outputs.should_run == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
RUSTC_WRAPPER: sccache
|
||||
@@ -206,14 +211,14 @@ jobs:
|
||||
- name: Rust cache
|
||||
uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: experimental/sgl-router
|
||||
workspaces: sgl-router
|
||||
shared-key: "sgl-router-cache"
|
||||
cache-all-crates: true
|
||||
cache-on-failure: true
|
||||
save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: cargo build (record wall time)
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
run: |
|
||||
START=$(date +%s)
|
||||
cargo build --locked --release --all-targets
|
||||
@@ -223,7 +228,7 @@ jobs:
|
||||
echo "::notice title=Tier-2 build duration::${BUILD_SECONDS}s"
|
||||
|
||||
- name: cargo test
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
# Skip the tokenizer parity_matrix test here: ubuntu-latest has
|
||||
# no HuggingFace cache, so the matrix would hard-fail (see the
|
||||
# test docstring). The e2e job runs it after pytest populates
|
||||
@@ -245,13 +250,13 @@ jobs:
|
||||
# through to a no-diff "pass" when no fixture was actually
|
||||
# regenerated.
|
||||
- name: kv_events hash parity fixture (drift check)
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python3 tests/scripts/generate_kv_events_hash_parity.py
|
||||
if ! git diff --exit-code -- tests/fixtures/kv_events_hash_parity.json; then
|
||||
echo "::error::kv_events hash parity fixture drifted." \
|
||||
"Re-run experimental/sgl-router/tests/scripts/generate_kv_events_hash_parity.py" \
|
||||
"Re-run sgl-router/tests/scripts/generate_kv_events_hash_parity.py" \
|
||||
"and commit the updated tests/fixtures/kv_events_hash_parity.json."
|
||||
exit 1
|
||||
fi
|
||||
@@ -268,8 +273,8 @@ jobs:
|
||||
|
||||
sgl-router-docker-build-test:
|
||||
name: tier-3 — docker (placeholder)
|
||||
needs: [sgl-router-gate, sgl-router-build-and-test]
|
||||
if: needs.sgl-router-gate.outputs.should_run == 'true'
|
||||
needs: [sgl-router-changes, sgl-router-gate, sgl-router-build-and-test]
|
||||
if: needs.sgl-router-changes.outputs.should_run == 'true'
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
@@ -286,8 +291,8 @@ jobs:
|
||||
# the GPU/cluster runners from spinning up.
|
||||
sgl-router-k8s-integration:
|
||||
name: tier-3 — k8s integration
|
||||
needs: [sgl-router-gate, sgl-router-build-and-test]
|
||||
if: needs.sgl-router-gate.outputs.should_run == 'true'
|
||||
needs: [sgl-router-changes, sgl-router-gate, sgl-router-build-and-test]
|
||||
if: needs.sgl-router-changes.outputs.should_run == 'true'
|
||||
runs-on: ubuntu-22.04
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
@@ -303,20 +308,20 @@ jobs:
|
||||
curl -Lo /tmp/kubectl https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl
|
||||
chmod +x /tmp/kubectl && sudo mv /tmp/kubectl /usr/local/bin/kubectl
|
||||
- name: Build router image
|
||||
run: docker build -t sgl-router:e2e -f experimental/sgl-router/tests/e2e/k8s_integration/Dockerfile.router .
|
||||
run: docker build -t sgl-router:e2e -f sgl-router/tests/e2e/k8s_integration/Dockerfile.router .
|
||||
- name: Build fake_worker image
|
||||
run: |
|
||||
docker build -t sgl-router-fake-worker:e2e \
|
||||
-f experimental/sgl-router/tests/e2e/k8s_integration/Dockerfile.fake_worker \
|
||||
experimental/sgl-router/tests/e2e/k8s_integration/
|
||||
-f sgl-router/tests/e2e/k8s_integration/Dockerfile.fake_worker \
|
||||
sgl-router/tests/e2e/k8s_integration/
|
||||
- name: Bootstrap kind + deploy
|
||||
run: bash experimental/sgl-router/tests/e2e/k8s_integration/setup.sh
|
||||
run: bash sgl-router/tests/e2e/k8s_integration/setup.sh
|
||||
- name: Set up Python
|
||||
run: |
|
||||
python3 -m venv /tmp/e2e-venv
|
||||
/tmp/e2e-venv/bin/pip install -r experimental/sgl-router/tests/e2e/k8s_integration/requirements.txt
|
||||
/tmp/e2e-venv/bin/pip install -r sgl-router/tests/e2e/k8s_integration/requirements.txt
|
||||
- name: Run E2E
|
||||
run: /tmp/e2e-venv/bin/pytest experimental/sgl-router/tests/e2e/k8s_integration/ -v --tb=short
|
||||
run: /tmp/e2e-venv/bin/pytest sgl-router/tests/e2e/k8s_integration/ -v --tb=short
|
||||
- name: Dump cluster + router diagnostics on failure
|
||||
if: failure()
|
||||
run: |
|
||||
@@ -333,8 +338,8 @@ jobs:
|
||||
|
||||
sgl-router-e2e:
|
||||
name: tier-3 — e2e
|
||||
needs: [sgl-router-gate, sgl-router-build-and-test]
|
||||
if: needs.sgl-router-gate.outputs.should_run == 'true'
|
||||
needs: [sgl-router-changes, sgl-router-gate, sgl-router-build-and-test]
|
||||
if: needs.sgl-router-changes.outputs.should_run == 'true'
|
||||
runs-on: 2-gpu-h100
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
@@ -344,10 +349,10 @@ jobs:
|
||||
- name: Rust cache
|
||||
uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: experimental/sgl-router
|
||||
workspaces: sgl-router
|
||||
shared-key: "sgl-router-cache"
|
||||
- name: cargo build (release)
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
run: |
|
||||
source "$HOME/.cargo/env"
|
||||
cargo build --locked --release --workspace
|
||||
@@ -365,7 +370,7 @@ jobs:
|
||||
|
||||
- name: Install e2e test dependencies
|
||||
run: |
|
||||
python3 -m pip install -r experimental/sgl-router/tests/e2e/requirements.txt
|
||||
python3 -m pip install -r sgl-router/tests/e2e/requirements.txt
|
||||
|
||||
# IMPORTANT: --ignore=tests/e2e/k8s_integration. The k8s integration
|
||||
# suite is owned by the `sgl-router-k8s-integration` job which has
|
||||
@@ -376,7 +381,7 @@ jobs:
|
||||
# image pull) must not surface as e2e failures on H100, and e2e
|
||||
# flakes (model load, HF auth) must not surface as k8s failures.
|
||||
- name: Run e2e
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
@@ -384,7 +389,7 @@ jobs:
|
||||
--ignore=tests/e2e/k8s_integration
|
||||
|
||||
- name: Tokenizer parity matrix (uses HF cache populated by e2e)
|
||||
working-directory: experimental/sgl-router
|
||||
working-directory: sgl-router
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
run: |
|
||||
@@ -394,25 +399,43 @@ jobs:
|
||||
sgl-router-finish:
|
||||
name: finish
|
||||
needs:
|
||||
- sgl-router-changes
|
||||
- sgl-router-gate
|
||||
- sgl-router-lint
|
||||
- sgl-router-build-and-test
|
||||
- sgl-router-docker-build-test
|
||||
- sgl-router-k8s-integration
|
||||
- sgl-router-e2e
|
||||
# `always()` lets `finish` run after upstream `skipped` outcomes
|
||||
# (when the gate decided to skip). The downstream-skipped path is
|
||||
# an explicit success — the workflow was correctly bypassed for a
|
||||
# PR that didn't need full CI. The downstream-failure /
|
||||
# downstream-cancelled path is a real fail. We also accept the
|
||||
# gate's `skipped` (impossible today, kept for symmetry) and
|
||||
# require the gate itself to have succeeded.
|
||||
if: |
|
||||
always() &&
|
||||
needs.sgl-router-gate.result == 'success' &&
|
||||
!contains(needs.*.result, 'failure') &&
|
||||
!contains(needs.*.result, 'cancelled')
|
||||
# `always()` lets `finish` run after upstream `skipped` outcomes.
|
||||
# Two outcomes are a success: every tier passed, or the PR touches
|
||||
# no sgl-router path and every tier was skipped. Anything else,
|
||||
# including a gate that failed for a missing `run-ci` label, fails
|
||||
# the run, so `/tag-and-rerun-ci` sees an unsuccessful run to rerun.
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: All required checks completed
|
||||
run: echo "OK"
|
||||
- name: Check required results
|
||||
env:
|
||||
CHANGES_RESULT: ${{ needs.sgl-router-changes.result }}
|
||||
SHOULD_RUN: ${{ needs.sgl-router-changes.outputs.should_run }}
|
||||
GATE_RESULT: ${{ needs.sgl-router-gate.result }}
|
||||
LINT_RESULT: ${{ needs.sgl-router-lint.result }}
|
||||
BUILD_RESULT: ${{ needs.sgl-router-build-and-test.result }}
|
||||
DOCKER_RESULT: ${{ needs.sgl-router-docker-build-test.result }}
|
||||
K8S_RESULT: ${{ needs.sgl-router-k8s-integration.result }}
|
||||
E2E_RESULT: ${{ needs.sgl-router-e2e.result }}
|
||||
run: |
|
||||
if [[ "$CHANGES_RESULT" != "success" ]]; then
|
||||
echo "::error::sgl-router path detection did not succeed ($CHANGES_RESULT)."
|
||||
exit 1
|
||||
fi
|
||||
tiers="$GATE_RESULT $LINT_RESULT $BUILD_RESULT $DOCKER_RESULT $K8S_RESULT $E2E_RESULT"
|
||||
if [[ "$SHOULD_RUN" == "true" && "$tiers" == "success success success success success success" ]]; then
|
||||
echo "All sgl-router checks passed."
|
||||
elif [[ "$SHOULD_RUN" == "false" && "$tiers" == "skipped skipped skipped skipped skipped skipped" ]]; then
|
||||
echo "No sgl-router-related changes; tiers skipped."
|
||||
else
|
||||
echo "::error::Unexpected sgl-router results: should_run=$SHOULD_RUN, gate=$GATE_RESULT, lint=$LINT_RESULT, build=$BUILD_RESULT, docker=$DOCKER_RESULT, k8s=$K8S_RESULT, e2e=$E2E_RESULT."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -93,31 +93,27 @@ jobs:
|
||||
suite: "stage-a-tp-test-cpu-intel"
|
||||
name: gnr
|
||||
|
||||
# Three SPR boxes each run ONE full-machine container taking one
|
||||
# partition of a 3-way split over the merged non-TP pool
|
||||
# (base-b non-TP + base-c). Single container per box (no socket
|
||||
# split): sglang auto-computes mem_fraction_static from host memory
|
||||
# and does not honor cgroup limits, so two parallel containers on
|
||||
# one box exhaust host RAM and OOM (tracked upstream). One container
|
||||
# per box keeps each server within the box's memory budget.
|
||||
# Each SPR box (2 sockets / 2 NUMA, 32 physical cores each) runs TWO
|
||||
# socket-pinned containers in parallel, each taking one partition of a
|
||||
# 4-way split over the merged non-TP pool. Each container is capped
|
||||
# with --memory so sglang (which reads cgroup memory.max, PR #38260 +
|
||||
# the empty-NUMA-node fix) sizes its KV pool against one socket's
|
||||
# budget, not the whole host -- so two parallel servers no longer
|
||||
# exhaust host RAM. 2 boxes x 2 sockets = 4 partitions.
|
||||
- runner: xeon-spr
|
||||
role: spr
|
||||
suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel"
|
||||
name: spr1
|
||||
part_id: 0
|
||||
part_size: 3
|
||||
part_lo: 0
|
||||
part_hi: 1
|
||||
part_size: 4
|
||||
- runner: xeon-spr
|
||||
role: spr
|
||||
suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel"
|
||||
name: spr2
|
||||
part_id: 1
|
||||
part_size: 3
|
||||
- runner: xeon-spr
|
||||
role: spr
|
||||
suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel"
|
||||
name: spr3
|
||||
part_id: 2
|
||||
part_size: 3
|
||||
part_lo: 2
|
||||
part_hi: 3
|
||||
part_size: 4
|
||||
steps:
|
||||
- name: Cleanup workspace
|
||||
run: |
|
||||
@@ -169,43 +165,70 @@ jobs:
|
||||
timeout-minutes: 120
|
||||
run: |
|
||||
docker exec -w /sglang-checkout/ ci_sglang_${{ matrix.name }} \
|
||||
bash -c "source /opt/.venv/bin/activate && cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --timeout-from-est-time"
|
||||
bash -c "source /opt/.venv/bin/activate && hf auth login --token \$HF_TOKEN && cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --timeout-from-est-time"
|
||||
|
||||
# ============ SPR: single full-machine container, one partition ====== #
|
||||
- name: Run container (SPR)
|
||||
if: matrix.role == 'spr'
|
||||
run: |
|
||||
docker rm -f ci_sglang_${{ matrix.name }} || true
|
||||
docker run -dt \
|
||||
-v ${{ github.workspace }}:/sglang-checkout/ --shm-size=64g \
|
||||
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
|
||||
-v $HOME/.cache/sgl_eval:/root/.cache/sgl_eval \
|
||||
-e HF_TOKEN="$(cat ~/huggingface_token.txt)" \
|
||||
--name ci_sglang_${{ matrix.name }} \
|
||||
sglang_xeon
|
||||
|
||||
- name: Check AMX support (SPR)
|
||||
if: matrix.role == 'spr'
|
||||
timeout-minutes: 5
|
||||
run: |
|
||||
docker exec -w /sglang-checkout/ ci_sglang_${{ matrix.name }} \
|
||||
bash -c "source /opt/.venv/bin/activate && python3 -c 'import torch; import sgl_kernel; assert torch.cpu._is_amx_tile_supported(); assert hasattr(torch.ops.sgl_kernel, \"convert_weight_packed\"); '"
|
||||
|
||||
- name: Run unit tests (SPR)
|
||||
# ========= SPR: two socket-pinned containers, run in parallel ======== #
|
||||
- name: Run unit tests (SPR, two sockets in parallel)
|
||||
if: matrix.role == 'spr'
|
||||
timeout-minutes: 120
|
||||
run: |
|
||||
docker exec -w /sglang-checkout/ ci_sglang_${{ matrix.name }} \
|
||||
bash -c "source /opt/.venv/bin/activate && cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --auto-partition-id ${{ matrix.part_id }} --auto-partition-size ${{ matrix.part_size }} --timeout-from-est-time"
|
||||
set -uo pipefail
|
||||
HF_TOKEN_VAL="$(cat ~/huggingface_token.txt)"
|
||||
|
||||
# One socket-pinned container. Physical cores only (0-31 = socket0,
|
||||
# 32-63 = socket1); logical/HT siblings (64-127) are never bound.
|
||||
# --memory caps the cgroup so sglang sizes KV against one socket, and
|
||||
# --cpuset-mems keeps allocations on the local memory bank.
|
||||
# $1 socket id, $2 cpuset-cpus, $3 cpuset-mems, $4 partition id
|
||||
run_socket () {
|
||||
local sid=$1 cpus=$2 mems=$3 pid=$4
|
||||
local cname=ci_sglang_${{ matrix.name }}_s${sid}
|
||||
docker rm -f "$cname" >/dev/null 2>&1 || true
|
||||
docker run -dt \
|
||||
--cpuset-cpus="$cpus" --cpuset-mems="$mems" --memory=240g \
|
||||
-v ${{ github.workspace }}:/sglang-checkout/ --shm-size=32g \
|
||||
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
|
||||
-v $HOME/.cache/sgl_eval:/root/.cache/sgl_eval \
|
||||
-e HF_TOKEN="$HF_TOKEN_VAL" \
|
||||
-e SGLANG_CPU_OMP_THREADS_BIND="$cpus" \
|
||||
--name "$cname" \
|
||||
sglang_xeon
|
||||
docker exec -w /sglang-checkout/ "$cname" bash -c \
|
||||
"source /opt/.venv/bin/activate && python3 -c 'import torch; import sgl_kernel; assert torch.cpu._is_amx_tile_supported(); assert hasattr(torch.ops.sgl_kernel, \"convert_weight_packed\")' && \
|
||||
hf auth login --token \$HF_TOKEN && \
|
||||
cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --auto-partition-id $pid --auto-partition-size ${{ matrix.part_size }} --timeout-from-est-time"
|
||||
}
|
||||
|
||||
# socket0 -> part_lo, socket1 -> part_hi; run concurrently, wait both.
|
||||
run_socket 0 "0-31" 0 ${{ matrix.part_lo }} > s0.log 2>&1 &
|
||||
P0=$!
|
||||
run_socket 1 "32-63" 1 ${{ matrix.part_hi }} > s1.log 2>&1 &
|
||||
P1=$!
|
||||
|
||||
rc=0
|
||||
wait $P0 || rc=1
|
||||
wait $P1 || rc=1
|
||||
|
||||
echo "==================== socket0 (partition ${{ matrix.part_lo }}) ===================="
|
||||
cat s0.log || true
|
||||
echo "==================== socket1 (partition ${{ matrix.part_hi }}) ===================="
|
||||
cat s1.log || true
|
||||
exit $rc
|
||||
|
||||
# ============================ Common ================================= #
|
||||
- name: Change permission
|
||||
if: always()
|
||||
timeout-minutes: 2
|
||||
run: |
|
||||
docker exec -u root ci_sglang_${{ matrix.name }} bash -c "rm -rf /tmp/ci-home && chown -R $(id -u):$(id -g) /sglang-checkout/ 2>/dev/null || true" 2>/dev/null || true
|
||||
# GNR has one container (ci_sglang_<name>); SPR has two socket
|
||||
# containers (ci_sglang_<name>_s0/_s1). Chown from whichever exist.
|
||||
for c in ci_sglang_${{ matrix.name }} ci_sglang_${{ matrix.name }}_s0 ci_sglang_${{ matrix.name }}_s1; do
|
||||
docker exec -u root "$c" bash -c "rm -rf /tmp/ci-home && chown -R $(id -u):$(id -g) /sglang-checkout/ 2>/dev/null || true" 2>/dev/null || true
|
||||
done
|
||||
|
||||
- name: Cleanup container
|
||||
if: always()
|
||||
run: |
|
||||
docker rm -f ci_sglang_${{ matrix.name }} || true
|
||||
for c in ci_sglang_${{ matrix.name }} ci_sglang_${{ matrix.name }}_s0 ci_sglang_${{ matrix.name }}_s1; do
|
||||
docker rm -f "$c" || true
|
||||
done
|
||||
|
||||
@@ -446,6 +446,7 @@ jobs:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- { runner_config: 2-gpu-gb300, run_timeout: '30', timeout_per_file: '1800' }
|
||||
- { runner_config: 4-gpu-gb300, run_timeout: '30', timeout_per_file: '1800' }
|
||||
uses: ./.github/workflows/_pr-test-stage.yml
|
||||
with:
|
||||
|
||||
@@ -2,10 +2,20 @@ name: Release Docker Images Nightly gfx1151 (AMD)
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
push:
|
||||
description: "Push the image to Docker Hub (uncheck to only check that the branch builds)"
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
schedule:
|
||||
# Stagger this build from the CDNA ROCm nightlies at 12:00 UTC.
|
||||
- cron: "0 14 * * *"
|
||||
|
||||
# Scheduled runs always push; a manual dispatch can turn the push off.
|
||||
env:
|
||||
PUSH_IMAGES: ${{ github.event_name != 'workflow_dispatch' || inputs.push }}
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
@@ -59,5 +69,9 @@ jobs:
|
||||
--no-cache \
|
||||
--progress=plain \
|
||||
-t "rocm/sgl-dev:${IMAGE_TAG}"
|
||||
docker push "rocm/sgl-dev:${IMAGE_TAG}"
|
||||
echo "Published rocm/sgl-dev:${IMAGE_TAG}" >> "$GITHUB_STEP_SUMMARY"
|
||||
if [ "${PUSH_IMAGES}" = "true" ]; then
|
||||
docker push "rocm/sgl-dev:${IMAGE_TAG}"
|
||||
echo "Published rocm/sgl-dev:${IMAGE_TAG}" >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
echo "Build-only run: not pushing rocm/sgl-dev:${IMAGE_TAG}" >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
|
||||
@@ -21,8 +21,17 @@ on:
|
||||
- 'all'
|
||||
- gfx942-rocm1000
|
||||
- gfx950-rocm1000
|
||||
push:
|
||||
description: 'Push the images to Docker Hub (uncheck to only check that the branch builds)'
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
schedule:
|
||||
- cron: '0 12 * * *'
|
||||
- cron: '0 10 * * *' # before the ROCm 7.2 build, so it is not queued behind it
|
||||
|
||||
# Scheduled runs always push; a manual dispatch can turn the push off.
|
||||
env:
|
||||
PUSH_IMAGES: ${{ github.event_name != 'workflow_dispatch' || inputs.push }}
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.number || github.sha }}
|
||||
@@ -106,15 +115,21 @@ jobs:
|
||||
--build-arg UBUNTU_MIRROR=https://archive.ubuntu.com \
|
||||
-t "rocm/sgl-dev:${image_tag}" \
|
||||
--no-cache
|
||||
docker push "rocm/sgl-dev:${image_tag}"
|
||||
if [ "${PUSH_IMAGES}" = "true" ]; then
|
||||
docker push "rocm/sgl-dev:${image_tag}"
|
||||
else
|
||||
echo "Build-only run: not pushing rocm/sgl-dev:${image_tag}"
|
||||
fi
|
||||
|
||||
- name: Login to Docker Hub (lmsys)
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Push to lmsysorg/sglang-rocm
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
run: |
|
||||
docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}"
|
||||
docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}"
|
||||
@@ -178,15 +193,21 @@ jobs:
|
||||
--build-arg UBUNTU_MIRROR=https://archive.ubuntu.com \
|
||||
-t "rocm/sgl-dev:${image_tag}" \
|
||||
--no-cache
|
||||
docker push "rocm/sgl-dev:${image_tag}"
|
||||
if [ "${PUSH_IMAGES}" = "true" ]; then
|
||||
docker push "rocm/sgl-dev:${image_tag}"
|
||||
else
|
||||
echo "Build-only run: not pushing rocm/sgl-dev:${image_tag}"
|
||||
fi
|
||||
|
||||
- name: Login to Docker Hub (lmsys)
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Push to lmsysorg/sglang-rocm
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
run: |
|
||||
docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}"
|
||||
docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}"
|
||||
@@ -253,15 +274,21 @@ jobs:
|
||||
--build-arg UBUNTU_MIRROR=https://archive.ubuntu.com \
|
||||
-t "rocm/sgl-dev:${image_tag}" \
|
||||
--no-cache
|
||||
docker push "rocm/sgl-dev:${image_tag}"
|
||||
if [ "${PUSH_IMAGES}" = "true" ]; then
|
||||
docker push "rocm/sgl-dev:${image_tag}"
|
||||
else
|
||||
echo "Build-only run: not pushing rocm/sgl-dev:${image_tag}"
|
||||
fi
|
||||
|
||||
- name: Login to Docker Hub (lmsys)
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Push to lmsysorg/sglang-rocm
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
run: |
|
||||
docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}"
|
||||
docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}"
|
||||
|
||||
@@ -21,8 +21,17 @@ on:
|
||||
- gfx950-rocm724
|
||||
- gfx942-rocm720
|
||||
- gfx950-rocm720
|
||||
push:
|
||||
description: 'Push the images to Docker Hub (uncheck to only check that the branch builds)'
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
schedule:
|
||||
- cron: '0 12 * * *'
|
||||
- cron: '0 11 * * *' # an hour after ROCm 10, ahead of the 12:00 UTC MILES builds
|
||||
|
||||
# Scheduled runs always push; a manual dispatch can turn the push off.
|
||||
env:
|
||||
PUSH_IMAGES: ${{ github.event_name != 'workflow_dispatch' || inputs.push }}
|
||||
|
||||
concurrency:
|
||||
# A PR number if a pull request and otherwise the commit hash. This cancels
|
||||
@@ -112,18 +121,24 @@ jobs:
|
||||
# UBUNTU_MIRROR forces apt over HTTPS to dodge port-80 reachability flakes
|
||||
# to Canonical's archive.ubuntu.com mirror IPs from the amd-docker-scale runner.
|
||||
docker build . -f docker/rocm.Dockerfile --build-arg SGL_BRANCH=${{ github.ref_name }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --build-arg GPU_ARCH=${{ matrix.gpu_arch }} --build-arg ENABLE_MORI=1 --build-arg ENABLE_NIXL=1 --build-arg SETUPTOOLS_SCM_PRETEND_VERSION=${pretend_version} --build-arg UBUNTU_MIRROR=https://archive.ubuntu.com -t rocm/sgl-dev:${tag}-${{ env.DATE }} --no-cache
|
||||
docker push rocm/sgl-dev:${tag}-${{ env.DATE }}
|
||||
if [ "${PUSH_IMAGES}" = "true" ]; then
|
||||
docker push rocm/sgl-dev:${tag}-${{ env.DATE }}
|
||||
else
|
||||
echo "Build-only run: not pushing rocm/sgl-dev:${tag}-${{ env.DATE }}"
|
||||
fi
|
||||
|
||||
# Persist the tag right after rocm/sgl-dev push succeeds so the local
|
||||
# registry mirror can run even if a later step in this job (lmsys push)
|
||||
# fails. By default this step only runs when the previous step succeeded,
|
||||
# so the artifact only exists when an image actually landed on Docker Hub.
|
||||
- name: Save published image tag
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
run: |
|
||||
mkdir -p image-tag
|
||||
echo "${{ env.IMAGE_TAG }}" > "image-tag/${{ matrix.gpu_arch }}.txt"
|
||||
|
||||
- name: Upload image tag artifact
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: image-tag-${{ matrix.gpu_arch }}
|
||||
@@ -131,12 +146,14 @@ jobs:
|
||||
retention-days: 1
|
||||
|
||||
- name: Login to Docker Hub (lmsys)
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
uses: docker/login-action@v2
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Push to lmsysorg/sglang-rocm
|
||||
if: env.PUSH_IMAGES == 'true'
|
||||
run: |
|
||||
docker tag rocm/sgl-dev:${{ env.IMAGE_TAG }} lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}
|
||||
docker push lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}
|
||||
|
||||
@@ -57,6 +57,7 @@ jobs:
|
||||
kernel_source=wheel
|
||||
kernel_ref=main
|
||||
latest_tag=latest
|
||||
image_tag="${image_tag}-main-whl"
|
||||
else
|
||||
# Pin the kernel commit so the tag names exactly what was built.
|
||||
kernel_ref=$(git ls-remote https://github.com/sgl-project/sgl-kernel-xpu.git refs/heads/main | cut -f1)
|
||||
@@ -66,7 +67,7 @@ jobs:
|
||||
fi
|
||||
kernel_source=source
|
||||
latest_tag=latest-kernel-main
|
||||
image_tag="${image_tag}-kernel-main-${kernel_ref:0:7}"
|
||||
image_tag="${image_tag}-main-main-${kernel_ref:0:7}"
|
||||
fi
|
||||
echo "DATE=${date_tag}" >> $GITHUB_ENV
|
||||
echo "commit_hash=${commit_hash}" >> $GITHUB_OUTPUT
|
||||
|
||||
@@ -83,7 +83,7 @@ jobs:
|
||||
provenance: false
|
||||
sbom: false
|
||||
build-args: |
|
||||
SGLANG_KERNEL_NPU_TAG=2026.9.0.post9
|
||||
SGLANG_KERNEL_NPU_TAG=2026.9.0.post10
|
||||
CANN_VERSION=${{ matrix.cann_version }}
|
||||
DEVICE_TYPE=${{ matrix.device_type }}
|
||||
|
||||
|
||||
@@ -87,7 +87,7 @@ jobs:
|
||||
push: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
|
||||
provenance: false
|
||||
build-args: |
|
||||
SGLANG_KERNEL_NPU_TAG=2026.9.0.post9
|
||||
SGLANG_KERNEL_NPU_TAG=2026.9.0.post10
|
||||
CANN_VERSION=${{ matrix.cann_version }}
|
||||
DEVICE_TYPE=${{ matrix.device_type }}
|
||||
SGLANG_TAG=${{ steps.version.outputs.version }}
|
||||
|
||||
@@ -161,7 +161,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.10"
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Update wheel index
|
||||
run: |
|
||||
|
||||
@@ -170,7 +170,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.10"
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Update wheel index
|
||||
run: |
|
||||
|
||||
@@ -104,7 +104,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.10"
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Upload to pypi
|
||||
run: |
|
||||
|
||||
@@ -26,7 +26,7 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
||||
python-version: ["3.11", "3.12", "3.13"]
|
||||
arch: [x86_64, aarch64]
|
||||
include:
|
||||
- arch: x86_64
|
||||
|
||||
@@ -168,7 +168,6 @@ jobs:
|
||||
# so bind eagerly to fail here on any missing symbol.
|
||||
LD_BIND_NOW=1 python3 - <<'PY'
|
||||
import ctypes
|
||||
import sys
|
||||
import torch
|
||||
import deep_ep
|
||||
import deep_gemm
|
||||
@@ -179,9 +178,6 @@ jobs:
|
||||
assert ctypes.CDLL("libnccl.so.2").ncclGetVersion(ctypes.byref(nccl_version)) == 0
|
||||
assert nccl_version.value == 23007, nccl_version.value
|
||||
assert hasattr(deep_ep, "ElasticBuffer"), "DeepEP ElasticBuffer is required"
|
||||
sys.path.insert(0, "DeepGEMM/third-party")
|
||||
from tilelang_ops import ref_mhc
|
||||
assert ref_mhc.has_baseline(), "Mega mHC reference operators are unavailable"
|
||||
print("DeepGEMM test dependencies ready; NCCL:", nccl_version.value)
|
||||
PY
|
||||
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
name: Release sgl-flash-mla
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version:
|
||||
description: "Wheel version (e.g. 1.0.0, 1.0.1rc0)"
|
||||
type: string
|
||||
required: true
|
||||
branch:
|
||||
description: "FlashMLA branch to build from (default: sgl)"
|
||||
type: string
|
||||
required: false
|
||||
default: 'sgl'
|
||||
validation_only:
|
||||
description: "Build and test without publishing wheels"
|
||||
type: boolean
|
||||
required: false
|
||||
default: false
|
||||
skip_tests:
|
||||
description: "Skip the GPU test suite (publishes untested wheels)"
|
||||
type: boolean
|
||||
required: false
|
||||
default: false
|
||||
|
||||
concurrency:
|
||||
group: release-sgl-flash-mla-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
# The wheel pins this; it is also what the test jobs install.
|
||||
# The extensions use PyTorch's stable C++ ABI and would load against any 2.10+,
|
||||
# but a release is tested against one version.
|
||||
TORCH_VER: "2.14.1"
|
||||
|
||||
jobs:
|
||||
build:
|
||||
if: github.repository == 'sgl-project/sglang'
|
||||
strategy:
|
||||
matrix:
|
||||
# One wheel per architecture. No Python matrix: the wheel is abi3
|
||||
# (cp310-abi3) and serves CPython 3.10+, so 3.12 only drives the build.
|
||||
python-version: ["3.12"]
|
||||
cuda-version: ["13.4"]
|
||||
arch: [x86_64, aarch64]
|
||||
include:
|
||||
- arch: x86_64
|
||||
runner: x64-kernel-build-node
|
||||
- arch: aarch64
|
||||
runner: arm-kernel-build-node
|
||||
runs-on: ${{ matrix.runner }}
|
||||
steps:
|
||||
- name: Clean workspace (remove root-owned files from prior runs)
|
||||
run: |
|
||||
docker run --rm -v "${{ github.workspace }}:/workspace" alpine:3 \
|
||||
sh -c 'rm -rf /workspace/..?* /workspace/.[!.]* /workspace/*' || true
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Checkout FlashMLA
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
repository: sgl-project/FlashMLA
|
||||
ref: ${{ inputs.branch || 'sgl' }}
|
||||
path: FlashMLA
|
||||
submodules: recursive
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
|
||||
- name: Set wheel version
|
||||
run: |
|
||||
echo -n "${{ inputs.version }}" > FlashMLA/flash_mla/VERSION
|
||||
cat FlashMLA/flash_mla/VERSION
|
||||
|
||||
- name: Build wheel
|
||||
run: |
|
||||
chmod +x ./scripts/build_sgl_flash_mla.sh ./scripts/rename_sgl_flash_mla_whl.sh
|
||||
./scripts/build_sgl_flash_mla.sh "${{ matrix.python-version }}" "${{ matrix.cuda-version }}" "${{ github.workspace }}/FlashMLA" "${{ matrix.arch }}"
|
||||
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: flashmla-wheel-${{ matrix.arch }}
|
||||
path: FlashMLA/dist/*.whl
|
||||
|
||||
- name: Upload PyPI artifacts
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: flashmla-pypi-${{ matrix.arch }}
|
||||
path: FlashMLA/dist-pypi/*.whl
|
||||
|
||||
test:
|
||||
needs: build
|
||||
if: ${{ !inputs.skip_tests }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
# SM90 runs only the frozen kernels: the current ones are sm_100a/sm_103a.
|
||||
# The frozen ones are built for sm_90a/sm_100a/sm_103a, so SM100 runs both.
|
||||
- arch_label: sm90
|
||||
runner: 8-gpu-h200
|
||||
wheel_arch: x86_64
|
||||
tests: legacy
|
||||
timeout: 90
|
||||
- arch_label: sm100
|
||||
runner: 8-gpu-b200
|
||||
wheel_arch: x86_64
|
||||
tests: current legacy
|
||||
timeout: 180
|
||||
- arch_label: sm103-aarch64
|
||||
runner: 4-gpu-gb300
|
||||
wheel_arch: aarch64
|
||||
tests: current legacy
|
||||
timeout: 180
|
||||
runs-on: ${{ matrix.runner }}
|
||||
timeout-minutes: ${{ matrix.timeout }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Checkout FlashMLA
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
repository: sgl-project/FlashMLA
|
||||
ref: ${{ inputs.branch || 'sgl' }}
|
||||
path: FlashMLA
|
||||
submodules: recursive
|
||||
|
||||
# The tests need only tests/. The source package would shadow the installed
|
||||
# wheel (its directory has no built extensions), so it is removed.
|
||||
- name: Remove the FlashMLA source package
|
||||
run: rm -rf FlashMLA/flash_mla
|
||||
|
||||
- name: Download wheel
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
path: dist/
|
||||
merge-multiple: true
|
||||
pattern: flashmla-wheel-${{ matrix.wheel_arch }}
|
||||
|
||||
- name: Install wheel and deps
|
||||
run: |
|
||||
python3 -m pip install --upgrade pip
|
||||
python3 -m pip install "torch==${TORCH_VER}" --index-url https://download.pytorch.org/whl/cu130 --force-reinstall
|
||||
python3 -m pip install numpy einops rich pytest
|
||||
# --force-reinstall: a base image may already carry this version, in
|
||||
# which case pip skips the wheel and the job tests a stale extension.
|
||||
python3 -m pip install dist/*.whl --force-reinstall --no-deps
|
||||
# Both extensions register through STABLE_TORCH_LIBRARY at dlopen time,
|
||||
# so a missing symbol surfaces here rather than at the first launch.
|
||||
python3 -c "
|
||||
import flash_mla
|
||||
from flash_mla._loader import get_ops
|
||||
print('flash_mla:', flash_mla.__file__)
|
||||
import os
|
||||
assert not flash_mla.__file__.startswith(os.environ['GITHUB_WORKSPACE']), 'not the installed wheel'
|
||||
print('version:', flash_mla.__version__)
|
||||
print('has_legacy:', flash_mla.has_legacy())
|
||||
assert get_ops(use_legacy=False).sparse_decode_fwd is not None
|
||||
assert get_ops(use_legacy=True).dense_decode_fwd is not None
|
||||
"
|
||||
|
||||
- name: Run tests (current kernels)
|
||||
if: contains(matrix.tests, 'current')
|
||||
working-directory: FlashMLA
|
||||
env:
|
||||
PYTHONPATH: ${{ github.workspace }}/FlashMLA/tests
|
||||
run: |
|
||||
python3 -c "import os, flash_mla; assert not flash_mla.__file__.startswith(os.environ['GITHUB_WORKSPACE']), flash_mla.__file__"
|
||||
python3 tests/test-sparse-prefill.py
|
||||
python3 tests/test-sparse-decode.py
|
||||
|
||||
- name: Run tests (frozen kernels)
|
||||
if: contains(matrix.tests, 'legacy')
|
||||
working-directory: FlashMLA
|
||||
env:
|
||||
# conftest.py only fixes sys.path under pytest; these are scripts.
|
||||
PYTHONPATH: ${{ github.workspace }}/FlashMLA/tests
|
||||
run: |
|
||||
python3 -c "import os, flash_mla; assert not flash_mla.__file__.startswith(os.environ['GITHUB_WORKSPACE']), flash_mla.__file__"
|
||||
python3 -m pytest tests/legacy/ -q
|
||||
# Dense MLA decoding is SM90-only.
|
||||
if [ "${{ matrix.arch_label }}" = sm90 ]; then
|
||||
python3 tests/legacy/test_flash_mla_dense_decoding.py
|
||||
fi
|
||||
python3 tests/legacy/test_flash_mla_sparse_prefill.py
|
||||
python3 tests/legacy/test_flash_mla_sparse_decoding.py
|
||||
python3 tests/legacy/test_flash_mla_sparse_decoding_v32_bit_equal.py
|
||||
|
||||
release:
|
||||
if: |
|
||||
!cancelled() && !inputs.validation_only &&
|
||||
needs.build.result == 'success' &&
|
||||
(needs.test.result == 'success' || needs.test.result == 'skipped')
|
||||
needs: [build, test]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Download artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
path: dist/
|
||||
merge-multiple: true
|
||||
pattern: flashmla-wheel-*
|
||||
|
||||
- name: Download PyPI artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
path: dist-pypi/
|
||||
merge-multiple: true
|
||||
pattern: flashmla-pypi-*
|
||||
|
||||
- name: Record that the GPU test suite was skipped
|
||||
if: ${{ inputs.skip_tests }}
|
||||
run: |
|
||||
echo "Releasing v${{ inputs.version }} with the GPU test suite skipped." \
|
||||
>> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Upload to PyPI
|
||||
run: |
|
||||
pip install --upgrade twine "packaging>=24.2"
|
||||
python3 -m twine upload --skip-existing dist-pypi/* -u __token__ -p ${{ secrets.SGL_FLASH_MLA_PYPI_TOKEN }}
|
||||
|
||||
- name: Release
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
tag_name: v${{ inputs.version }}
|
||||
repository: sgl-project/whl
|
||||
token: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }}
|
||||
files: |
|
||||
dist/*
|
||||
|
||||
- name: Clone wheel index
|
||||
run: git clone https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git sgl-whl
|
||||
env:
|
||||
WHL_TOKEN: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }}
|
||||
|
||||
- name: Update wheel index
|
||||
run: python3 scripts/update_flashmla_whl_index.py --cuda 130
|
||||
|
||||
- name: Push wheel index
|
||||
run: |
|
||||
cd sgl-whl
|
||||
git config --local user.name "sglang-bot"
|
||||
git config --local user.email "sglangbot@gmail.com"
|
||||
git add -A
|
||||
git commit -m "update sgl-flash-mla whl index for v${{ inputs.version }}${{ inputs.skip_tests && ' (GPU tests skipped)' || '' }}"
|
||||
git push
|
||||
@@ -39,7 +39,7 @@ jobs:
|
||||
(github.event_name == 'push' || github.event.inputs.target == 'all' || github.event.inputs.target == 'cu130')
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.10"]
|
||||
python-version: ["3.11"]
|
||||
cuda-version: ["13.0"]
|
||||
arch: [x86_64, aarch64]
|
||||
include:
|
||||
@@ -174,8 +174,9 @@ jobs:
|
||||
matrix:
|
||||
python-version: ["3.10"]
|
||||
# ROCm 10.0.0 builds inside a Python 3.12 container; the wheel is still
|
||||
# tagged cp310-abi3 (wheel.py-api in pyproject.toml), so it stays under
|
||||
# the same python-version leg as the 7.x wheels.
|
||||
# tagged cp310-abi3 (build_rocm.sh sets wheel.py-api to cp310 for every
|
||||
# ROCm flavor), so it stays under the same python-version leg as the 7.x
|
||||
# wheels.
|
||||
#
|
||||
# A dispatch that names one flavor builds only that flavor. Every extra
|
||||
# leg is another image build on the shared AMD node, and its release job
|
||||
|
||||
@@ -183,7 +183,7 @@ jobs:
|
||||
env:
|
||||
TEST_COMMAND: ${{ matrix.test_command }}
|
||||
REQUESTED_TEST_COMMAND: ${{ inputs.test_command }}
|
||||
HF_MUSE_SPARK_TOKEN: ${{ inputs.runs_on == '8-gpu-b300' && inputs.pr_head_sha == '' && contains(matrix.test_command, 'registered/e2e/models/test_muse_spark_12_b300_pd.py') && secrets.HF_MUSE_SPARK_TOKEN || '' }}
|
||||
HF_MUSE_SPARK_TOKEN: ${{ inputs.pr_head_sha == '' && inputs.runs_on == '8-gpu-b300' && (contains(matrix.test_command, 'registered/e2e/models/test_muse_spark_12_b300_pd.py') || contains(matrix.test_command, 'registered/e2e/models/test_muse_spark_unified_memory.py')) && secrets.HF_MUSE_SPARK_TOKEN || '' }}
|
||||
SGLANG_PRECISION_HF_TOKEN: ${{ inputs.refresh_precision_baseline && inputs.mode == 'cuda' && inputs.runs_on == '8-gpu-h200' && inputs.test_command == 'registered/debug_utils/test_nightly_precision_regression.py' && inputs.pr_head_sha == '' && secrets.HF_TOKEN_PRECISION_STORE || '' }}
|
||||
run: |
|
||||
if [[ "${{ inputs.refresh_precision_baseline }}" == "true" ]]; then
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# Runs only the per-commit CPU unit suite (base-a-test-cpu) against a PR head,
|
||||
# dispatched by the `/run-cpu-ci` slash command. A cheap check for small PRs:
|
||||
# pair it with `/rerun-group <module>` instead of a full `run-ci` round.
|
||||
#
|
||||
# Always dispatched on main, so the workflow definition and every reusable
|
||||
# workflow it calls are trusted; only the code under test comes from the PR.
|
||||
# That code runs on hosted runners with a read-only token and no secrets
|
||||
# (nothing here inherits them), the same exposure as `/rerun-test` cpu mode.
|
||||
name: Run CPU CI
|
||||
run-name: "[run-cpu-ci] ${{ inputs.git_ref }}"
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
git_ref:
|
||||
description: "PR head SHA to test"
|
||||
required: true
|
||||
type: string
|
||||
pr_number:
|
||||
description: "PR number (informational)"
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
# Read by _pr-test-stage-cpu.yml via caller_inputs. Its fail-fast health
|
||||
# check inspects sibling jobs of a pull_request run, which this is not.
|
||||
skip_pr_test_health_check:
|
||||
description: "Skip the PR test health check fail-fast"
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
# Also read via caller_inputs. This run is on main's ref, so a cache it
|
||||
# saved would be restored by main and every PR, yet hold PR-written files.
|
||||
skip_cache_save:
|
||||
description: "Do not save the HF / Rust caches"
|
||||
required: false
|
||||
type: boolean
|
||||
default: true
|
||||
|
||||
concurrency:
|
||||
group: run-cpu-ci-${{ inputs.git_ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: read
|
||||
pull-requests: read
|
||||
|
||||
jobs:
|
||||
# pr_test_yml stays pr-test.yml: compute_partitions.py reads it from the PR
|
||||
# checkout, which may predate this file, and sizing shards off the per-commit
|
||||
# stage's run_timeout_minutes keeps the split identical to PR CI.
|
||||
check-changes:
|
||||
uses: ./.github/workflows/_pr-test-check-changes.yml
|
||||
with:
|
||||
git_ref: ${{ inputs.git_ref }}
|
||||
run_all_tests: true
|
||||
|
||||
# self_name must be the suite name: the stage passes it to run_suite.py.
|
||||
# No rust_ext_artifact: its compile job runs on a self-hosted build node, so
|
||||
# the stage falls back to its cache and compiles on a miss.
|
||||
base-a-test-cpu:
|
||||
needs: check-changes
|
||||
uses: ./.github/workflows/_pr-test-stage-cpu.yml
|
||||
with:
|
||||
self_name: base-a-test-cpu
|
||||
check_changes: ${{ toJson(needs.check-changes.outputs) }}
|
||||
caller_inputs: ${{ toJson(inputs) }}
|
||||
partitions: ${{ needs.check-changes.outputs.partitions }}
|
||||
run_timeout_minutes: '15'
|
||||
@@ -7,6 +7,7 @@ on:
|
||||
paths:
|
||||
- '.github/workflows/runner-utilization.yml'
|
||||
- 'scripts/ci/utils/runner_utilization_report.py'
|
||||
- 'scripts/ci/utils/test_runner_utilization_report.py'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
hours:
|
||||
@@ -50,7 +51,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Unit-test report logic
|
||||
# Pure-logic tests (stdlib only, no API calls) — guards the
|
||||
@@ -70,7 +71,7 @@ jobs:
|
||||
run: |
|
||||
# PR-trigger is just a script smoke check; scan a 20-minute window
|
||||
# (0.34h) to bound API cost. schedule / dispatch run the full report.
|
||||
python scripts/ci/utils/runner_utilization_report.py \
|
||||
python -u scripts/ci/utils/runner_utilization_report.py \
|
||||
--repo ${{ github.repository }} \
|
||||
--hours ${{ (github.event_name == 'pull_request' && '0.34') || inputs.hours || '24' }} \
|
||||
${{ inputs.filter && format('--filter {0}', inputs.filter) || '' }} \
|
||||
|
||||
@@ -27,6 +27,8 @@ jobs:
|
||||
contains(github.event.comment.body, '/run-extra-ci') ||
|
||||
contains(github.event.comment.body, '/rerun-full-ci') ||
|
||||
contains(github.event.comment.body, '/rerun-extra-ci') ||
|
||||
contains(github.event.comment.body, '/run-cpu-ci') ||
|
||||
contains(github.event.comment.body, '/rerun-cpu-ci') ||
|
||||
contains(github.event.comment.body, '/rerun-group') ||
|
||||
contains(github.event.comment.body, '/rerun-test') ||
|
||||
startsWith(github.event.comment.body, '/tag-') ||
|
||||
@@ -92,7 +94,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
|
||||
@@ -21,7 +21,7 @@ jobs:
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.10'
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Update est_time values
|
||||
env:
|
||||
|
||||
+3
-2
@@ -73,7 +73,8 @@ instance/
|
||||
|
||||
# PyBuilder
|
||||
.pybuilder/
|
||||
target/
|
||||
# `**/` because .dockerignore links here and Docker anchors bare patterns at the context root.
|
||||
**/target/
|
||||
|
||||
# Jupyter Notebook
|
||||
.ipynb_checkpoints
|
||||
@@ -232,7 +233,7 @@ work_dirs/
|
||||
Cargo.lock
|
||||
!rust/Cargo.lock
|
||||
!rust/sglang-radix-tree/Cargo.lock
|
||||
!experimental/sgl-router/Cargo.lock
|
||||
!sgl-router/Cargo.lock
|
||||
|
||||
# Other repos
|
||||
lmms-eval
|
||||
|
||||
+15
-7
@@ -11,10 +11,6 @@ repos:
|
||||
- id: end-of-file-fixer
|
||||
- id: check-yaml
|
||||
args: [--allow-multiple-documents]
|
||||
# Helm chart templates embed Go template syntax ({{- ... -}})
|
||||
# that is not valid YAML on its own; the rendered output is
|
||||
# validated by `helm template` / `helm lint`.
|
||||
exclude: ^experimental/sgl-router/helm/.*/templates/.*\.(yaml|tpl)$
|
||||
- id: check-toml
|
||||
- id: check-ast
|
||||
- id: check-added-large-files
|
||||
@@ -74,6 +70,12 @@ repos:
|
||||
- '--extra-keys=metadata.kernelspec metadata.language_info.version'
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: forbid-test-srt-path
|
||||
name: forbid test/srt/ (use test/registered/ or test/manual/)
|
||||
entry: >-
|
||||
test/srt/ is retired. Move these tests under test/registered/ or test/manual/ instead.
|
||||
language: fail
|
||||
files: ^test/srt/
|
||||
- id: forbid-docs-new-path
|
||||
name: forbid docs_new/ (renamed to docs/)
|
||||
# docs_new/ was renamed to docs/ in #32123. `language: fail` fires only
|
||||
@@ -110,6 +112,12 @@ repos:
|
||||
language: system
|
||||
files: ^(\.github/actions/download-rust-ext/action\.yml|\.github/workflows/_pr-test-rust-ext-build\.yml)$
|
||||
pass_filenames: false
|
||||
- id: check-dockerfile-wheel-pins
|
||||
name: check Dockerfile sgl-deep-gemm/sgl-deep-ep versions match python/pyproject.toml
|
||||
entry: python3 scripts/lint/check_dockerfile_wheel_pins.py
|
||||
language: system
|
||||
files: ^(python/pyproject\.toml|docker/.*Dockerfile.*|scripts/lint/check_dockerfile_wheel_pins\.py)$
|
||||
pass_filenames: false
|
||||
- id: check-no-bare-pytest-main
|
||||
name: reject bare pytest.main calls in __main__ blocks
|
||||
entry: python3 scripts/lint/check_no_bare_pytest_main.py
|
||||
@@ -134,10 +142,10 @@ repos:
|
||||
files: ^sgl-model-gateway/.*\.rs$
|
||||
pass_filenames: false
|
||||
- id: rustfmt-sgl-router
|
||||
name: rustfmt experimental/sgl-router
|
||||
entry: bash -c 'cd experimental/sgl-router && cargo fmt --all -- --check'
|
||||
name: rustfmt sgl-router
|
||||
entry: bash -c 'cd sgl-router && cargo fmt --all -- --check'
|
||||
language: system
|
||||
files: ^experimental/sgl-router/.*\.rs$
|
||||
files: ^sgl-router/.*\.rs$
|
||||
pass_filenames: false
|
||||
- id: clippy-rust-workspace
|
||||
name: clippy rust/ workspace (auto-fix)
|
||||
|
||||
+15
-1
@@ -56,6 +56,19 @@ if [[ "${ROCM_VERSION}" == "1000" ]]; then
|
||||
WHEEL_ROCM_VERSION="1000"
|
||||
fi
|
||||
|
||||
# The build reuses the CUDA pyproject.toml, which requires Python 3.11 and tags
|
||||
# its wheel cp311-abi3. The ROCm 7.2.0 image and amd-sglang still run on Python
|
||||
# 3.10, so the ROCm wheels keep the Python 3.10 floor and the cp310-abi3 tag.
|
||||
sed -i \
|
||||
-e 's/^requires-python = ">=3.11"$/requires-python = ">=3.10"/' \
|
||||
-e 's/^wheel.py-api = "cp311"$/wheel.py-api = "cp310"/' \
|
||||
pyproject.toml
|
||||
if ! grep -qx 'requires-python = ">=3.10"' pyproject.toml \
|
||||
|| ! grep -qx 'wheel.py-api = "cp310"' pyproject.toml; then
|
||||
echo "ERROR: could not set the ROCm Python 3.10 / cp310 metadata in pyproject.toml." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pull and run the latest image
|
||||
echo "Pulling Docker image: ${IMAGE}"
|
||||
docker pull "${IMAGE}"
|
||||
@@ -86,5 +99,6 @@ ${ROCM_SETUP}
|
||||
rm -rf CMakeLists.txt && mv CMakeLists_rocm.txt CMakeLists.txt && \
|
||||
${PYTHON_ROOT_PATH}/python rocm_hipify.py && \
|
||||
${PYTHON_ROOT_PATH}/python -m uv build --wheel -Cbuild-dir=build . --color=always --no-build-isolation && \
|
||||
./rename_wheels_rocm.sh ${WHEEL_ROCM_VERSION}
|
||||
./rename_wheels_rocm.sh ${WHEEL_ROCM_VERSION} && \
|
||||
${PYTHON_ROOT_PATH}/pip install --dry-run --no-deps dist/*.whl
|
||||
"
|
||||
|
||||
@@ -22,6 +22,11 @@ SGLang is an open-source inference framework for large language, vision-language
|
||||
|
||||
👋 Get started below, or meet the community at [SGLang Events](https://www.sglang.io/events), including meetups, developer meetings, workshops, and office hours.
|
||||
|
||||
> **[SGLang Summit 2026: Meet the builders of open AI](https://www.sglang.io/summit)**<br>
|
||||
> **Our flagship annual gathering · November 12–13 · San Francisco**
|
||||
>
|
||||
> Go deep on inference, training, and AI hardware through technical talks, live demos, and hands-on workshops. Bring your deployment challenges and meet the engineers and researchers building the stack.
|
||||
|
||||
## Get Started
|
||||
|
||||
Pull the Docker image, which includes SGLang and its dependencies:
|
||||
@@ -49,7 +54,7 @@ SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platfo
|
||||
| --- | --- |
|
||||
| [NVIDIA](https://docs.sglang.io/docs/hardware-platforms/nvidia-gpus) | B200/B300/GB200/GB300; H100/H200/H800/H20; A100; RTX 30/40/50 series, RTX 6000 Ada / PRO 6000; DGX Spark, Jetson Orin |
|
||||
| [AMD](https://docs.sglang.io/docs/hardware-platforms/amd_gpu) | Instinct MI300X, MI325X, MI350X, MI355X |
|
||||
| [Google TPU](https://docs.sglang.io/docs/hardware-platforms/tpu) | v6e, v7; [SGL-JAX](https://github.com/sgl-project/sglang-jax) / [SGL-torchtpu](https://lmsys.org/blog/2026-07-30-sglang-google-tpu/) |
|
||||
| [Google TPU](https://docs.sglang.io/docs/hardware-platforms/tpu) | v6e, v7; [SGLang-TorchTPU](https://github.com/sgl-project/sglang-torchtpu) / [SGL-JAX](https://github.com/sgl-project/sglang-jax) |
|
||||
| Intel ([GPU](https://docs.sglang.io/docs/hardware-platforms/xpu) / [CPU](https://docs.sglang.io/docs/hardware-platforms/cpu_server)) | Arc / Arc Pro B-Series GPUs, Xeon CPUs |
|
||||
| [Apple Silicon](https://docs.sglang.io/docs/hardware-platforms/apple_metal) | Macs via Metal / MLX |
|
||||
| [Huawei Ascend](https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/installation) | A2, A3, 950PR/DT NPUs |
|
||||
@@ -57,6 +62,8 @@ SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platfo
|
||||
|
||||
Integrations in progress: AWS Trainium, [Alibaba T-Head PPU](https://github.com/sgl-project/sglang/issues/37519), [Cambricon MLU](https://github.com/sgl-project/sglang/pull/26898), Qualcomm QAIC, MetaX, Hygon HCU/DCU, Iluvatar CoreX, and more.
|
||||
|
||||
SGLang-TorchTPU brings the SGLang serving engine to Google TPUs through PyTorch, torch_tpu, and Pallas kernels. It supports LLM and vision-language serving on TPU7x (Ironwood), including tensor and expert parallelism across hosts. SGL-JAX provides the JAX-based TPU serving path.
|
||||
|
||||
See the [Cookbook](https://cookbook.sglang.io/) and platform guides for model compatibility and setup.
|
||||
|
||||
## SGL Ecosystem
|
||||
|
||||
@@ -20,10 +20,10 @@ import triton
|
||||
from sglang.kernels.ops.attention.cutedsl_kda import (
|
||||
cutedsl_fused_sigmoid_gating_kda_update,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
from sglang.kernels.ops.attention.fla.kda import chunk_kda
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.kda import chunk_kda
|
||||
|
||||
|
||||
def make_inputs(
|
||||
|
||||
@@ -32,7 +32,7 @@ import triton
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent import (
|
||||
fused_recurrent_gated_delta_rule_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
|
||||
|
||||
@@ -26,12 +26,12 @@ import torch
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent import (
|
||||
fused_recurrent_kda_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
from sglang.kernels.ops.attention.helion.kda_decode import (
|
||||
helion_fused_recurrent_kda_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
|
||||
|
||||
def make_inputs(
|
||||
|
||||
@@ -37,7 +37,7 @@ import triton
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent import (
|
||||
fused_recurrent_gated_delta_rule_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent_linear_replayssm import (
|
||||
from sglang.kernels.ops.attention.linear.fused_recurrent_linear_replayssm import (
|
||||
fused_recurrent_gdn_replayssm_decode,
|
||||
)
|
||||
|
||||
|
||||
@@ -22,11 +22,11 @@ import argparse
|
||||
|
||||
import torch
|
||||
|
||||
import sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify as fused_mod
|
||||
from sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify import (
|
||||
import sglang.kernels.ops.attention.linear.fused_kda_conv_recurrent_verify as fused_mod
|
||||
from sglang.kernels.ops.attention.linear.fused_kda_conv_recurrent_verify import (
|
||||
fused_kda_conv_gating_verify,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
from sglang.kernels.ops.mamba.causal_conv1d_triton import causal_conv1d_update
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
Compares the legacy gather (``gather_dequant_requant_fp8_paged_legacy``:
|
||||
fresh ``torch.zeros`` destination + one program per (token, 128-elem
|
||||
slice)) against the new gather (``gather_dequant_requant_fp8_paged``:
|
||||
slice)) against the new gather (``gather_dequant_requant_fp8_paged_dsa``:
|
||||
no pre-zeroing needed, fused pad-row zero-fill, TOKENS_PER_PROG tokens
|
||||
per program with 16B-vectorized access), in three flavors:
|
||||
|
||||
@@ -43,6 +43,8 @@ import sys
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
_MOD_PATH = _REPO_ROOT / "python/sglang/kernels/ops/attention/dsa/dequant_k_cache.py"
|
||||
@@ -54,8 +56,127 @@ _spec = importlib.util.spec_from_file_location("dequant_k_cache", _MOD_PATH)
|
||||
_mod = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_mod)
|
||||
|
||||
gather_new = _mod.gather_dequant_requant_fp8_paged
|
||||
gather_legacy = _mod.gather_dequant_requant_fp8_paged_legacy
|
||||
gather_new = _mod.gather_dequant_requant_fp8_paged_dsa
|
||||
_infer_dsa_dims = _mod._infer_dsa_dims
|
||||
|
||||
|
||||
def gather_dequant_requant_fp8_paged_legacy(
|
||||
quant_k_cache: torch.Tensor,
|
||||
page_table_1_flattened: torch.Tensor,
|
||||
group_size: int = 128,
|
||||
extra_rows: int = 0,
|
||||
) -> torch.Tensor:
|
||||
"""Legacy (pre-vectorization) gather + dequant + requant.
|
||||
|
||||
Kept as the bit-exactness / performance reference for
|
||||
``gather_dequant_requant_fp8_paged_dsa`` (see
|
||||
``benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py``). Allocates and zero-fills the
|
||||
full destination each call, then launches one program per
|
||||
(token, 128-elem slice).
|
||||
"""
|
||||
dim_quant = quant_k_cache.shape[-1]
|
||||
dim_nope, dim_rope = _infer_dsa_dims(dim_quant)
|
||||
quant_k_cache = quant_k_cache.view((-1, dim_quant))
|
||||
|
||||
num_tokens = page_table_1_flattened.shape[0]
|
||||
assert quant_k_cache.dtype == torch.float8_e4m3fn
|
||||
num_tiles = dim_nope // group_size
|
||||
out_dim = dim_nope + dim_rope
|
||||
assert num_tiles * group_size == dim_nope
|
||||
|
||||
total_rows = num_tokens + extra_rows
|
||||
# Allocate a fresh zero-filled buffer. The extra landing-pad rows at
|
||||
# the tail must read as zeros (the kernel may over-read past
|
||||
# num_tokens for masked indices).
|
||||
output = torch.zeros(
|
||||
(total_rows, 1, out_dim),
|
||||
dtype=torch.float8_e4m3fn,
|
||||
device=quant_k_cache.device,
|
||||
)
|
||||
|
||||
num_blocks_per_token = num_tiles + triton.cdiv(dim_rope, group_size)
|
||||
|
||||
input_nope_q = quant_k_cache[:, :dim_nope]
|
||||
input_nope_s = quant_k_cache[:, dim_nope : dim_nope + num_tiles * 4].view(
|
||||
torch.float32
|
||||
)
|
||||
input_rope = quant_k_cache[:, dim_nope + num_tiles * 4 :].view(torch.bfloat16)
|
||||
|
||||
_gather_dequant_requant_fp8_paged_kernel[(num_tokens, num_blocks_per_token)](
|
||||
output,
|
||||
input_nope_q,
|
||||
input_nope_s,
|
||||
input_rope,
|
||||
page_table_1_flattened,
|
||||
output.stride(0),
|
||||
input_nope_q.stride(0),
|
||||
input_nope_s.stride(0),
|
||||
input_rope.stride(0),
|
||||
NUM_NOPE_BLOCKS=num_tiles,
|
||||
GROUP_SIZE=group_size,
|
||||
DIM_NOPE=dim_nope,
|
||||
DIM_ROPE=dim_rope,
|
||||
)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _gather_dequant_requant_fp8_paged_kernel(
|
||||
output_ptr,
|
||||
input_nope_q_ptr,
|
||||
input_nope_s_ptr,
|
||||
input_rope_ptr,
|
||||
page_table_1_ptr,
|
||||
output_stride_0: int,
|
||||
input_nope_q_stride_0: int,
|
||||
input_nope_s_stride_0: int,
|
||||
input_rope_stride_0: int,
|
||||
NUM_NOPE_BLOCKS: tl.constexpr,
|
||||
GROUP_SIZE: tl.constexpr,
|
||||
DIM_NOPE: tl.constexpr,
|
||||
DIM_ROPE: tl.constexpr,
|
||||
):
|
||||
"""Fused gather + dequant(per-group) + requant(per-tensor) -> fp8."""
|
||||
token_id = tl.program_id(0)
|
||||
token_id_paged = tl.load(page_table_1_ptr + token_id).to(tl.int32)
|
||||
raw_block_id = tl.program_id(1)
|
||||
|
||||
if raw_block_id < NUM_NOPE_BLOCKS:
|
||||
# nope: read fp8, mul group scale -> f32, cast to fp8_e4m3fn
|
||||
effective_block_id = raw_block_id
|
||||
offs_q = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
|
||||
mask = offs_q < DIM_NOPE
|
||||
|
||||
ptr_q = input_nope_q_ptr + token_id_paged * input_nope_q_stride_0 + offs_q
|
||||
ptr_s = (
|
||||
input_nope_s_ptr
|
||||
+ token_id_paged * input_nope_s_stride_0
|
||||
+ effective_block_id
|
||||
)
|
||||
|
||||
y_q = tl.load(ptr_q, mask=mask, other=0.0).to(tl.float32)
|
||||
y_s = tl.load(ptr_s)
|
||||
|
||||
# dequant -> f32 -> requant to fp8
|
||||
y = (y_q * y_s).to(tl.float8e4nv)
|
||||
|
||||
dst_ptr = output_ptr + token_id * output_stride_0 + offs_q
|
||||
tl.store(dst_ptr, y, mask=mask)
|
||||
else:
|
||||
# rope: read bf16, cast to fp8
|
||||
effective_block_id = raw_block_id - NUM_NOPE_BLOCKS
|
||||
offs = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
|
||||
mask = offs < DIM_ROPE
|
||||
|
||||
src_ptr = input_rope_ptr + token_id_paged * input_rope_stride_0 + offs
|
||||
dst_ptr = output_ptr + token_id * output_stride_0 + DIM_NOPE + offs
|
||||
|
||||
data = tl.load(src_ptr, mask=mask).to(tl.float8e4nv)
|
||||
tl.store(dst_ptr, data, mask=mask)
|
||||
|
||||
|
||||
gather_legacy = gather_dequant_requant_fp8_paged_legacy
|
||||
|
||||
PAGE_SIZE = 64
|
||||
DIM_QUANT = 656 # 512 nope fp8 + 16 scale bytes + 128 rope bytes
|
||||
|
||||
@@ -54,7 +54,7 @@ import argparse
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
from sglang.kernels.ops.attention.mla_qk_prep import (
|
||||
absorbed_bmm_concat_cast_q_fp8,
|
||||
concat_and_cast_q_fp8_pad,
|
||||
)
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Tune the gfx950 native MXFP8 dense route for one (N, K) and write its rows into
|
||||
python/sglang/kernels/ops/quantization/mxfp8_gemv_gfx95_configs.json.
|
||||
python/sglang/kernels/ops/gemm/mxfp8_gemv_gfx95_configs.json.
|
||||
|
||||
Two tables: the scaled-MFMA GEMV config per M bucket (1 ... 32) and the dot_scaled tile per
|
||||
M bucket (64 ... 16384). Each candidate is timed by HIP-graph replay rotating over weight
|
||||
@@ -18,7 +18,7 @@ import torch
|
||||
from triton.runtime.errors import OutOfResources
|
||||
|
||||
from sglang.kernels.jit.utils import empty_sentinel
|
||||
from sglang.kernels.ops.quantization import mxfp8_native_amd_gfx95 as native
|
||||
from sglang.kernels.ops.gemm import mxfp8_native_amd_gfx95 as native
|
||||
from sglang.kernels.ops.quantization.mxfp8_amd_gfx95 import fp8_grid_quantize
|
||||
|
||||
GRAPH_CALLS = 20
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Benchmark seeded versus unseeded EAGLE draft Gumbel sampling.
|
||||
|
||||
Example:
|
||||
python benchmark/kernels/sampling/bench_deterministic_fast_sample.py \
|
||||
--batch-sizes 1 8 32 128 --vocab-sizes 32000 128256 163840
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
||||
import torch
|
||||
import triton
|
||||
|
||||
from sglang.srt.speculative.spec_utils import fast_sample
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--batch-sizes", type=int, nargs="+", default=[1, 8, 32, 128])
|
||||
parser.add_argument(
|
||||
"--vocab-sizes", type=int, nargs="+", default=[32000, 128256, 163840]
|
||||
)
|
||||
parser.add_argument("--warmup-ms", type=int, default=100)
|
||||
parser.add_argument("--rep-ms", type=int, default=500)
|
||||
parser.add_argument(
|
||||
"--modes", nargs="+", choices=("eager", "graph"), default=["graph"]
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def capture(fn):
|
||||
fn()
|
||||
torch.cuda.synchronize()
|
||||
graph = torch.cuda.CUDAGraph()
|
||||
with torch.cuda.graph(graph):
|
||||
fn()
|
||||
return graph.replay
|
||||
|
||||
|
||||
def main():
|
||||
args = parse_args()
|
||||
device = torch.device("cuda")
|
||||
print(f"device={torch.cuda.get_device_name(device)}")
|
||||
print("mode\tbatch\tvocab\tunseeded_us\tseeded_us\tslowdown")
|
||||
|
||||
for vocab_size in args.vocab_sizes:
|
||||
for batch_size in args.batch_sizes:
|
||||
logits = torch.randn(
|
||||
(batch_size, vocab_size), device=device, dtype=torch.float32
|
||||
)
|
||||
probs = torch.softmax(logits, dim=-1)
|
||||
seeds = torch.arange(batch_size, device=device, dtype=torch.int64) + 12345
|
||||
positions = (
|
||||
torch.arange(batch_size, device=device, dtype=torch.int64) + 4096
|
||||
)
|
||||
|
||||
def unseeded():
|
||||
return fast_sample(probs)
|
||||
|
||||
def seeded():
|
||||
return fast_sample(
|
||||
probs,
|
||||
sampling_seed=seeds,
|
||||
positions=positions,
|
||||
draft_step=1,
|
||||
)
|
||||
|
||||
# Compile/JIT and populate allocator caches before timing.
|
||||
unseeded()
|
||||
seeded()
|
||||
torch.cuda.synchronize()
|
||||
|
||||
for mode in args.modes:
|
||||
unseeded_fn = capture(unseeded) if mode == "graph" else unseeded
|
||||
seeded_fn = capture(seeded) if mode == "graph" else seeded
|
||||
unseeded_ms = triton.testing.do_bench(
|
||||
unseeded_fn, warmup=args.warmup_ms, rep=args.rep_ms
|
||||
)
|
||||
seeded_ms = triton.testing.do_bench(
|
||||
seeded_fn, warmup=args.warmup_ms, rep=args.rep_ms
|
||||
)
|
||||
print(
|
||||
f"{mode}\t{batch_size}\t{vocab_size}\t"
|
||||
f"{unseeded_ms * 1000:.1f}\t{seeded_ms * 1000:.1f}\t"
|
||||
f"{seeded_ms / unseeded_ms:.2f}x"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -626,6 +626,26 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv pip install --reinstall /tmp/gateway_wheels/*.whl \
|
||||
&& rm -rf /tmp/gateway_wheels
|
||||
|
||||
# sgl-nvidia-mega: the NVIDIA megakernel plugin, inert unless a server passes
|
||||
# --enable-nvidia-megakernel. Installed with CUDA >= 12.9 only (its kernels are
|
||||
# SM103, compiled at runtime); disable with --build-arg ENABLE_SGL_NVIDIA_MEGA=0.
|
||||
ARG ENABLE_SGL_NVIDIA_MEGA=1
|
||||
ARG SGL_NVIDIA_MEGA_REPO="https://github.com/sgl-project/sgl-nvidia-mega.git"
|
||||
ARG SGL_NVIDIA_MEGA_COMMIT="3333e4cfac0f2b15603565d702b4c67538f1628f"
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
if [ "${ENABLE_SGL_NVIDIA_MEGA}" != "1" ]; then \
|
||||
echo "[sgl-nvidia-mega] skip (ENABLE_SGL_NVIDIA_MEGA=${ENABLE_SGL_NVIDIA_MEGA})"; \
|
||||
elif ! printf '%s\n' 12.9 "${CUDA_VERSION}" | sort -V -C; then \
|
||||
echo "[sgl-nvidia-mega] skip (CUDA_VERSION=${CUDA_VERSION} < 12.9)"; \
|
||||
else \
|
||||
git clone "${SGL_NVIDIA_MEGA_REPO}" /sgl-workspace/sgl-nvidia-mega \
|
||||
&& cd /sgl-workspace/sgl-nvidia-mega \
|
||||
&& git checkout -f "${SGL_NVIDIA_MEGA_COMMIT}" \
|
||||
&& uv pip install --no-deps -e . \
|
||||
&& cd / \
|
||||
&& python3 -c "from importlib.metadata import entry_points; assert any(e.name == 'nvidia_megakernel' for e in entry_points(group='sglang.srt.plugins')), 'sgl-nvidia-mega plugin not registered'"; \
|
||||
fi
|
||||
|
||||
# Set workspace directory
|
||||
WORKDIR /sgl-workspace/sglang
|
||||
|
||||
@@ -691,6 +711,7 @@ RUN --mount=type=cache,target=/var/cache/apt,id=runtime-apt \
|
||||
# Runtime libraries
|
||||
libopenmpi3 \
|
||||
libnuma1 \
|
||||
numactl \
|
||||
libibverbs1 \
|
||||
libibumad3 \
|
||||
librdmacm1 \
|
||||
|
||||
+32
-5
@@ -1,13 +1,13 @@
|
||||
# CUDA 13.4 only args
|
||||
# Torch nighly args
|
||||
ARG TORCH_NIGHTLY_INDEX=https://download.pytorch.org/whl/nightly/cu134
|
||||
ARG TORCH_NIGHTLY_VERSION=2.15.0.dev20260818+cu134
|
||||
ARG TORCHVISION_NIGHTLY_VERSION=0.30.0.dev20260819+cu134
|
||||
ARG TORCHAUDIO_NIGHTLY_VERSION=2.11.0.dev20260818+cu134
|
||||
ARG TORCH_NIGHTLY_VERSION=2.15.0.dev20261005+cu134
|
||||
ARG TORCHVISION_NIGHTLY_VERSION=0.30.0.dev20261005+cu134
|
||||
ARG TORCHAUDIO_NIGHTLY_VERSION=2.11.0.dev20261004+cu134
|
||||
# Wheel build args
|
||||
ARG MANYLINUX_IMAGE=pytorch/manylinuxaarch64-builder:cuda13.4
|
||||
ARG DEEPGEMM_REF=v0.2.1.post1
|
||||
ARG SGL_DEEP_GEMM_VERSION=0.2.1.post1
|
||||
ARG DEEPGEMM_REF=v0.2.3.post2
|
||||
ARG SGL_DEEP_GEMM_VERSION=0.2.3.post2
|
||||
ARG DEEPEP_SOURCE_REF=sgl-deepep
|
||||
ARG SGL_DEEP_EP_VERSION=0.1.2.post1
|
||||
|
||||
@@ -39,6 +39,9 @@ ENV DEBIAN_FRONTEND=noninteractive \
|
||||
ENV TRITON_PTXAS_PATH=/usr/local/cuda-13.4/bin/ptxas \
|
||||
TRITON_PTXAS_BLACKWELL_PATH=/usr/local/cuda-13.4/bin/ptxas
|
||||
|
||||
# Turn off torch.distributed's NCCL2 path in the torch nightly.
|
||||
ENV TORCH_DIST_USE_NCCL2=0
|
||||
|
||||
# Add GKE default lib and bin locations
|
||||
ENV PATH="${PATH}:/usr/local/nvidia/bin" \
|
||||
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/nvidia/lib:/usr/local/nvidia/lib64"
|
||||
@@ -877,6 +880,26 @@ RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
python3 -m pip install --force-reinstall /tmp/gateway_wheels/*.whl \
|
||||
&& rm -rf /tmp/gateway_wheels
|
||||
|
||||
# sgl-nvidia-mega: the NVIDIA megakernel plugin, inert unless a server passes
|
||||
# --enable-nvidia-megakernel. Installed with CUDA >= 12.9 only (its kernels are
|
||||
# SM103, compiled at runtime); disable with --build-arg ENABLE_SGL_NVIDIA_MEGA=0.
|
||||
ARG ENABLE_SGL_NVIDIA_MEGA=1
|
||||
ARG SGL_NVIDIA_MEGA_REPO="https://github.com/sgl-project/sgl-nvidia-mega.git"
|
||||
ARG SGL_NVIDIA_MEGA_COMMIT="3333e4cfac0f2b15603565d702b4c67538f1628f"
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
if [ "${ENABLE_SGL_NVIDIA_MEGA}" != "1" ]; then \
|
||||
echo "[sgl-nvidia-mega] skip (ENABLE_SGL_NVIDIA_MEGA=${ENABLE_SGL_NVIDIA_MEGA})"; \
|
||||
elif ! printf '%s\n' 12.9 "${CUDA_VERSION}" | sort -V -C; then \
|
||||
echo "[sgl-nvidia-mega] skip (CUDA_VERSION=${CUDA_VERSION} < 12.9)"; \
|
||||
else \
|
||||
git clone "${SGL_NVIDIA_MEGA_REPO}" /sgl-workspace/sgl-nvidia-mega \
|
||||
&& cd /sgl-workspace/sgl-nvidia-mega \
|
||||
&& git checkout -f "${SGL_NVIDIA_MEGA_COMMIT}" \
|
||||
&& python3 -m pip install --no-deps -e . \
|
||||
&& cd / \
|
||||
&& python3 -c "from importlib.metadata import entry_points; assert any(e.name == 'nvidia_megakernel' for e in entry_points(group='sglang.srt.plugins')), 'sgl-nvidia-mega plugin not registered'"; \
|
||||
fi
|
||||
|
||||
# Set workspace directory
|
||||
WORKDIR /sgl-workspace/sglang
|
||||
|
||||
@@ -927,6 +950,9 @@ ENV DEBIAN_FRONTEND=noninteractive \
|
||||
ENV TRITON_PTXAS_PATH=/usr/local/cuda-13.4/bin/ptxas \
|
||||
TRITON_PTXAS_BLACKWELL_PATH=/usr/local/cuda-13.4/bin/ptxas
|
||||
|
||||
# Turn off torch.distributed's NCCL2 path in the torch nightly.
|
||||
ENV TORCH_DIST_USE_NCCL2=0
|
||||
|
||||
# Add GKE default lib and bin locations + CUDA compiler paths for FlashInfer JIT
|
||||
ENV PATH="${PATH}:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/cuda/nvvm/bin" \
|
||||
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/nvidia/lib:/usr/local/nvidia/lib64"
|
||||
@@ -947,6 +973,7 @@ RUN --mount=type=cache,target=/var/cache/apt,id=runtime-apt \
|
||||
# Runtime libraries
|
||||
libopenmpi3 \
|
||||
libnuma1 \
|
||||
numactl \
|
||||
libibverbs1 \
|
||||
libibumad3 \
|
||||
librdmacm1 \
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
# Beam serving image (CUDA). Build from the repository root; application source
|
||||
# is cloned from upstream:
|
||||
# docker buildx build --platform linux/amd64 --load \
|
||||
# -f docker/beam/beam.Dockerfile -t beam:local-test .
|
||||
FROM lmsysorg/sglang:v0.5.22@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae AS base
|
||||
|
||||
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
|
||||
|
||||
# Replace the stock SGLang with the upstream beam branch. Pass a commit SHA as
|
||||
# BEAM_REF to pin the source of a published image.
|
||||
ARG BEAM_REF=beam
|
||||
WORKDIR /sgl-workspace
|
||||
RUN git init -q /sgl-workspace/sglang-beam \
|
||||
&& git -C /sgl-workspace/sglang-beam fetch -q --depth 1 \
|
||||
https://github.com/sgl-project/sglang.git "${BEAM_REF}" \
|
||||
&& git -C /sgl-workspace/sglang-beam checkout -q --detach FETCH_HEAD \
|
||||
&& uv pip uninstall sglang \
|
||||
&& uv pip install --no-deps -e /sgl-workspace/sglang-beam/python
|
||||
|
||||
# Pins the beam branch adds on top of the v0.5.22 release: OpenAI 2.8.0 for the
|
||||
# Responses shell types and the cuTile compiler for the Beam router GEMM. Record
|
||||
# the new versions so later installs keep them.
|
||||
RUN sed -i -E '/^(openai|nvidia-cuda-tileiras)==/d' /sgl-workspace/constraints.txt \
|
||||
&& uv pip install -c /sgl-workspace/constraints.txt \
|
||||
openai==2.8.0 nvidia-cuda-tileiras==13.4.92 \
|
||||
&& uv pip freeze | grep -E '^(openai|nvidia-cuda-tileiras)==' \
|
||||
>> /sgl-workspace/constraints.txt \
|
||||
&& python3 -c "from importlib.metadata import version; assert version('openai') == '2.8.0'; assert version('nvidia-cuda-tileiras') == '13.4.92'"
|
||||
|
||||
WORKDIR /sgl-workspace/sglang-beam
|
||||
@@ -20,7 +20,7 @@ ARG TORCH_NPU_VERSION="2.10.0.post6"
|
||||
ARG TORCH_NPU_INDEX_URL="https://ascend.devcloud.huaweicloud.com/pypi/simple/"
|
||||
ARG SGLANG_TAG=main
|
||||
ARG ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit
|
||||
ARG SGLANG_KERNEL_NPU_TAG=2026.9.0.post9
|
||||
ARG SGLANG_KERNEL_NPU_TAG=2026.9.0.post10
|
||||
ARG PIP_INSTALL="python3 -m pip install --no-cache-dir"
|
||||
ARG DEVICE_TYPE
|
||||
ARG MODELSCOPE_VERSION=""
|
||||
|
||||
+31
-13
@@ -65,7 +65,7 @@ ENV BUILD_TRITON="0"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
|
||||
# ===============================
|
||||
# Base image 942 with rocm720 and args
|
||||
@@ -75,7 +75,7 @@ ENV BUILD_TRITON="1"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
ENV TRITON_COMMIT_DEFAULT="42270451990532c67e69d753fbd026f28fcc4840"
|
||||
|
||||
# ===============================
|
||||
@@ -86,7 +86,7 @@ ENV BUILD_TRITON="1"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
# Pin the ROCm torch stack for every pip invocation in this flavor. The file is
|
||||
# filled in after the torch 2.11 upgrade below; it must already exist (empty is
|
||||
# valid) because pip reads PIP_CONSTRAINT from the first pip call onwards.
|
||||
@@ -110,7 +110,7 @@ ENV BUILD_TRITON="0"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
|
||||
# ===============================
|
||||
# Base image 950 with rocm720 and args
|
||||
@@ -120,7 +120,7 @@ ENV BUILD_TRITON="1"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
ENV TRITON_COMMIT_DEFAULT="42270451990532c67e69d753fbd026f28fcc4840"
|
||||
|
||||
# ===============================
|
||||
@@ -131,7 +131,7 @@ ENV BUILD_TRITON="1"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
# Pin the ROCm torch stack for every pip invocation in this flavor. The file is
|
||||
# filled in after the torch 2.11 upgrade below; it must already exist (empty is
|
||||
# valid) because pip reads PIP_CONSTRAINT from the first pip call onwards.
|
||||
@@ -308,7 +308,7 @@ ENV BUILD_TRITON="0"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
# Same reasoning as the rocm724 stages: keep pip from resolving the image's
|
||||
# ROCm torch away to a PyPI CUDA build. Populated after the stack is in place.
|
||||
ENV PIP_CONSTRAINT="/etc/sglang/constraints/torch-rocm.txt"
|
||||
@@ -322,7 +322,7 @@ ENV BUILD_TRITON="0"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
ENV AITER_COMMIT_DEFAULT="2bb549102ba2d9e51da8ef21107361827075d8b6"
|
||||
ENV AITER_COMMIT_DEFAULT="285579b6211b0da30f35b91a890c29c823597cd1"
|
||||
ENV PIP_CONSTRAINT="/etc/sglang/constraints/torch-rocm.txt"
|
||||
RUN mkdir -p /etc/sglang/constraints && : > /etc/sglang/constraints/torch-rocm.txt
|
||||
|
||||
@@ -356,7 +356,12 @@ ENV BUILD_VLLM="0"
|
||||
# Keep the Triton that comes with the SDK. The rocm1000 flavor builds a newer
|
||||
# Triton from source, and that one no longer has block pointers or
|
||||
# triton_kernels (triton-lang/triton#10833), which sglang still uses.
|
||||
ENV BUILD_TRITON="0"
|
||||
ENV BUILD_TRITON="1"
|
||||
ENV TRITON_COMMIT_DEFAULT="76940ad348795521b3dc9f6c79acd7309ff924e3"
|
||||
# triton-lang/triton#10833 removed block pointers, which sglang's FLA kernels
|
||||
# (fla/, used by Qwen3-Next, MiniMax and Kimi Linear) still call. The revert is
|
||||
# pure Python: #9668 had already moved block pointers out of C++/MLIR.
|
||||
ENV TRITON_REVERT_COMMITS_DEFAULT="9debefe44117a44c274b05033cfe9e4858883b86"
|
||||
ENV BUILD_LLVM="0"
|
||||
ENV BUILD_AITER_ALL="1"
|
||||
ENV BUILD_MOONCAKE="1"
|
||||
@@ -399,6 +404,8 @@ ARG SETUPTOOLS_SCM_PRETEND_VERSION=""
|
||||
|
||||
ARG TRITON_REPO="https://github.com/triton-lang/triton.git"
|
||||
ENV TRITON_COMMIT="${TRITON_COMMIT:-${TRITON_COMMIT_DEFAULT}}"
|
||||
ARG TRITON_REVERT_COMMITS=""
|
||||
ENV TRITON_REVERT_COMMITS="${TRITON_REVERT_COMMITS:-${TRITON_REVERT_COMMITS_DEFAULT}}"
|
||||
|
||||
# ROCm 7.2 Triton (BUILD_TRITON=1 stages only). Both wheels are the same
|
||||
# upstream revision, triton-lang/triton@89002410. AITER only requires
|
||||
@@ -440,7 +447,8 @@ ARG FHT_COMMIT="46efb7d776d38638fc39f3c803eaee3dd7016bd1"
|
||||
# gfx950); disable with --build-arg ENABLE_SGL_AMD_MONO=0.
|
||||
ARG ENABLE_SGL_AMD_MONO=1
|
||||
ARG SGL_AMD_MONO_REPO="https://github.com/sgl-project/sgl-amd-mono.git"
|
||||
ARG SGL_AMD_MONO_COMMIT="815b104f483d180027054a8f44bad0c2a3f895fd"
|
||||
# empty: the commit pip installed through the monokernels_hip extra
|
||||
ARG SGL_AMD_MONO_COMMIT=""
|
||||
|
||||
ARG ENABLE_MORI=0
|
||||
ARG NIC_BACKEND=none
|
||||
@@ -895,9 +903,11 @@ RUN /bin/bash -lc 'set -euo pipefail; \
|
||||
RUN /bin/bash -lc 'set -euo pipefail; \
|
||||
[ "${ENABLE_SGL_AMD_MONO}" = "1" ] || { echo "[sgl-amd-mono] skip (ENABLE_SGL_AMD_MONO=${ENABLE_SGL_AMD_MONO})"; exit 0; }; \
|
||||
case "${GPU_ARCH}" in gfx950*) ;; *) echo "[sgl-amd-mono] skip (GPU_ARCH=${GPU_ARCH})"; exit 0 ;; esac; \
|
||||
python3 -m pip show sgl-amd-mono > /dev/null || { echo "[sgl-amd-mono] skip (this SGLang does not install it)"; exit 0; }; \
|
||||
commit="${SGL_AMD_MONO_COMMIT:-$(python3 -c "import json, importlib.metadata as md; print(json.loads(md.distribution(\"sgl-amd-mono\").read_text(\"direct_url.json\"))[\"vcs_info\"][\"commit_id\"])")}"; \
|
||||
git clone "${SGL_AMD_MONO_REPO}" /sgl-workspace/sgl-amd-mono; \
|
||||
cd /sgl-workspace/sgl-amd-mono; \
|
||||
git checkout -f "${SGL_AMD_MONO_COMMIT}"; \
|
||||
git checkout -f "${commit}"; \
|
||||
python3 -m pip install --no-cache-dir --no-deps -e .; \
|
||||
python3 -c "from importlib.metadata import entry_points; \
|
||||
assert any(e.name == \"sgl_amd_mono\" for e in entry_points(group=\"sglang.srt.plugins\")), \"sgl-amd-mono plugin not registered\""'
|
||||
@@ -1165,13 +1175,21 @@ RUN cd /tmp/whl \
|
||||
# torch 2.11 names this `triton-rocm`; uninstall it so the pin is the only Triton.
|
||||
RUN if [ "$BUILD_TRITON" = "1" ]; then \
|
||||
case "${GPU_ARCH}" in \
|
||||
gfx1250-rocm1000) \
|
||||
gfx1250-rocm1000|gfx1250-rocm1010) \
|
||||
echo "[Triton] gfx1250: building ${TRITON_COMMIT} from source"; \
|
||||
pip uninstall -y triton triton-rocm || true \
|
||||
&& apt-get update && apt-get install -y --no-install-recommends cmake && rm -rf /var/lib/apt/lists/* \
|
||||
&& git clone ${TRITON_REPO} triton-custom \
|
||||
&& cd triton-custom \
|
||||
&& git checkout ${TRITON_COMMIT} \
|
||||
&& for rev in ${TRITON_REVERT_COMMITS}; do \
|
||||
echo "[Triton] reverting $rev"; \
|
||||
git -c user.email=build@sglang -c user.name=sglang-build \
|
||||
revert --no-commit "$rev" \
|
||||
|| git checkout --ours -- Makefile; \
|
||||
git -c user.email=build@sglang -c user.name=sglang-build \
|
||||
commit -q -am "Revert $rev" || exit 1; \
|
||||
done \
|
||||
&& pip install -r python/requirements.txt \
|
||||
&& pip install -e . \
|
||||
&& if [ -d python/triton_kernels ]; then pip install -e python/triton_kernels --no-deps; fi; \
|
||||
@@ -1187,7 +1205,7 @@ RUN if [ "$BUILD_TRITON" = "1" ]; then \
|
||||
fi
|
||||
|
||||
# torch 2.11 still Requires-Dist: triton-rocm after the swap above.
|
||||
RUN case "${GPU_ARCH}" in *-rocm724) python3 -c "import pathlib,re,importlib.metadata as m; p=pathlib.Path(m.distribution('torch')._path)/'METADATA'; v=m.version('triton'); t,n=re.subn(r'^Requires-Dist: (?:triton|triton-rocm)==[^ ;]+', 'Requires-Dist: triton=='+v, p.read_text(), count=1, flags=re.M); assert n==1, n; p.write_text(t)" ;; esac
|
||||
RUN case "${GPU_ARCH}" in *-rocm724|gfx1250-rocm1010) python3 -c "import pathlib,re,importlib.metadata as m; p=pathlib.Path(m.distribution('torch')._path)/'METADATA'; v=m.version('triton'); t,n=re.subn(r'^Requires-Dist: (?:triton|triton-rocm)==[^\s;]+', 'Requires-Dist: triton=='+v, p.read_text(), count=1, flags=re.M); assert n==1, n; p.write_text(t)" ;; esac
|
||||
|
||||
# -----------------------
|
||||
# Performance environment variable.
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
ARG BASE_IMG=pytorch/manylinux2_28-builder
|
||||
# FlashMLA's kernels use `cvt` with `.scaled::n2::ue8m0`, which needs nvcc >= 13.1,
|
||||
# and older toolkits emit suboptimal instructions for them. Build with 13.4.
|
||||
ARG CUDA_VERSION=13.4
|
||||
|
||||
FROM ${BASE_IMG}:cuda${CUDA_VERSION}
|
||||
|
||||
ARG ARCH=x86_64
|
||||
ARG CUDA_VERSION=13.4
|
||||
ARG PYTHON_VERSION=3.12
|
||||
ARG PYTHON_TAG=cp312-cp312
|
||||
ARG TORCH_VER=2.14.1
|
||||
ARG PIP_DEFAULT_INDEX=https://pypi.python.org/simple
|
||||
ARG PYTORCH_MIRROR=download.pytorch.org
|
||||
|
||||
ENV PYTHON_ROOT_PATH=/opt/python/${PYTHON_TAG}
|
||||
ENV PATH=${PYTHON_ROOT_PATH}/bin:${PATH}
|
||||
|
||||
RUN yum install -y --nogpgcheck git wget tar gcc gcc-c++ make \
|
||||
&& yum clean all && rm -rf /var/cache/yum
|
||||
|
||||
# The build links -lcuda, which on a machine with no driver resolves against the
|
||||
# toolkit's stub.
|
||||
RUN set -eux; \
|
||||
if [ "${ARCH}" = "aarch64" ]; then _LIB=sbsa; else _LIB="${ARCH}"; fi; \
|
||||
mkdir -p /usr/lib/${ARCH}-linux-gnu/; \
|
||||
ln -sf /usr/local/cuda-${CUDA_VERSION}/targets/${_LIB}-linux/lib/stubs/libcuda.so /usr/lib/${ARCH}-linux-gnu/libcuda.so
|
||||
|
||||
# torch is needed at build time (setup.py imports it) and is what the wheel pins.
|
||||
# The CUDA minor version of this torch does not have to match CUDA_VERSION above:
|
||||
# the extensions link libcudart.so.13, whose SONAME carries only the major, and
|
||||
# use no API newer than 13.0.
|
||||
RUN --mount=type=cache,id=sgl-flash-mla-pip,target=/root/.cache/pip \
|
||||
set -eux; \
|
||||
${PYTHON_ROOT_PATH}/bin/pip install torch==${TORCH_VER} --index-url https://${PYTORCH_MIRROR}/whl/cu130; \
|
||||
${PYTHON_ROOT_PATH}/bin/pip install --index-url ${PIP_DEFAULT_INDEX} \
|
||||
ninja setuptools wheel build numpy
|
||||
@@ -12,7 +12,7 @@
|
||||
# manifests → cargo fetch → copy src" approach caches only the fetched
|
||||
# registry; every source change still recompiles every dep.
|
||||
#
|
||||
# `experimental/sgl-router/Cargo.lock` is committed, and every cargo step
|
||||
# `sgl-router/Cargo.lock` is committed, and every cargo step
|
||||
# below runs with `--locked`, so the image ships exactly the dependency
|
||||
# graph CI checked, tested, and license-audited.
|
||||
#
|
||||
@@ -32,9 +32,9 @@ ARG DEBIAN_VERSION=bookworm
|
||||
######################## STAGE 1 — chef recipe ##########################
|
||||
FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS chef
|
||||
RUN cargo install cargo-chef --locked --version ^0.1
|
||||
WORKDIR /work/experimental/sgl-router
|
||||
COPY experimental/sgl-router/Cargo.toml experimental/sgl-router/Cargo.lock ./
|
||||
COPY experimental/sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml
|
||||
WORKDIR /work/sgl-router
|
||||
COPY sgl-router/Cargo.toml sgl-router/Cargo.lock ./
|
||||
COPY sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml
|
||||
# sglang-processor is a path dependency that inherits from the rust/ workspace manifest.
|
||||
COPY rust/Cargo.toml /work/rust/Cargo.toml
|
||||
COPY rust/sglang-api-types /work/rust/sglang-api-types
|
||||
@@ -57,7 +57,7 @@ RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends protobuf-compiler \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& cargo install cargo-chef --locked --version ^0.1
|
||||
WORKDIR /work/experimental/sgl-router
|
||||
WORKDIR /work/sgl-router
|
||||
|
||||
# `dynamo-tokenizers` pulls in `pcre2-sys`, whose build.rs links the SYSTEM
|
||||
# libpcre2-8 whenever pkg-config finds it (it does here — the rust:bookworm
|
||||
@@ -67,8 +67,8 @@ WORKDIR /work/experimental/sgl-router
|
||||
# it statically, keeping the runtime self-contained.
|
||||
ENV PCRE2_SYS_STATIC=1
|
||||
|
||||
COPY --from=chef /work/experimental/sgl-router/recipe.json ./recipe.json
|
||||
COPY experimental/sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml
|
||||
COPY --from=chef /work/sgl-router/recipe.json ./recipe.json
|
||||
COPY sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml
|
||||
# The recipe omits path dependencies outside this workspace, so cook against a
|
||||
# stub of sglang-processor; its real sources arrive with the router's below.
|
||||
COPY rust/Cargo.toml /work/rust/Cargo.toml
|
||||
@@ -82,12 +82,12 @@ RUN mkdir -p /work/rust/sglang-processor/src && touch /work/rust/sglang-processo
|
||||
RUN cargo chef cook --locked --release --recipe-path recipe.json
|
||||
|
||||
# Now bring in the real sources and the manifests they need.
|
||||
COPY experimental/sgl-router/Cargo.toml experimental/sgl-router/Cargo.lock ./
|
||||
COPY experimental/sgl-router/src ./src
|
||||
COPY experimental/sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml
|
||||
COPY experimental/sgl-router/sgl-kv-indexer/build.rs sgl-kv-indexer/build.rs
|
||||
COPY experimental/sgl-router/sgl-kv-indexer/proto sgl-kv-indexer/proto
|
||||
COPY experimental/sgl-router/sgl-kv-indexer/src sgl-kv-indexer/src
|
||||
COPY sgl-router/Cargo.toml sgl-router/Cargo.lock ./
|
||||
COPY sgl-router/src ./src
|
||||
COPY sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml
|
||||
COPY sgl-router/sgl-kv-indexer/build.rs sgl-kv-indexer/build.rs
|
||||
COPY sgl-router/sgl-kv-indexer/proto sgl-kv-indexer/proto
|
||||
COPY sgl-router/sgl-kv-indexer/src sgl-kv-indexer/src
|
||||
COPY rust/sglang-processor/src /work/rust/sglang-processor/src
|
||||
|
||||
RUN touch sgl-kv-indexer/build.rs /work/rust/sglang-processor/src/lib.rs \
|
||||
@@ -97,7 +97,7 @@ RUN touch sgl-kv-indexer/build.rs /work/rust/sglang-processor/src/lib.rs \
|
||||
######################## STAGE 3 — runtime ##############################
|
||||
FROM gcr.io/distroless/cc-debian12:nonroot AS runtime
|
||||
|
||||
COPY --from=builder /work/experimental/sgl-router/target/release/sgl-router /usr/local/bin/sgl-router
|
||||
COPY --from=builder /work/sgl-router/target/release/sgl-router /usr/local/bin/sgl-router
|
||||
|
||||
# Default config path; mount your own via `-v <host-path>:/etc/sgl-router`.
|
||||
ENV SGL_ROUTER_CONFIG=/etc/sgl-router/sgl-router.yaml
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
@@ -1,7 +1,6 @@
|
||||
---
|
||||
title: Ornith-1.0
|
||||
description: "Deploy DeepReinforce Ornith-1.0 with SGLang - a self-improving agentic-coding model family with 397B, 35B, and 9B checkpoints plus FP8 and GGUF variants."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
title: Ornith-1.5
|
||||
description: "Deploy Ornith-1.5 with SGLang on NVIDIA H100 and H200 GPUs, with recipes for 9B, 35B-A3B, 122B, and 397B models, FP8, reasoning, and tool calling."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/ornith-ai/ornith-1.5.jsx";
|
||||
|
||||
## Deployment
|
||||
|
||||
These recipes target a single Linux node with NVIDIA CUDA GPUs. The Ornith-1.5 model cards recommend SGLang `>=0.5.9`. Both parsers are included in every launch command: `qwen3` separates reasoning from the answer, and `qwen3_coder` parses tool calls.
|
||||
|
||||
<Accordion title="Install SGLang">
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Python">
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade "sglang>=0.5.9"
|
||||
```
|
||||
|
||||
Run the Python command from the deployment panel in this environment. See the [installation guide](/docs/get-started/install) for platform-specific requirements.
|
||||
|
||||
</Tab>
|
||||
<Tab title="Docker">
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
Select Docker in the deployment panel to generate the container command.
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Choose your hardware, model size, and checkpoint format. The 9B recipe uses one GPU, 35B-A3B uses two, and 122B and 397B use eight. BF16 397B requires H200 in this matrix; its FP8 checkpoint is also available on H100. Select **BF16** for a BF16 checkpoint or **FP8** for an FP8 checkpoint. All commands use Hugging Face repository IDs and default to port **30000**.
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
## 1. Model introduction
|
||||
|
||||
Ornith-1.5 is an agentic model family published by [Ornith AI](https://huggingface.co/ornith-ai). The 9B, 35B-A3B, 122B, and 397B checkpoints use Qwen3.5 architectures. The family extends Ornith-1.0's self-improvement loop by jointly optimizing task generation, scaffold construction, and solution rollouts through reinforcement learning. The model cards describe applications in coding, reasoning, and tool-using agents.
|
||||
|
||||
The recipes expose an OpenAI-compatible API with separate reasoning output, parsed tool calls, and a 262,144-token context window. The table lists the checkpoints covered by this recipe.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse"}}>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px"}}>Checkpoint</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px"}}>Format</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px"}}>Recipe</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/ornith-ai/Ornith-1.5-9B">Ornith-1.5-9B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>Dense; 1× H100 or H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B">Ornith-1.5-35B-A3B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>MoE, ~3B active; 2× H100 or H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-FP8">Ornith-1.5-35B-A3B-FP8</a></td>
|
||||
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||
<td style={{padding: "9px 12px"}}>2× H100 or H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/ornith-ai/Ornith-1.5-122B">Ornith-1.5-122B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>MoE, 256 experts with 8 routed per token; 8× H100 or H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/ornith-ai/Ornith-1.5-397B">Ornith-1.5-397B</a></td>
|
||||
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||
<td style={{padding: "9px 12px"}}>MoE; 8× H200</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px"}}><a href="https://huggingface.co/ornith-ai/Ornith-1.5-397B-FP8">Ornith-1.5-397B-FP8</a></td>
|
||||
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||
<td style={{padding: "9px 12px"}}>8× H100 or H200</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
The official [9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B), [35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B), [122B](https://huggingface.co/ornith-ai/Ornith-1.5-122B), and [397B](https://huggingface.co/ornith-ai/Ornith-1.5-397B) model cards declare the MIT license.
|
||||
|
||||
**Resources:** [Ornith-1.5 model repositories](https://huggingface.co/ornith-ai/models?search=Ornith-1.5) · [Ornith-1.5 blog](https://ornith.ai/ornith_1_5.html)
|
||||
|
||||
## 2. Configuration tips
|
||||
|
||||
- **Model IDs:** use the `ornith-ai` namespace. The 35B repository includes the `-A3B` suffix. FP8 selections use the actual `-FP8` repository rather than the BF16 repository shown in their model-card quickstarts.
|
||||
- **Context and memory:** every recipe uses `--context-length 262144`. The 9B, 35B-A3B, and 397B recipes use `--mem-fraction-static 0.85`; the 122B recipe uses `0.80`. Lower the context length if you need more memory headroom.
|
||||
- **Reasoning:** `--reasoning-parser qwen3` exposes reasoning as `message.reasoning_content`, with the answer in `message.content`.
|
||||
- **Tool calling:** `--tool-call-parser qwen3_coder` exposes parsed calls in `message.tool_calls`. Your application executes the requested function and submits its result in a follow-up tool message.
|
||||
- **Sampling:** 35B-A3B and 397B recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20`. For 9B and 122B, the model cards recommend `temperature=1.0` and `presence_penalty=1.5` for general tasks, or `temperature=0.6` and `presence_penalty=0.0` for precise coding; both use `top_p=0.95`, `top_k=20`, `min_p=0.0`, and `repetition_penalty=1.0`. These settings are distinct from benchmark-specific settings. The Python examples below use the precise-coding sampling settings.
|
||||
|
||||
## 3. Usage examples
|
||||
|
||||
These requests use the precise-coding sampling settings shared by 9B and 122B. Set `model` to the deployment panel's `--served-model-name`, such as `Ornith-1.5-9B` or `Ornith-1.5-122B`. The examples use the default port; match the client port if you change the launch port.
|
||||
|
||||
<Accordion title="Chat completion with reasoning (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="Ornith-1.5-9B",
|
||||
messages=[{"role": "user", "content": "Write a compact Python is_prime function."}],
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
presence_penalty=0.0,
|
||||
max_tokens=1024,
|
||||
extra_body={"top_k": 20, "min_p": 0.0, "repetition_penalty": 1.0},
|
||||
)
|
||||
message = response.choices[0].message
|
||||
print("Reasoning:", getattr(message, "reasoning_content", None))
|
||||
print("Answer:", message.content)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Tool-call request (Python)">
|
||||
|
||||
```python Example
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="Ornith-1.5-9B",
|
||||
messages=[{"role": "user", "content": "Run the parser module's unit tests."}],
|
||||
tools=[{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run_tests",
|
||||
"description": "Run the project's test suite.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"target": {"type": "string"}},
|
||||
"required": ["target"],
|
||||
},
|
||||
},
|
||||
}],
|
||||
tool_choice="auto",
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
presence_penalty=0.0,
|
||||
max_tokens=2048,
|
||||
extra_body={"top_k": 20, "min_p": 0.0, "repetition_penalty": 1.0},
|
||||
)
|
||||
message = response.choices[0].message
|
||||
print("Reasoning:", getattr(message, "reasoning_content", None))
|
||||
print("Tool calls:", message.tool_calls)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
### Agent integration
|
||||
|
||||
Configure your agent or coding CLI to use the same OpenAI-compatible endpoint:
|
||||
|
||||
- **API base URL:** `http://localhost:30000/v1`.
|
||||
- **API key:** `EMPTY` for a local server without API-key authentication.
|
||||
- **Model:** the selected checkpoint's `--served-model-name`, such as `Ornith-1.5-122B`.
|
||||
|
||||
Use the served model name rather than the Hugging Face repository ID when the two differ. Keep both parsers enabled so the agent receives separate reasoning and structured tool calls. Your agent executes tools and returns their results to the server in subsequent messages.
|
||||
@@ -26,8 +26,6 @@ For more details, please refer to the [official DeepSeek-OCR repository](https:/
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
@@ -37,8 +37,6 @@ For more details, please refer to the [official DeepSeek-R1 repository](https://
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides deployment configurations optimized for different hardware platforms and use cases.
|
||||
|
||||
@@ -22,8 +22,6 @@ SGLang offers multiple installation methods. You can choose the most suitable in
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
@@ -18,8 +18,6 @@ SGLang offers multiple installation methods. You can choose the most suitable in
|
||||
|
||||
Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.
|
||||
|
||||
For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).
|
||||
|
||||
## 3. Model Deployment
|
||||
|
||||
This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.
|
||||
|
||||
@@ -51,11 +51,11 @@ docker run --gpus all \
|
||||
AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
|
||||
|
||||
For example:
|
||||
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.21-rocm720-mi35x-20261007`
|
||||
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.21-rocm720-mi30x-20261007`
|
||||
- **MI355X** → `lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261016`
|
||||
- **MI300X** → `lmsysorg/sglang-rocm:v0.5.22-rocm720-mi30x-20261016`
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.21-rocm720-{mi35x,mi30x}-20261007
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.22-rocm720-{mi35x,mi30x}-20261016
|
||||
|
||||
docker run \
|
||||
--device=/dev/kfd --device=/dev/dri \
|
||||
@@ -65,7 +65,7 @@ docker run \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
lmsysorg/sglang-rocm:v0.5.21-rocm720-{mi35x,mi30x}-20261007 \
|
||||
lmsysorg/sglang-rocm:v0.5.22-rocm720-{mi35x,mi30x}-20261016 \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
@@ -77,7 +77,7 @@ docker run \
|
||||
|
||||
For **MI355X → Pro / Pro Official → FP4 → Multi-Nodes**, the panel generates a **1P1D deployment**: one prefill node and one decode node, each running an independent worker. Select **Router**, **Prefill**, or **Decode** to copy its command. Start the [UMBP tier](#3-8-unified-cache-external-linker-umbp) on the prefill node, then both workers, then the router. Send client requests to the router on port `8000`.
|
||||
|
||||
The MI355X multi-node strategies follow the [InferenceX AgentX submission](https://github.com/SemiAnalysisAI/InferenceX/pull/3664): **Low-Latency**, **Balanced**, and **High-Throughput** use fixed concurrency presets of **16**, **48**, and **256**, respectively. **Pro** uses EAGLE/MTP; **Pro Official (0813)** uses DSpark, matching the InferenceX checkpoint. Both enable UMBP on prefill by default. The Playground inherits the selected worker, including its speculative decoder and cache settings.
|
||||
The MI355X multi-node strategies **Low-Latency**, **Balanced**, and **High-Throughput** use fixed concurrency presets of **16**, **48**, and **256**, respectively. **Pro** uses EAGLE/MTP; **Pro Official (0813)** uses DSpark. Both enable UMBP on prefill by default. The Playground inherits the selected worker, including its speculative decoder and cache settings.
|
||||
|
||||
Run these recipes' **Python** commands inside the ROCm image shown in the **Docker** tab, where Aiter is already enabled. Add explicit parser and network-device settings through the Playground's **Parsers** and **PD Disagg → IB Device** controls.
|
||||
|
||||
@@ -773,13 +773,13 @@ Four adjustments are not in the commands below. Add them when they apply:
|
||||
- `--cuda-graph-backend-prefill breakable` at concurrency 1–4.
|
||||
- `--enable-prefill-cp --cp-strategy interleave --enable-cp-decode-attn-tp` on the TP-only commands at concurrency 16 and above. Prefill context parallelism splits each prompt across all eight ranks (CP size = TP size).
|
||||
- `SGLANG_DSV4_GROUPED_ASM=1` on the TP-only commands at concurrency 16 and above — the kernel needs at least 16 requests per draft token.
|
||||
- `--prefill-decode-interval 15` on the MegaMoE command at concurrency 384, replacing the `20` it ships with.
|
||||
- At concurrency 384 on the MegaMoE command: `--prefill-decode-interval 20`, `SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR=1024`, and `AITER_MEGA_FIXED_SLOT_MAX_MTPR=2047`.
|
||||
|
||||
**TP8, concurrency 1–16:**
|
||||
```bash Command
|
||||
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
|
||||
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
|
||||
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
|
||||
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
|
||||
SGLANG_USE_ROCM700A=0 \
|
||||
TORCH_BLAS_PREFER_HIPBLASLT=1 \
|
||||
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
|
||||
@@ -819,8 +819,8 @@ python3 -m sglang.launch_server \
|
||||
**DP8 (DP Attention), concurrency 128:**
|
||||
```bash Command
|
||||
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
|
||||
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
|
||||
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
|
||||
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
|
||||
SGLANG_USE_ROCM700A=0 \
|
||||
TORCH_BLAS_PREFER_HIPBLASLT=1 \
|
||||
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
|
||||
@@ -851,7 +851,7 @@ python3 -m sglang.launch_server \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--enforce-shared-experts-fusion \
|
||||
--mem-fraction-static 0.92 \
|
||||
--chunked-prefill-size 65536 \
|
||||
--chunked-prefill-size 131072 \
|
||||
--tool-call-parser deepseekv4 \
|
||||
--reasoning-parser deepseek-v4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
@@ -864,13 +864,13 @@ python3 -m sglang.launch_server \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
|
||||
`--chunked-prefill-size` is a global budget divided by `--attn-dp-size`, so 65536 keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware`, so a multi-turn session lands on the rank already holding its prefix; above concurrency 160 add `--balance-abs-threshold 32`. Both apply to the MegaMoE command below as well.
|
||||
`--chunked-prefill-size` is a global budget divided by `--attn-dp-size`, so 131072 keeps 16384 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware`, so a multi-turn session lands on the rank already holding its prefix; above concurrency 160 add `--balance-abs-threshold 32`. Both apply to the MegaMoE command below as well.
|
||||
|
||||
**DP8 EP8 MegaMoE, concurrency 128–384:**
|
||||
```bash Command
|
||||
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
|
||||
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
|
||||
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
|
||||
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
|
||||
SGLANG_USE_ROCM700A=0 \
|
||||
TORCH_BLAS_PREFER_HIPBLASLT=1 \
|
||||
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
|
||||
@@ -878,12 +878,15 @@ AITER_BF16_FP8_MOE_BOUND=0 \
|
||||
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
|
||||
SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \
|
||||
SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \
|
||||
SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR=8192 \
|
||||
SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR=16384 \
|
||||
SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR=512 \
|
||||
AITER_MEGA_FIXED_SLOT_MAX_MTPR=1023 \
|
||||
SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \
|
||||
MORI_SHMEM_HEAP_SIZE=17179869184 \
|
||||
SGLANG_AITER_MEGA_RANK_SYNC=1 \
|
||||
SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1 \
|
||||
SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1 \
|
||||
SGLANG_EPLB_P2P_BATCH_CHUNK_SIZE=7 \
|
||||
SGLANG_DP_USE_REDUCE_SCATTER=0 \
|
||||
GPU_MAX_HW_QUEUES=5 \
|
||||
python3 -m sglang.launch_server \
|
||||
@@ -897,7 +900,7 @@ python3 -m sglang.launch_server \
|
||||
--enable-dp-attention-local-control-broadcast \
|
||||
--tokenizer-worker-num 8 \
|
||||
--stream-interval 20 \
|
||||
--prefill-decode-interval 20 \
|
||||
--prefill-decode-interval 40 \
|
||||
--prefill-delayer-token-usage-low-watermark 0.7 \
|
||||
--load-balance-method total_requests \
|
||||
--attention-backend dsv4 \
|
||||
@@ -912,7 +915,7 @@ python3 -m sglang.launch_server \
|
||||
--eplb-rebalance-num-iterations 200 \
|
||||
--expert-distribution-recorder-mode stat \
|
||||
--mem-fraction-static 0.85 \
|
||||
--chunked-prefill-size 65536 \
|
||||
--chunked-prefill-size 131072 \
|
||||
--tool-call-parser deepseekv4 \
|
||||
--reasoning-parser deepseek-v4 \
|
||||
--speculative-algorithm DSPARK \
|
||||
@@ -925,7 +928,7 @@ python3 -m sglang.launch_server \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
|
||||
This command adds expert parallelism on top of the DP8 one. [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io) explains `SGLANG_AMD_USE_FLYDSL_MEGA_MOE`, `SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR` and the MORI symmetric heap; the heap is 16 GiB here rather than 8 GiB because `--mem-fraction-static 0.85` leaves more GPU memory free. `SGLANG_DP_USE_REDUCE_SCATTER=0` turns off the reduce-scatter path that MegaMoE's all-to-all replaces. The three `SGLANG_AITER_MEGA_*` variables and `--enable-eplb` enable expert-load balancing, recorded during prefill only.
|
||||
This command adds expert parallelism on top of the DP8 one. [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io) explains `SGLANG_AMD_USE_FLYDSL_MEGA_MOE`, `SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR` and the MORI symmetric heap; the heap is 16 GiB here rather than 8 GiB because `--mem-fraction-static 0.85` leaves more GPU memory free. `SGLANG_DP_USE_REDUCE_SCATTER=0` turns off the reduce-scatter path that MegaMoE's all-to-all replaces. The three `SGLANG_AITER_MEGA_*` variables and `--enable-eplb` enable expert-load balancing, recorded during prefill only. `SGLANG_EPLB_P2P_BATCH_CHUNK_SIZE=7` keeps each RCCL P2P group under 31 ops during EPLB rebalance.
|
||||
|
||||
### 3.7 PD Disaggregation on MI355X (MORI-IO)
|
||||
|
||||
@@ -944,17 +947,17 @@ docker run \
|
||||
--ulimit nofile=1048576:1048576 \
|
||||
--network host --ipc=host --shm-size 32g \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
lmsysorg/sglang-rocm:v0.5.21-rocm724-mi35x-20261001 \
|
||||
lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261019 \
|
||||
python3 -m sglang.launch_server <role args below>
|
||||
```
|
||||
|
||||
The Deployment and Playground Docker outputs retain this `v0.5.21` base image for the MI355X Pro and Pro Official FP4 multi-node recipes. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts.
|
||||
Use this image for the **MI355X Pro / Pro Official FP4 multi-node** workers and router, including the additional concurrency points below. The Deployment and Playground Docker outputs select it for all three PD roles. It supports the documented `SGLANG_DSV4_DENSE_MXSCALE` and `SGLANG_ROUTER_P2C_LOAD_METRIC` settings. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts.
|
||||
|
||||
For the **MI355X Pro / Pro Official FP4** presets, **High-Throughput** sets `MORI_IO_QP_MAX_SEND_WR=32767` and `MORI_IO_QP_MAX_CQE=32768` on both MORI roles. Low-Latency and Balanced use the transport's default queue sizes. The send-queue backoff timeout uses MORI's default of 5 seconds.
|
||||
|
||||
#### Role-based configuration
|
||||
|
||||
Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. The commands use the [submitted recipe at `df5e433239d7`](https://github.com/SemiAnalysisAI/InferenceX/blob/df5e433239d72106529ece4dd97cc36e4c3adf4c/inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/dsv4/sglang/mi355x-fp4/agentx/disagg-umbp-dspark.yaml), with cluster-specific paths replaced by local placeholders and benchmark acceptance simulation removed. The Pro Official commands additionally apply the kernel and routing updates described below.
|
||||
Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. Replace the host and model-path placeholders with your deployment's values. The Pro Official commands use the kernel and routing settings described below.
|
||||
|
||||
| Strategy | Concurrency | Prefill → Decode | `--max-running-requests` | DSpark block size |
|
||||
|---|---|---|---|---|
|
||||
@@ -975,7 +978,7 @@ Selecting a TP-only role forces **DP-Attention** off and greys the control. `--m
|
||||
</Note>
|
||||
|
||||
<Warning>
|
||||
**Speculative decoding changes how you size `--max-running-requests`.** For a target concurrency of N, set it to `N*2` on **both** roles; at N the served batch stays below the concurrency you are aiming for. This holds for DSpark and EAGLE/MTP alike. The table above follows it: 32 serves a concurrency of 16, 96 serves 48, and 512 serves the high-throughput concurrency of 256.
|
||||
**Speculative decoding changes how you size `--max-running-requests`.** For the strategy presets above, a target concurrency of N uses `N*2` on **both** roles; at N the served batch stays below the concurrency you are aiming for. This holds for DSpark and EAGLE/MTP alike. The table above follows it: 32 serves a concurrency of 16, 96 serves 48, and 512 serves the high-throughput concurrency of 256. The Pro Official c1 override below is an exception: retain 8 request slots on each role even though client concurrency is 1.
|
||||
|
||||
The decode role's `--cuda-graph-bs-decode` then has to cover `N*2 / attn_dp_size`, the per-rank batch. The strategy presets apply both rules: decode graphs cover 1–32, 1–96, and 1–64, respectively.
|
||||
</Warning>
|
||||
@@ -1137,7 +1140,7 @@ The multi-node router presets retain the default request timeout. The standalone
|
||||
|
||||
The panel keeps its c16 / c48 / c256 strategy presets. Apply the following overrides to the **Multi-Nodes** commands to reproduce the optimized recipe's endpoints. Here `cN` means a target concurrency of N.
|
||||
|
||||
**c1 (replaces c4):** start from Low-Latency. Keep prefill TP4, set decode to `--tp 4`, and set `--max-running-requests 2` on both roles. Replace decode's graph list with `--cuda-graph-bs-decode 1 2`. Keep DSpark block size 6 (7 draft tokens), `MORI_MAX_DISPATCH_TOKENS_DECODE=448`, and both kernel switches on both roles. The source also sets `SGLANG_MLA_HCA_KV_SPLITS=8` on both roles; the FP8 KV path does not read this setting. Benchmark with concurrency 1.
|
||||
**c1 (replaces c4):** start from Low-Latency. Keep prefill TP4, set decode to `--tp 4`, and set `--max-running-requests 8` on both roles. Replace decode's graph list with `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8`. Keep DSpark block size 6 (7 draft tokens), `MORI_MAX_DISPATCH_TOKENS_DECODE=448`, and both kernel switches on both roles. The source also sets `SGLANG_MLA_HCA_KV_SPLITS=8` on both roles; the FP8 KV path does not read this setting. Keep these 8 server slots and graph sizes when benchmarking with client concurrency 1; do not reduce them to 2 using the strategy presets' `N*2` rule.
|
||||
|
||||
**c32:** start from Low-Latency, keep TP4 → TP8 and block size 6, set both admission limits to 64, and capture decode graph sizes 1–64. Both roles use FP8 KV and mxscale, as at c16 and c48.
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ docker run --gpus all \
|
||||
**AMD GPUs (ROCm)** — AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 10 version.
|
||||
|
||||
```bash Command
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261016
|
||||
docker pull lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261021
|
||||
|
||||
docker run \
|
||||
--device=/dev/kfd --device=/dev/dri \
|
||||
@@ -52,7 +52,7 @@ docker run \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<your-hf-token>" \
|
||||
lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261016 \
|
||||
lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261021 \
|
||||
sglang serve <use args below>
|
||||
```
|
||||
|
||||
@@ -105,7 +105,7 @@ import { agentx } from "/src/snippets/agentx/deepseek-ai/deepseek-v4_1.jsx";
|
||||
|
||||
**Engram.** An additive n-gram hash memory at two layers. Token ids are normalized *before* hashing, so `" The"`, `"the"` and `"THE"` cannot fork into separate rows. Its two fp8 tables are the largest single block of weight in the checkpoint, and by default they load row-sharded across the TP group, which costs an all-reduce per engram layer. `SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1` (**opt-in**) moves them to host memory instead: the freed HBM goes to the KV pool and output is bitwise unchanged — at the cost of host RAM, a longer load, and needing huge-page backing to keep the gather cheap. `SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT` trades the two all-reduces for page size: `shared` (the default) maps a single copy all ranks read and needs no reduce; `per_rank` splits that same table into one shard per rank and keeps the reduce, but is the only one the kernel reliably backs with 2 MiB pages.
|
||||
|
||||
**SWA bounded replay.** Decoder SWA bounded replay runs the early layers over the whole extend and the late layers over only each request’s last 128 tokens, sharing late-layer KV from source layer 20. It defaults on for supported standalone CUDA/HIP configurations without context parallelism. Encoder SWA replay can be combined with decoder replay, subject to encoder-specific validation. Eager and breakable prefill are supported; full prefill graphs keep it disabled. With DP attention, automatic enablement requires attention TP 1 and the `none` or `megamoe` MoE backend on CUDA, or `none` on HIP. Use `--enable-decoder-swa-bounded-replay` to explicitly enable replay, subject to compatibility validation. Bounded replay is not numerically equivalent to full prefill. A request for prompt logprobs or for every hidden state runs its whole prompt through the late layers with full sliding windows; other requests, including their output-token logprobs, use the bounded computation. Use `--no-enable-decoder-swa-bounded-replay` to retain full prefill, for example for RL rollouts whose logprobs must match the trainer.
|
||||
**SWA bounded replay.** Decoder SWA bounded replay runs the early layers over the whole extend and the late layers over only each request’s last 128 tokens, sharing late-layer KV from source layer 20. It defaults on for supported CUDA/HIP configurations without context parallelism, including both PD roles on CUDA. Encoder SWA replay can be combined with decoder replay, subject to encoder-specific validation. Eager and breakable prefill are supported; full prefill graphs keep it disabled. With DP attention, automatic enablement requires attention TP 1 and the `none` or `megamoe` MoE backend on CUDA, or `none` on HIP. Use `--enable-decoder-swa-bounded-replay` to explicitly enable replay, subject to compatibility validation. Bounded replay is not numerically equivalent to full prefill. A request for prompt logprobs or for every hidden state runs its whole prompt through the late layers with full sliding windows; other requests, including their output-token logprobs, use the bounded computation. Use `--no-enable-decoder-swa-bounded-replay` to retain full prefill, for example for RL rollouts whose logprobs must match the trainer.
|
||||
|
||||
**Kernels.** At decode token counts these kernels are launch-bound rather than bandwidth-bound, so the work is fusion and overlap: the compressor projection, ratio-2 pooling, RoPE with fp4 quantization and indexer packing, paged fp4 indexer scoring with top-k resolved straight to cache addresses, and the mHC statistics and Sinkhorn — with compression, indexing and mHC overlapped against attention and FFN on dedicated streams. All of it is selected automatically; the environment switches it once sat behind were removed, so there is nothing to turn on. One caveat outranks the fusions: **output is not bitwise stable across batch composition today** — two default kernels are shape-guarded to a single token — and `--enable-deterministic-inference` is refused on this backend.
|
||||
|
||||
@@ -202,6 +202,8 @@ What to expect:
|
||||
|
||||
Prefill/decode disaggregation is validated token-identical against a single server across 32 greedy prompts, with GSM8K matching through the router. Use the **PD Disaggregation** card in the Playground to generate the prefill role, the decode role and the router command.
|
||||
|
||||
Both roles run MegaAttention (`--dsv4-attn-backend flashmla_mega`) by default on SM100/SM103, as a single server does. Prefill and decode must store KV in the same layout, so they must agree on `--dsv4-attn-backend` and `SGLANG_DSV4_KV_LAYOUT`; a decode server paired with a prefill whose layout differs fails its first request with a `KV layout mismatch` error. A decode on a GPU other than SM100/SM103 therefore needs `--dsv4-attn-backend flashmla` on the prefill too.
|
||||
|
||||
One deployment note: Mooncake needs the RDMA fabric visible inside the container, so launch with `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1`. Without it Mooncake selects its NVLink transport, which only serves buffers from its own allocator and fails with `Requested address ... not found`. If you hit that, force TCP with `MOONCAKE_PROTOCOL=tcp` and `MC_FORCE_TCP=1` — the Playground's Mooncake option sets both.
|
||||
|
||||
PD runs with DSpark: on a DSpark recipe both Playground roles keep the speculative flags, and the prefill and decode servers must both pass them. The server accepts this with Mooncake or NIXL transfer, static verification (`SGLANG_RAGGED_VERIFY_MODE`, `static` by default), and no context parallelism on decode (CP=1, DCP=1); it rejects any other combination at startup.
|
||||
@@ -222,14 +224,14 @@ Only the L2 tier is exposed here. For the storage (L3) tier and the canonical fl
|
||||
|
||||
Agentic replay is a different workload: long-context prompts, multi-turn sessions that reuse their own prefix across turns, and subagent branches that fan out within one session. Concurrency below counts live sessions, not requests. Every command keeps the engram tables on the host, and none adds a HiCache tier. The sessions fit the device pool.
|
||||
|
||||
Two commands follow, one per concurrency range. TP2 covers concurrency 1 to 32 and TP4 covers 1 to 16; at 32 and 64 the same two GPUs also run DP attention with TP MoE.
|
||||
TP2 covers concurrency 1 to 32 and TP4 covers 1 to 16; at 32 and 64 the same two GPUs also run DP attention with TP MoE.
|
||||
|
||||
**TP2 or TP4, concurrency 1–32:**
|
||||
**TP2 or TP4, concurrency 1–16:**
|
||||
|
||||
- TP2 raises `--mem-fraction-static` from concurrency 16 and continues through 32; TP4 stops at 16 and holds 0.70, with a quarter of the checkpoint per rank leaving room to spare.
|
||||
- TP2 raises `--mem-fraction-static` at concurrency 16; TP4 stops at 16 and holds 0.70, with a quarter of the checkpoint per rank leaving room to spare.
|
||||
- A longer `--prefill-decode-interval` runs more decode rounds between prefill batches, trading TTFT for P90 inter-token latency.
|
||||
- Up to concurrency 8 it pairs with `--schedule-policy lpm`, which admits the requests with the longest cached prefix first.
|
||||
- Concurrency ≤ 16 uses the same low-latency path as §3's Low-Latency card (`--monokernel-backend amd` and `--disable-shared-experts-fusion`, as in the command below); at concurrency 32 swap those two flags for `--enforce-shared-experts-fusion`.
|
||||
- These points set `--monokernel-backend amd`. Do not set `SGLANG_HIP_DSV41_INDEX_K_PAGE8`: the monokernel refuses that layout.
|
||||
|
||||
The braced values in the command follow concurrency:
|
||||
|
||||
@@ -250,13 +252,11 @@ SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
|
||||
SGLANG_USE_AITER=1 \
|
||||
SGLANG_USE_ROCM700A=0 \
|
||||
TORCH_BLAS_PREFER_HIPBLASLT=1 \
|
||||
AITER_BF16_FP8_MOE_BOUND=0 \
|
||||
TRITON_HIP_USE_ASYNC_COPY=0 \
|
||||
ROCM_QUICK_REDUCE_QUANTIZATION=INT8 \
|
||||
SGLANG_OPT_HIP_OPUS_SPARSE_PREFILL=true \
|
||||
SGLANG_OPT_HIP_FLYDSL_MXFP8=1 \
|
||||
SGLANG_HIP_FP4_INDEXER_ROWGROUP=1 \
|
||||
SGLANG_HIP_DSV41_INDEX_K_PAGE8=1 \
|
||||
HSA_NO_SCRATCH_RECLAIM=0 \
|
||||
GPU_MAX_HW_QUEUES=2 \
|
||||
sglang serve \
|
||||
@@ -267,8 +267,9 @@ sglang serve \
|
||||
--speculative-algorithm DSPARK \
|
||||
--speculative-dspark-block-size 5 \
|
||||
--monokernel-backend amd \
|
||||
--disable-shared-experts-fusion \
|
||||
--enforce-shared-experts-fusion \
|
||||
--fp8-gemm-backend aiter \
|
||||
--enable-decoder-swa-bounded-replay \
|
||||
--cuda-graph-backend-prefill disabled \
|
||||
--mem-fraction-static {mem-fraction-static} \
|
||||
--chunked-prefill-size 16384 \
|
||||
@@ -282,6 +283,8 @@ sglang serve \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
|
||||
**TP2, concurrency 32:** the command above with the concurrency-32 column, without `--monokernel-backend amd`, and with `SGLANG_HIP_DSV41_INDEX_K_PAGE8=1`.
|
||||
|
||||
**DP2, concurrency 32–64:** two DP attention ranks, one per GPU, with the MoE still tensor parallel across both. The braced values follow concurrency:
|
||||
|
||||
| Concurrency | 32 | 64 |
|
||||
@@ -298,7 +301,6 @@ SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
|
||||
SGLANG_USE_AITER=1 \
|
||||
SGLANG_USE_ROCM700A=0 \
|
||||
TORCH_BLAS_PREFER_HIPBLASLT=1 \
|
||||
AITER_BF16_FP8_MOE_BOUND=0 \
|
||||
TRITON_HIP_USE_ASYNC_COPY=0 \
|
||||
ROCM_QUICK_REDUCE_QUANTIZATION=INT8 \
|
||||
SGLANG_OPT_HIP_OPUS_SPARSE_PREFILL=true \
|
||||
@@ -327,6 +329,7 @@ sglang serve \
|
||||
--speculative-dspark-block-size 5 \
|
||||
--enforce-shared-experts-fusion \
|
||||
--fp8-gemm-backend aiter \
|
||||
--enable-decoder-swa-bounded-replay \
|
||||
--cuda-graph-backend-prefill disabled \
|
||||
--mem-fraction-static {mem-fraction-static} \
|
||||
--chunked-prefill-size 32768 \
|
||||
|
||||
@@ -15,7 +15,7 @@ ERNIE-4.5 delivers advanced features as below:
|
||||
- **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing.
|
||||
- **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations.
|
||||
- **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions.
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs.
|
||||
- **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, MI355X GPUs and Intel Xeon CPUs.
|
||||
|
||||
## 2. SGLang Installation
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: GLM-5.2
|
||||
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X."
|
||||
description: "Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X, plus Ascend NPU recipes."
|
||||
---
|
||||
|
||||
## Deployment
|
||||
@@ -39,6 +39,55 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Install and prepare Ascend NPUs">
|
||||
|
||||
Use a Linux host with the Ascend driver and a compatible CANN / TorchNPU environment. Follow the
|
||||
[Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) and the
|
||||
[GLM-5.2 deployment tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2#installation) to start a container with the NPU devices, driver mounts, and checkpoint directory. Run the setup and the generated **Python** command inside that container.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="A3">
|
||||
|
||||
```bash Command
|
||||
docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16
|
||||
```
|
||||
|
||||
Run the following setup before copying the generated command (CANN 9.0.0). CPU tuning requires host privileges.
|
||||
|
||||
```bash Command
|
||||
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="950PR/DT series">
|
||||
|
||||
Use the compatible 950PR/DT series SGLang checkout, CANN environment, and custom operators used for your deployment. Set `SGLANG_ROOT` to that checkout; these recipes do not prescribe an image tag.
|
||||
|
||||
```bash Command
|
||||
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
unset HCCL_IF_IP HCCL_SOCKET_FAMILY RANK_TABLE_FILE
|
||||
SGLANG_ROOT=/path/to/sglang
|
||||
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
Replace the model path in the generated command with your mounted ModelSlim checkpoint directory. Include its configuration, tokenizer, quantization metadata, and MTP weights.
|
||||
|
||||
For A3, use the locally prepared **GLM-5.2-w8a8** checkpoint described in the [model-weight prerequisites](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2#model-weights).
|
||||
For 950PR/DT series, use your locally prepared GLM-5.2 **W8A8C8_MXFP8** checkpoint, including MTP weights, and replace the model path in the generated command with its mounted path.
|
||||
|
||||
Replace network placeholders such as `<network-interface>`, `<prefill-head-ip>`, and `<decode-head-ip>` in the generated NPU command with your cluster settings.
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: GLM-5.3-Flash
|
||||
description: "Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, GB300, and MI355X, with MTP and multimodal serving."
|
||||
description: "Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, GB300, MI355X, and TPU7x, with MTP and multimodal serving, plus Ascend A3 and 950PR/DT series NPU recipes."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
@@ -20,6 +20,67 @@ For the listed NVIDIA hardware, the deployment panel can render a complete `dock
|
||||
|
||||
For MI355X, follow the [AMD GPU installation guide](/docs/hardware-platforms/amd_gpu) and run the generated **Python** command on four GPUs. Select FP8 or [MXFP4](https://huggingface.co/OneNexus/GLM-5.3-Flash-MXFP4); SGLang downloads the model and tokenizer automatically.
|
||||
|
||||
For TPU7x, install [sglang-torchtpu](https://github.com/sgl-project/sglang-torchtpu) from source (its `main` branch; the `v0.1.0-tpu` image predates the GLM-5.3-Flash MTP fixes) and run the generated **Python** command on one TPU7x host with 4 chips:
|
||||
|
||||
```bash Command
|
||||
git clone https://github.com/sgl-project/sglang-torchtpu.git
|
||||
cd sglang-torchtpu
|
||||
cp python/pyproject_tpu.toml python/pyproject.toml
|
||||
pip install --extra-index-url https://download.pytorch.org/whl/cpu -e python/
|
||||
pip install --no-deps -e .
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Install and prepare Ascend NPUs">
|
||||
|
||||
Use a Linux host with the Ascend driver and a compatible CANN / TorchNPU environment. Follow the
|
||||
[Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) and the
|
||||
[GLM-5.3-Flash deployment tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash#installation) to start a container with the NPU devices, driver mounts, and checkpoint directory. Run the setup and the generated **Python** command inside that container.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="A3">
|
||||
|
||||
```bash Command
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-glm5.3-flash
|
||||
```
|
||||
|
||||
Run the following setup before copying the generated command (CANN 9.1.0). CPU tuning requires host privileges.
|
||||
|
||||
```bash Command
|
||||
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="950PR/DT series">
|
||||
|
||||
Use a CANN 9.1.0 SGLang image with the custom operators and MemFabric runtime required by the tutorial; PD mode also requires DP-attention support. The tutorial does not specify a 950PR/DT series image tag.
|
||||
|
||||
```bash Command
|
||||
NPU_IMAGE="<your-compatible-950pr-dt-series-image>"
|
||||
docker pull "$NPU_IMAGE"
|
||||
```
|
||||
|
||||
```bash Command
|
||||
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
|
||||
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash
|
||||
source /usr/local/memfabric_hybrid/set_env.sh
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
Replace the model path in the generated command with your mounted ModelSlim checkpoint directory. Include its configuration, tokenizer, quantization metadata, and MTP weights.
|
||||
|
||||
Use [W8A8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8) for A3 and [W8A8_MXFP8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8-mxfp8) for 950PR/DT series.
|
||||
|
||||
Replace network placeholders such as `<network-interface>`, `<prefill-head-ip>`, and `<decode-head-ip>` in the generated NPU command with your cluster settings.
|
||||
|
||||
</Accordion>
|
||||
|
||||
The cookbook opens on **B300**. Choose your hardware, then choose the operating point that matches your workload:
|
||||
@@ -27,7 +88,7 @@ The cookbook opens on **B300**. Choose your hardware, then choose the operating
|
||||
- **Low Latency** starts with MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses.
|
||||
- **High Throughput** starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches.
|
||||
|
||||
Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
|
||||
Every listed GPU platform exposes both strategies; TPU7x has a Low Latency recipe only. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.
|
||||
|
||||
The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, Breakable Cuda Graph, and HiCache tiers. The MI355X commands are marked **Not Verified** because the selector defaults to HiCache off, whereas the measured AgentX configuration adds the host tier described in [Extend the cache hierarchy](#extend-the-cache-hierarchy). Do not treat the CUDA benchmark rows as MI355X measurements.
|
||||
|
||||
@@ -110,6 +171,10 @@ The **Speculative** card in the Playground changes the algorithm without leaving
|
||||
|
||||
Neither algorithm runs with DP-Attention; the card disables the affected chips and names the reason. DFlash2's FA4 draft path is also disabled on MI355X.
|
||||
|
||||
### Serve on TPU7x
|
||||
|
||||
The TPU7x recipe serves the language model with DeepSeek Sparse Attention on the Pallas sparse MLA kernels and MTP 3/1/4 on the experimental chain speculative worker (`SGLANG_TPU_ENABLE_EXPERIMENTAL_SPEC_WORKER=1`). On TPU7x, 3/1/4 decodes faster than 5/1/6 at the same accept length, because each extra draft row widens the target verify. It is sized for one request with 8K input and 1K output: `--context-length 10240`, `--max-running-requests 1`, and a 20,480-token KV pool. Raise `--context-length`, `--max-total-tokens`, `--max-running-requests` and `--max-mamba-cache-size` together for longer or concurrent requests; those settings are outside the verified command. The first start compiles every decode and prefill shape listed in `--tpu-compile-bs` and `--tpu-compile-tokens`, so it takes longer than later starts that reuse the compilation cache.
|
||||
|
||||
### Size both memory pools
|
||||
|
||||
GLM-5.3-Flash maintains a paged KV pool for attention and a separate KDA state pool. The KDA state pool can limit concurrency before the KV pool is full. If startup reduces `max_running_requests` because of KDA state capacity, increase `--mamba-full-memory-ratio` or set `--max-mamba-cache-size` for the expected concurrency, then tune `--max-running-requests` to the workload.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: GLM-5.3
|
||||
description: "Deploy GLM-5.3 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X."
|
||||
description: "Deploy GLM-5.3 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X, plus Ascend NPU recipes."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
@@ -40,6 +40,60 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Install and prepare Ascend NPUs">
|
||||
|
||||
Use a Linux host with the Ascend driver and a compatible CANN / TorchNPU environment. Follow the
|
||||
[Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) and the
|
||||
[GLM-5.3 deployment tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3#installation) to start a container with the NPU devices, driver mounts, and checkpoint directory. Run the setup and the generated **Python** command inside that container.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="A3">
|
||||
|
||||
```bash Command
|
||||
docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080
|
||||
```
|
||||
|
||||
This CANN 9.1.0 image requires a compatible SGLang build, TorchNPU, ATB, and custom operators before serving. Set `SGLANG_ROOT` to that source checkout.
|
||||
|
||||
Run the following setup before copying the generated command (CANN 9.1.0). CPU tuning requires host privileges.
|
||||
|
||||
```bash Command
|
||||
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
|
||||
SGLANG_ROOT=/path/to/sglang
|
||||
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
|
||||
export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib:${LD_LIBRARY_PATH:-}"
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="950PR/DT series">
|
||||
|
||||
Use the compatible 950PR/DT series SGLang checkout, CANN environment, and custom operators used for your deployment. Set `SGLANG_ROOT` to that checkout; these recipes do not prescribe an image tag.
|
||||
|
||||
```bash Command
|
||||
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
|
||||
unset HCCL_IF_IP HCCL_SOCKET_FAMILY RANK_TABLE_FILE
|
||||
SGLANG_ROOT=/path/to/sglang
|
||||
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
Replace the model path in the generated command with your mounted ModelSlim checkpoint directory. Include its configuration, tokenizer, quantization metadata, and MTP weights.
|
||||
|
||||
Use a locally prepared **W8A8** checkpoint for A3. On 950PR/DT series, both unified serving and PD disaggregation use the same **W8A8C8_MXFP8** checkpoint. Both 950PR/DT series recipes explicitly use FP8 E4M3 KV cache and BF16 draft KV cache.
|
||||
|
||||
Replace network placeholders such as `<network-interface>`, `<prefill-head-ip>`, and `<decode-head-ip>` in the generated NPU command with your cluster settings.
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:
|
||||
|
||||
- **Low-Latency** — fastest reply for a single user. Pick for chat.
|
||||
@@ -50,7 +104,7 @@ import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||
import { config } from "/src/snippets/configs/zai-org/glm-5.3.jsx";
|
||||
import { benchmarks } from "/src/snippets/configs/zai-org/glm-5.3-benchmarks.jsx";
|
||||
|
||||
<Deployment config={config} benchmarks={benchmarks} />
|
||||
<Deployment config={config} benchmarks={benchmarks} agenticLink="agentic-long-context-deployment" />
|
||||
|
||||
<Warning>
|
||||
All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.3.
|
||||
@@ -64,6 +118,19 @@ import { Playground } from "/src/snippets/_playground.jsx";
|
||||
|
||||
<Playground config={config} />
|
||||
|
||||
## Agentic Long-Context Deployment
|
||||
|
||||
Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.
|
||||
|
||||
- **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
|
||||
- **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
|
||||
- **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.
|
||||
|
||||
import { AgentX } from "/src/snippets/_agentx.jsx";
|
||||
import { agentx } from "/src/snippets/agentx/zai-org/glm-5.3.jsx";
|
||||
|
||||
<AgentX data={agentx} />
|
||||
|
||||
## 1. Model Introduction
|
||||
|
||||
**GLM-5.3** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It keeps the same base architecture while updating post-training for complex coding and long-horizon tasks. It ships in two precisions — **FP8** (`zai-org/GLM-5.3`) and full **BF16** (`zai-org/GLM-5.3-BF16`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, RadixArk publishes an **experimental NVFP4** build (`RadixArk/GLM-5.3-NVFP4`, Model Optimizer) that quantizes only the routed experts' linear weights and activations to 4-bit (attention, shared experts, dense layers, MTP, embeddings, and the LM head stay unquantized), cutting the weight footprint to ~0.45 TB so a 4-GPU GB300 node can serve it with TP4. **The NVFP4 cells are experimental** — benchmark data (accuracy / throughput) is still pending; treat them as provisional until the benchmark cards fill in.
|
||||
|
||||
@@ -97,7 +97,7 @@ For other installation methods, please refer to the [official SGLang installatio
|
||||
|
||||
### 3.1 Basic Configuration
|
||||
|
||||
The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). The recommended launch configurations vary by hardware and model size.
|
||||
The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), Xeon CPUs. The recommended launch configurations vary by hardware and model size.
|
||||
|
||||
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant.
|
||||
|
||||
|
||||
@@ -22,11 +22,23 @@ Select **Docker** in the command panel to launch the matching image. The **Pytho
|
||||
The first launch in a fresh container spends about five extra minutes compiling JIT kernels for the speculative-decoding graphs (about 4–5 minutes once, then about 30 seconds). Mount a persistent cache volume, for example `-v ~/.cache:/root/.cache`, to skip it on later starts.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
**AMD MI325X / MI350X:** the published ROCm images predate the ROCm gated-MLA fix that Ling-3.1-flash needs; without it the server fails while capturing decode CUDA graphs. Until a ROCm image includes the fix, build a thin image that switches the nightly's editable SGLang source to the Ling-3.1-flash branch (use the `mi35x` base for MI350X), then substitute it for the image in the generated Docker command:
|
||||
|
||||
```dockerfile Dockerfile
|
||||
FROM lmsysorg/sglang-rocm:v0.5.21-rocm724-mi30x-20261013
|
||||
RUN cd /sgl-workspace/sglang \
|
||||
&& git fetch https://github.com/sgl-project/sglang.git bailing-v3-day0 \
|
||||
&& git checkout FETCH_HEAD
|
||||
```
|
||||
</Note>
|
||||
|
||||
</Accordion>
|
||||
|
||||
Pick your hardware to generate the launch command:
|
||||
|
||||
- **Low-Latency** — single-user chat and interactive requests, with NEXTN speculative decoding. Verified on H200 and GB300; H20-3e, H800, H100, and B200 recipes are provided unverified.
|
||||
- **Low-Latency** — TP-only (EP=1) with NEXTN speculative decoding for single-user chat and interactive requests. Verified on H200.
|
||||
- **High-Throughput** — expert-parallel (EP) serving for batch and multi-user traffic: TP8 + EP8 on H200/H800/H100/B200-class nodes, TP4 + EP4 on GB300, TP8 + EP8 with AITER on AMD MI300X/MI325X/MI350X/MI355X (without speculative decoding). Verified on H200 and GB300; other platforms are unverified. AMD MI325X and MI350X recipes (TP8 + EP8 with AITER, without speculative decoding) are provided unverified; the MI325X accuracy in the benchmark card was measured by AMD.
|
||||
|
||||
For prefix-heavy multi-turn workloads, enable **HiCache** in the Playground after starting the Mooncake services in §3.3. The combination is verified: prefix hits return from Mooncake L3 on every TP rank.
|
||||
|
||||
@@ -60,6 +72,8 @@ The model has about 560B total parameters (512 routed experts with 8 active plus
|
||||
- On H200, enable `SGLANG_ENABLE_FP8_BLOCK_GEMV=1` and `SGLANG_ENABLE_FUSED_KDA_OUTPUT_QUANT=1`. Both are included in the generated commands; they are no-ops on GB300 (SM10x).
|
||||
- Leave `--mem-fraction-static` at its default. SGLang auto-selects the `ling3` reasoning and tool-call parsers for this architecture; you can change the parser settings in the Playground.
|
||||
- **NEXTN:** add `--speculative-algorithm NEXTN` for built-in MTP speculative decoding. Measured acceptance length is 2.61 on random tokens and 2.13 on real-text probes.
|
||||
- **Long context:** the checkpoint supports 262,144 tokens natively. For up to 1,048,576 tokens, add the YaRN override from §3.4.
|
||||
- **Topology:** the Low-Latency recipe runs TP-only (no `--ep-size`) and is clearly faster at batch size 1 (about 9-13% lower TPOT on H200, verified on both 2048/512 and 8192/1024 workloads). At concurrency 16-32 the two recipes deliver the same throughput within ~2%. TP-only relies on the loader's TP-requant support; it fails loudly at startup on builds without it.
|
||||
- For prefix-heavy, multi-turn workloads, select **Cache fork and tail** in the Playground. It combines `--mamba-radix-cache-strategy extra_buffer`, `--chunked-prefill-size 4096`, and `SGLANG_SPLIT_PREFILL_AT_MAMBA_FORK=1`. GPU checks confirmed both fork and tail states are cached, saving about 576 tokens per subsequent turn in the measured workload.
|
||||
|
||||
### Playground support
|
||||
@@ -71,7 +85,7 @@ The model has about 560B total parameters (512 routed experts with 8 active plus
|
||||
|
||||
### Performance
|
||||
|
||||
On H200 TP8 + EP8 with NEXTN and the decode gates enabled, batch-size-1 output throughput measures **251.2 tokens/s**. GB300 NEXTN batch-size-1 result: `180.0 tok/s (TPOT 5.14 ms, accept 2.59)`.
|
||||
On H200 TP8 with NEXTN and the decode gates enabled, batch-size-1 output throughput measures **269.2 tokens/s** (TP8 + EP8: 239.5 tokens/s). GB300 TP4 with NEXTN measures **175.3 tokens/s** (`TPOT 5.22 ms, accept 2.60`; TP4 + EP4: 160.0 tokens/s).
|
||||
|
||||
Acceptance length is the mean number of tokens accepted per speculative verification step, including the target bonus token:
|
||||
|
||||
@@ -108,7 +122,7 @@ python3 -m sglang.bench_serving \
|
||||
--flush-cache --seed 11
|
||||
```
|
||||
|
||||
For the concurrency-32 row in the benchmark widget:
|
||||
For the saturation row in the benchmark widget, launch the H200 **High-Throughput** recipe instead and run:
|
||||
|
||||
```bash Command
|
||||
python3 -m sglang.bench_serving \
|
||||
@@ -116,9 +130,9 @@ python3 -m sglang.bench_serving \
|
||||
--host localhost --port 30000 \
|
||||
--model inclusionAI/Ling-3.1-flash \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 --random-output-len 512 \
|
||||
--random-input-len 8192 --random-output-len 1024 \
|
||||
--random-range-ratio 1 \
|
||||
--num-prompts 96 --max-concurrency 32 \
|
||||
--num-prompts 256 --max-concurrency 48 \
|
||||
--flush-cache --seed 11
|
||||
```
|
||||
|
||||
@@ -133,7 +147,7 @@ sgl-eval run gsm8k \
|
||||
--num-threads 32
|
||||
```
|
||||
|
||||
Evaluate all 1,319 questions and inspect completion reasons as well as accuracy. For the H200 NEXTN recipe, compare the stop rate with the measured 99.85%; accuracy alone can hide generations that reach the token limit.
|
||||
Evaluate all 1,319 questions and inspect completion reasons as well as accuracy. For the H200 NEXTN recipes, compare the stop rate with the measured 99.77% (Low-Latency) or 99.92% (High-Throughput); accuracy alone can hide generations that reach the token limit.
|
||||
|
||||
</Accordion>
|
||||
|
||||
@@ -277,30 +291,48 @@ print(follow_up.choices[0].message.content)
|
||||
|
||||
```json Output
|
||||
{
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"reasoning_content": "The user wants to know the current weather in Hangzhou. I'll call the get_weather function with city \"Hangzhou\". The unit parameter is optional - I'll use celsius as default since Hangzhou is in China.",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_2bae8419df9d44f2bb54c22f",
|
||||
"index": 0,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"arguments": "{\"city\": \"Hangzhou\", \"unit\": \"celsius\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"finish_reason": "tool_calls"
|
||||
"content": "",
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_ecd286d7dccb4c70a8c9c586",
|
||||
"function": {
|
||||
"arguments": "{\"city\": \"Hangzhou\", \"unit\": \"celsius\"}",
|
||||
"name": "get_weather"
|
||||
},
|
||||
"type": "function",
|
||||
"index": 0
|
||||
}
|
||||
],
|
||||
"reasoning_content": "The user is asking about the current weather in Hangzhou. I need to call the get_weather function with the city parameter set to \"Hangzhou\". The unit parameter is optional - I'll default to celsius since that's commonly used in China."
|
||||
}
|
||||
```
|
||||
|
||||
The follow-up assistant turn answers with both `reasoning_content` and `content` populated.
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example Output (follow-up)">
|
||||
|
||||
```text Output
|
||||
The user asked about the weather in Hangzhou and I got the result. Let me provide a natural response.
|
||||
The weather in Hangzhou right now is sunny with a temperature of 22°C. It's a pleasant day!
|
||||
```
|
||||
|
||||
The first line is `reasoning_content`; the second is `content`. `finish_reason` is `stop`.
|
||||
|
||||
</Accordion>
|
||||
|
||||
### 3.4 Long context with YaRN (up to 1M tokens)
|
||||
|
||||
The model's native context is 262,144 tokens. Verified on H200 and GB300, YaRN extrapolation serves up to **1,048,576 tokens**:
|
||||
|
||||
```bash Command
|
||||
--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144, "rope_theta": 6000000, "partial_rotary_factor": 0.5}, "max_position_embeddings": 1048576}'
|
||||
```
|
||||
|
||||
Add it to the Low-Latency serve command. All four keys are required: the override replaces the checkpoint's RoPE parameters, so omitting `rope_theta` or `partial_rotary_factor` silently changes the RoPE base, and omitting `max_position_embeddings` keeps the 262,144 limit. Only enable it for prompts beyond 256K.
|
||||
|
||||
Verification on the current checkpoint: a fact planted at token 700,023 of a 900,022-token prompt is retrieved exactly, NEXTN acceptance stays about 2.1 at 900K, and a 300,101-token prompt is answered correctly (H200 TTFT 15.4 s, GB300 12.6 s). One 1M-token request fits alongside normal serving traffic on both platforms.
|
||||
|
||||
### 3.3 HiCache with Mooncake
|
||||
|
||||
With HiCache enabled, prefix state is written through to Mooncake L3 and later requests prefetch it back: measured 5952 of 6000 tokens restored from storage on every TP rank. Mooncake runs as L3 prefix storage over TCP; start the metadata server, master, and storage client before launching SGLang. Run these services on the same host as the SGLang container:
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: Kev
|
||||
description: "Deploy the Kev decision models with SGLang. Kev returns calibrated probabilities for typed questions about a document, in sizes from 0.8B to 27B, each on one NVIDIA H200 in BF16."
|
||||
tag: NEW
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
Every Kev size runs on one GPU in BF16 with TP=1 and needs no extra flag. SGLang reads the checkpoint's `head.pt` and configures the server.
|
||||
|
||||
```bash Command
|
||||
sglang serve --model-path jaredpalmer/kev-27b --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
Replace the model path with any size in the table below. Each was validated on one H200.
|
||||
|
||||
| Model | Hugging Face ID | Checkpoint | Backbone | GPU memory for weights |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Kev-27B | `jaredpalmer/kev-27b` | full weights | Qwen3.8-27B | 47.8 GB |
|
||||
| Kev-9B | `jaredpalmer/kev-9b` | LoRA adapter | Qwen3.5-9B-Base | 14.9 GB |
|
||||
| Kev-4B | `jaredpalmer/kev-4b` | LoRA adapter | Qwen3.5-4B-Base | 8.0 GB |
|
||||
| Kev-0.8B | `jaredpalmer/kev-0.8b` | LoRA adapter | Qwen3.5-0.8B-Base | 1.5 GB |
|
||||
|
||||
For a LoRA adapter, SGLang loads the base revision that the adapter's `head.pt` pins and merges the adapter into its weights at startup, in FP32 with one rounding to BF16, as the reference code merges it. The base is downloaded on first use.
|
||||
|
||||
## 1. Model introduction
|
||||
|
||||
[Kev](https://huggingface.co/collections/jaredpalmer/kev) is a family of decision models published by Jared Palmer under the Apache-2.0 license, with [reference code](https://github.com/jaredpalmer/kev). Each model keeps a Qwen3.5 or Qwen3.8 backbone and replaces the LM head with a pointer head. Each question runs as its own prompt after the shared state. The head compares the final hidden state at each option's closing token with the one at the prompt's last token and returns one logit per option, with no text generation. A temperature stored in `head.pt` calibrates the probabilities.
|
||||
|
||||
Pass a text string, JSON object, or JSON array as the state. `/v1/systemone` returns one answer per question: a `choice` over allowed options, a `noul` probability of true, or a `score` over ordered levels. JSON states are flattened into labeled text as the reference code does.
|
||||
|
||||
## 2. Configuration tips
|
||||
|
||||
- **Route.** Send requests to `/v1/systemone`, or set your System One client's base URL to the server. This includes TypeSafe SDK clients. See the [API reference](/docs/supported-models/decision_models#system-one-compatible-api) for request and response fields. Kev refuses `/v1/decisions`, chat, generation, and `/encode` requests.
|
||||
- **One prefill per question.** The model reads the state once for every question, as the reference does, so a request with many questions over a long state costs that many prefills. SGLang enables embedding mode and disables radix caching, chunked prefill, and CUDA graphs. Keep TP=1 and PP=1.
|
||||
- **Answers.** `probabilities` are the softmax of the head's logits divided by the calibrated temperature. `confidence` uses the System One adapter formulas, as Kev's own server does. SGLang does not round the values to four decimals. Answers omit `x_label_mass` because the head scores only the allowed options.
|
||||
- **Long states.** The reference server accepts states of up to 65,536 tokens. In SGLang each question's prompt must also fit one prefill, 16,384 tokens by default, so longer prompts are refused with an error that gives both lengths. Raising `--max-prefill-tokens` is not supported for this model yet, because the server does not reserve memory for longer unchunked prefills and can run out of memory under load.
|
||||
- **Inputs.** Requests are text only, so `images` and `chat_template_kwargs` are refused. A `score` question takes up to 10 levels and a `choice` question up to 255 options.
|
||||
- **Not supported.** Kev-27B v1, the `v1-lora` revision of `jaredpalmer/kev-27b`, is an adapter trained on a BF16 backbone that the reference applies unmerged, so SGLang refuses it and serves the current full weight release instead. Kev-0.5B is a prototype on a Qwen2.5 base and is refused too.
|
||||
|
||||
## 3. Advanced usage
|
||||
|
||||
### 3.1 TypeSafe SDK
|
||||
|
||||
```bash Command
|
||||
pip install typesafe-sdk
|
||||
```
|
||||
|
||||
<Accordion title="Text decision example (Python)">
|
||||
|
||||
```python Example
|
||||
from typesafe_sdk import Choice, Noul, TypeSafeClient
|
||||
|
||||
client = TypeSafeClient(api_key="local", base_url="http://localhost:30000", model="kev-latest")
|
||||
response = client.system_one(
|
||||
state="I was charged twice for order 1182. Please refund one of the charges.",
|
||||
questions={
|
||||
"team": Choice(
|
||||
instructions="Which team should handle this?",
|
||||
criteria={"billing": "Charges and refunds", "shipping": "Deliveries", "returns": "Exchanges"},
|
||||
),
|
||||
"urgent": Noul(instructions="Does this need a reply today?"),
|
||||
},
|
||||
)
|
||||
print(response.choices["team"].choice, response.nouls["urgent"].noul)
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Example output">
|
||||
|
||||
Captured with Kev-27B on one H200. Probabilities can vary with the model size, hardware, and numerical backend.
|
||||
|
||||
```text Output
|
||||
billing 0.7591662905989526
|
||||
```
|
||||
|
||||
</Accordion>
|
||||
@@ -48,6 +48,445 @@ import { agentx } from "/src/snippets/agentx/meta-models/muse-spark-open.jsx";
|
||||
|
||||
<AgentX data={agentx} />
|
||||
|
||||
### GB300 deployments measured with Artificial Analysis AgentPerf
|
||||
|
||||
Five aggregated Dynamo + SGLang deployments on GB300, measured with the [Artificial Analysis AgentPerf](https://github.com/ArtificialAnalysis/agentperf-client) coding-agent workload. Together they span the trade-off from the lowest per-user latency (concurrency 1) to the highest throughput per GPU (concurrency 160), ordered from low latency to high throughput. Each one starts from the Artificial Analysis SGLang submission for this model and changes only flags, environment variables and parallelism.
|
||||
|
||||
- **Container image.** `lmsysorg/sglang:dev-muse-spark-open@sha256:8cfc5a72cb296771f5f8a4a4688cb6d05685610a830cbb616fa4ab0eb7bb2b0a` with the Dynamo wheels added: `pip install --no-deps ai-dynamo==1.6.0.dev20261019 ai-dynamo-runtime==1.6.0.dev20261019`.
|
||||
- **Router.** One Dynamo frontend with the KV-aware router. Session affinity keeps requests that carry the same `X-Dynamo-Session-ID` header on one worker; the AgentPerf client sends one ID per agent session.
|
||||
- **Checkpoints.** `meta-models/Muse-Spark-Open-816B-A42B-NVFP4` with the DFlash2 draft `RadixArk/Muse-Spark-Open-DFlash2` (8 draft tokens, FP8 KV cache).
|
||||
- **HiCache.** `--hicache-size` is the host tier in GB per rank; scale it to your host memory.
|
||||
- **Two-node deployments.** The TP8·EP8 worker spans two GB300 trays (4 GPUs each) over multi-node NVLink: run the worker command on both nodes, with `NODE_RANK` 0 on the first.
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Agg TP8·EP8 (2 nodes), lowest latency · 8 GPUs · concurrency 1">
|
||||
<Tabs sync={false}>
|
||||
<Tab title="Env Vars">
|
||||
```bash
|
||||
export ETCD_IP=<node running etcd>
|
||||
export AGG_HEAD_IP=<first node of the worker>
|
||||
export NODE_RANK=<0 on the first node, 1 on the second>
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="etcd">
|
||||
```bash
|
||||
etcd \
|
||||
--data-dir /tmp/etcd \
|
||||
--listen-client-urls http://0.0.0.0:2379 \
|
||||
--advertise-client-urls http://$ETCD_IP:2379
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Dynamo frontend">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export DYN_ROUTER_QUEUE_THRESHOLD=None
|
||||
export DYN_ROUTER_TEMPERATURE=0
|
||||
export DYN_TOKENIZER=fastokens
|
||||
export DYN_TOKENIZER_CACHE=1
|
||||
export DYN_TOKENIZER_CACHE_BYTES=8000000000
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
|
||||
python3 -m dynamo.frontend \
|
||||
--http-port 8000 \
|
||||
--router-mode kv \
|
||||
--router-session-affinity-ttl-secs 14400 \
|
||||
--active-decode-blocks-threshold None \
|
||||
--active-prefill-tokens-threshold None \
|
||||
--active-prefill-tokens-threshold-frac None
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="SGLang worker (run on each of the 2 nodes)">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
export NCCL_CUMEM_ENABLE=1
|
||||
export NCCL_MNNVL_ENABLE=1
|
||||
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
||||
export SGLANG_TIMEOUT_KEEP_ALIVE=900
|
||||
|
||||
python3 -m dynamo.sglang \
|
||||
--model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--dist-init-addr $AGG_HEAD_IP:20000 \
|
||||
--nnodes 2 \
|
||||
--node-rank $NODE_RANK \
|
||||
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
|
||||
--request-plane tcp \
|
||||
--dyn-reasoning-parser muse \
|
||||
--dyn-tool-call-parser muse \
|
||||
--expert-parallel-size 8 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--max-running-requests 32 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--mm-feature-transport cpu \
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--quantization modelopt_fp4 \
|
||||
--reasoning-parser muse \
|
||||
--speculative-algorithm DFLASH \
|
||||
--speculative-draft-attention-backend fa4 \
|
||||
--speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
|
||||
--speculative-num-draft-tokens 8 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
</Accordion>
|
||||
<Accordion title="Agg TP8·EP8 (2 nodes), low latency · 8 GPUs · concurrency 4">
|
||||
<Tabs sync={false}>
|
||||
<Tab title="Env Vars">
|
||||
```bash
|
||||
export ETCD_IP=<node running etcd>
|
||||
export AGG_HEAD_IP=<first node of the worker>
|
||||
export NODE_RANK=<0 on the first node, 1 on the second>
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="etcd">
|
||||
```bash
|
||||
etcd \
|
||||
--data-dir /tmp/etcd \
|
||||
--listen-client-urls http://0.0.0.0:2379 \
|
||||
--advertise-client-urls http://$ETCD_IP:2379
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Dynamo frontend">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export DYN_ROUTER_QUEUE_THRESHOLD=None
|
||||
export DYN_ROUTER_TEMPERATURE=0
|
||||
export DYN_TOKENIZER=fastokens
|
||||
export DYN_TOKENIZER_CACHE=1
|
||||
export DYN_TOKENIZER_CACHE_BYTES=8000000000
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
|
||||
python3 -m dynamo.frontend \
|
||||
--http-port 8000 \
|
||||
--router-mode kv \
|
||||
--router-session-affinity-ttl-secs 14400 \
|
||||
--active-decode-blocks-threshold None \
|
||||
--active-prefill-tokens-threshold None \
|
||||
--active-prefill-tokens-threshold-frac None
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="SGLang worker (run on each of the 2 nodes)">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
export NCCL_CUMEM_ENABLE=1
|
||||
export NCCL_MNNVL_ENABLE=1
|
||||
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
||||
export SGLANG_TIMEOUT_KEEP_ALIVE=900
|
||||
|
||||
python3 -m dynamo.sglang \
|
||||
--model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--dist-init-addr $AGG_HEAD_IP:20000 \
|
||||
--nnodes 2 \
|
||||
--node-rank $NODE_RANK \
|
||||
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
|
||||
--request-plane tcp \
|
||||
--dyn-reasoning-parser muse \
|
||||
--dyn-tool-call-parser muse \
|
||||
--expert-parallel-size 8 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--max-running-requests 32 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--mm-feature-transport cpu \
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--quantization modelopt_fp4 \
|
||||
--reasoning-parser muse \
|
||||
--speculative-algorithm DFLASH \
|
||||
--speculative-draft-attention-backend fa4 \
|
||||
--speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
|
||||
--speculative-num-draft-tokens 8 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
</Accordion>
|
||||
<Accordion title="Agg TP4·EP4, balanced · 4 GPUs · concurrency 16">
|
||||
<Tabs sync={false}>
|
||||
<Tab title="Env Vars">
|
||||
```bash
|
||||
export ETCD_IP=<node running etcd>
|
||||
export AGG_HEAD_IP=<first node of the worker>
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="etcd">
|
||||
```bash
|
||||
etcd \
|
||||
--data-dir /tmp/etcd \
|
||||
--listen-client-urls http://0.0.0.0:2379 \
|
||||
--advertise-client-urls http://$ETCD_IP:2379
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Dynamo frontend">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export DYN_ROUTER_QUEUE_THRESHOLD=None
|
||||
export DYN_ROUTER_TEMPERATURE=0
|
||||
export DYN_TOKENIZER=fastokens
|
||||
export DYN_TOKENIZER_CACHE=1
|
||||
export DYN_TOKENIZER_CACHE_BYTES=8000000000
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
|
||||
python3 -m dynamo.frontend \
|
||||
--http-port 8000 \
|
||||
--router-mode kv \
|
||||
--router-session-affinity-ttl-secs 14400 \
|
||||
--active-decode-blocks-threshold None \
|
||||
--active-prefill-tokens-threshold None \
|
||||
--active-prefill-tokens-threshold-frac None
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="SGLang worker">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
export NCCL_CUMEM_ENABLE=1
|
||||
export SGLANG_TIMEOUT_KEEP_ALIVE=900
|
||||
|
||||
python3 -m dynamo.sglang \
|
||||
--model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
|
||||
--request-plane tcp \
|
||||
--dyn-reasoning-parser muse \
|
||||
--dyn-tool-call-parser muse \
|
||||
--enable-layernorm-sp \
|
||||
--enable-megamoe-for-layernorm-sp \
|
||||
--expert-parallel-size 4 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--max-running-requests 32 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--mm-feature-transport cpu \
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--prefill-decode-interval 64 \
|
||||
--quantization modelopt_fp4 \
|
||||
--reasoning-parser muse \
|
||||
--speculative-algorithm DFLASH \
|
||||
--speculative-draft-attention-backend fa4 \
|
||||
--speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
|
||||
--speculative-num-draft-tokens 8 \
|
||||
--swa-full-tokens-ratio 0.1 \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
</Accordion>
|
||||
<Accordion title="Agg TP4·EP4, high throughput · 4 GPUs · concurrency 64">
|
||||
<Tabs sync={false}>
|
||||
<Tab title="Env Vars">
|
||||
```bash
|
||||
export ETCD_IP=<node running etcd>
|
||||
export AGG_HEAD_IP=<first node of the worker>
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="etcd">
|
||||
```bash
|
||||
etcd \
|
||||
--data-dir /tmp/etcd \
|
||||
--listen-client-urls http://0.0.0.0:2379 \
|
||||
--advertise-client-urls http://$ETCD_IP:2379
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Dynamo frontend">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export DYN_ROUTER_QUEUE_THRESHOLD=None
|
||||
export DYN_ROUTER_TEMPERATURE=0
|
||||
export DYN_TOKENIZER=fastokens
|
||||
export DYN_TOKENIZER_CACHE=1
|
||||
export DYN_TOKENIZER_CACHE_BYTES=8000000000
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
|
||||
python3 -m dynamo.frontend \
|
||||
--http-port 8000 \
|
||||
--router-mode kv \
|
||||
--router-session-affinity-ttl-secs 14400 \
|
||||
--active-decode-blocks-threshold None \
|
||||
--active-prefill-tokens-threshold None \
|
||||
--active-prefill-tokens-threshold-frac None
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="SGLang worker">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
export NCCL_CUMEM_ENABLE=1
|
||||
export SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1
|
||||
export SGLANG_TIMEOUT_KEEP_ALIVE=900
|
||||
|
||||
python3 -m dynamo.sglang \
|
||||
--model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
|
||||
--request-plane tcp \
|
||||
--cuda-graph-max-bs-decode 64 \
|
||||
--dyn-reasoning-parser muse \
|
||||
--dyn-tool-call-parser muse \
|
||||
--enable-layernorm-sp \
|
||||
--enable-megamoe-for-layernorm-sp \
|
||||
--expert-parallel-size 4 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--max-running-requests 64 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--mm-feature-transport cpu \
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--prefill-decode-interval 24 \
|
||||
--quantization modelopt_fp4 \
|
||||
--reasoning-parser muse \
|
||||
--speculative-algorithm DFLASH \
|
||||
--speculative-draft-attention-backend fa4 \
|
||||
--speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
|
||||
--speculative-num-draft-tokens 8 \
|
||||
--swa-full-tokens-ratio 0.15 \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
</Accordion>
|
||||
<Accordion title="Agg TP4·EP4 + HiCache, max throughput · 4 GPUs · concurrency 160">
|
||||
<Tabs sync={false}>
|
||||
<Tab title="Env Vars">
|
||||
```bash
|
||||
export ETCD_IP=<node running etcd>
|
||||
export AGG_HEAD_IP=<first node of the worker>
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="etcd">
|
||||
```bash
|
||||
etcd \
|
||||
--data-dir /tmp/etcd \
|
||||
--listen-client-urls http://0.0.0.0:2379 \
|
||||
--advertise-client-urls http://$ETCD_IP:2379
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Dynamo frontend">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export DYN_ROUTER_QUEUE_THRESHOLD=None
|
||||
export DYN_ROUTER_TEMPERATURE=0
|
||||
export DYN_TOKENIZER=fastokens
|
||||
export DYN_TOKENIZER_CACHE=1
|
||||
export DYN_TOKENIZER_CACHE_BYTES=8000000000
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
|
||||
python3 -m dynamo.frontend \
|
||||
--http-port 8000 \
|
||||
--router-mode kv \
|
||||
--router-session-affinity-ttl-secs 14400 \
|
||||
--active-decode-blocks-threshold None \
|
||||
--active-prefill-tokens-threshold None \
|
||||
--active-prefill-tokens-threshold-frac None
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="SGLang worker">
|
||||
```bash
|
||||
export DYN_REQUEST_PLANE=tcp
|
||||
export ETCD_ENDPOINTS=http://$ETCD_IP:2379
|
||||
export NCCL_CUMEM_ENABLE=1
|
||||
export SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1
|
||||
export SGLANG_TIMEOUT_KEEP_ALIVE=900
|
||||
|
||||
python3 -m dynamo.sglang \
|
||||
--model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
|
||||
--request-plane tcp \
|
||||
--cuda-graph-max-bs-decode 176 \
|
||||
--dyn-reasoning-parser muse \
|
||||
--dyn-tool-call-parser muse \
|
||||
--enable-hierarchical-cache \
|
||||
--enable-layernorm-sp \
|
||||
--enable-megamoe-for-layernorm-sp \
|
||||
--expert-parallel-size 4 \
|
||||
--hicache-size 120 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--max-running-requests 176 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--mm-feature-transport cpu \
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--prefill-decode-interval 6 \
|
||||
--quantization modelopt_fp4 \
|
||||
--reasoning-parser muse \
|
||||
--speculative-algorithm DFLASH \
|
||||
--speculative-draft-attention-backend fa4 \
|
||||
--speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
|
||||
--speculative-num-draft-tokens 8 \
|
||||
--swa-full-tokens-ratio 0.15 \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--watchdog-timeout 3600
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
### MI355X deployment measured with SemiAnalysis InferenceX
|
||||
|
||||
TP8·EP8 with DFlash V2 on 8× MI355X, measured in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) at concurrency 1–64.
|
||||
|
||||
- **Container image.** `lmsysorg/sglang-rocm:muse-spark-open-1020`.
|
||||
- **HiCache.** At concurrency 16 and above, add the HiCache flags below. `--hicache-size` is the host tier in GB per rank, so 253 GB needs about 2 TB of host memory across the eight ranks.
|
||||
|
||||
```bash Command
|
||||
SGLANG_USE_AITER=1 \
|
||||
SGLANG_USE_AITER_UNIFIED_ATTN=1 \
|
||||
MUSE_MOE_BACKEND=flydsl \
|
||||
python -m sglang.launch_server \
|
||||
--model-path amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--attention-backend aiter \
|
||||
--page-size 16 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--max-running-requests 128 \
|
||||
--swa-full-tokens-ratio 0.2 \
|
||||
--prefill-decode-interval 20 \
|
||||
--scheduler-recv-interval 20 \
|
||||
--tokenizer-worker-num 4 \
|
||||
--speculative-algorithm DFLASH \
|
||||
--speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
|
||||
--speculative-draft-attention-backend triton \
|
||||
--speculative-num-draft-tokens 8 \
|
||||
--speculative-draft-window-size 2048 \
|
||||
--reasoning-parser muse \
|
||||
--tool-call-parser muse \
|
||||
--watchdog-timeout 1800 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
```bash HiCache (concurrency 16 and above)
|
||||
--enable-hierarchical-cache \
|
||||
--hicache-size 253 \
|
||||
--hicache-write-policy write_through \
|
||||
--hicache-io-backend kernel \
|
||||
--hicache-mem-layout page_first
|
||||
```
|
||||
|
||||
## 1. Model introduction
|
||||
|
||||
This cookbook provides deployment recipes for the Muse Spark Open 816B-A42B checkpoints:
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user