[SMG-GO] implement a Go SGLang Model Gateway - OpenAI Compatible API Server (#14770)

2025-12-10 06:03:28 -08:00
parent 12b7a4fab0
commit 766476f52a
38 changed files with 9115 additions and 271 deletions
@@ -0,0 +1,121 @@
+/tmp/ShareGPT_V3_unfiltered_cleaned_split.json: 100%|████████████████████| 642M/642M [10:02<00:00, 1.12MB/s]
+#Input tokens: 50561
+#Output tokens: 25883
+Starting warmup with 5 sequences...
+Warmup completed with 5 sequences. Starting main benchmark run...
+
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    20.0
+Max request concurrency:                 20
+Successful requests:                     100
+Benchmark duration (s):                  107.24
+Total input tokens:                      50561
+Total input text tokens:                 50561
+Total input vision tokens:               0
+Total generated tokens:                  25883
+Total generated tokens (retokenized):    129591
+Request throughput (req/s):              0.93
+Input token throughput (tok/s):          471.48
+Output token throughput (tok/s):         241.36
+Total token throughput (tok/s):          712.84
+Concurrency:                             16.42
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   17609.46
+Median E2E Latency (ms):                 12343.82
+---------------Time to First Token----------------
+Mean TTFT (ms):                          190.71
+Median TTFT (ms):                        164.86
+P99 TTFT (ms):                           397.72
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          162.55
+Median TPOT (ms):                        63.51
+P99 TPOT (ms):                           1337.20
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           25.85
+Median ITL (ms):                         24.26
+P95 ITL (ms):                            48.26
+P99 ITL (ms):                            119.04
+Max ITL (ms):                            194.58
+==================================================
+
+✓ E2E test completed
+
+
+## Rust
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    20.0
+Max request concurrency:                 20
+Successful requests:                     100
+Benchmark duration (s):                  37.71
+Total input tokens:                      50561
+Total input text tokens:                 50561
+Total input vision tokens:               0
+Total generated tokens:                  25883
+Total generated tokens (retokenized):    25599
+Request throughput (req/s):              2.65
+Input token throughput (tok/s):          1340.75
+Output token throughput (tok/s):         686.35
+Total token throughput (tok/s):          2027.10
+Concurrency:                             18.58
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   7008.05
+Median E2E Latency (ms):                 7061.24
+---------------Time to First Token----------------
+Mean TTFT (ms):                          156.09
+Median TTFT (ms):                        133.81
+P99 TTFT (ms):                           318.53
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          26.59
+Median TPOT (ms):                        26.75
+P99 TPOT (ms):                           29.18
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           26.71
+Median ITL (ms):                         23.61
+P95 ITL (ms):                            66.11
+P99 ITL (ms):                            115.30
+Max ITL (ms):                            201.08
+==================================================
+
+
+## golang
+#Input tokens: 50561
+#Output tokens: 25883
+Starting warmup with 5 sequences...
+Warmup completed with 5 sequences. Starting main benchmark run...
+
+============ Serving Benchmark Result ============
+Backend:                                 sglang-oai-chat
+Traffic request rate:                    20.0
+Max request concurrency:                 20
+Successful requests:                     100
+Benchmark duration (s):                  34.22
+Total input tokens:                      50561
+Total input text tokens:                 50561
+Total input vision tokens:               0
+Total generated tokens:                  22970
+Total generated tokens (retokenized):    31740
+Request throughput (req/s):              2.92
+Input token throughput (tok/s):          1477.70
+Output token throughput (tok/s):         671.32
+Total token throughput (tok/s):          2149.03
+Concurrency:                             18.42
+----------------End-to-End Latency----------------
+Mean E2E Latency (ms):                   6303.33
+Median E2E Latency (ms):                 6294.46
+---------------Time to First Token----------------
+Mean TTFT (ms):                          157.10
+Median TTFT (ms):                        149.16
+P99 TTFT (ms):                           251.98
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          26.49
+Median TPOT (ms):                        27.15
+P99 TPOT (ms):                           28.73
+---------------Inter-Token Latency----------------
+Mean ITL (ms):                           26.97
+Median ITL (ms):                         24.61
+P95 ITL (ms):                            52.39
+P99 ITL (ms):                            86.52
+Max ITL (ms):                            194.55
+==================================================