minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[SPEC] feat: add adaptive speculative decoding metrics (#25940)

Browse Source 
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Jarrod Barnes <jbarnes850@gmail.com>
This commit is contained in:
shuwenn
2026-06-01 13:53:30 -07:00
committed by GitHub
co-authored by Claude Opus 4.7 Jarrod Barnes
parent 106092123f
commit a0670b5ba3
4 changed files with 92 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/references/production_metrics.mdx
+6
View File
@@ -126,6 +126,12 @@ sglang:gen_throughput{model_name="meta-llama/Llama-3.1-8B-Instruct"} 86.50814177
# HELP sglang:num_queue_reqs The number of requests in the waiting queue
# TYPE sglang:num_queue_reqs gauge
sglang:num_queue_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2826.0
# HELP sglang:spec_num_steps Currently active speculative_num_steps.
# TYPE sglang:spec_num_steps gauge
sglang:spec_num_steps{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0
# HELP sglang:spec_num_draft_tokens Currently active speculative_num_draft_tokens (decouples from steps under topk>1).
# TYPE sglang:spec_num_draft_tokens gauge
sglang:spec_num_draft_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.0
```
## Setup Guide