MFU metrics in Prometheus (#19395)

docs/advanced_features/server_arguments.md

@@ -185,6 +185,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
 | `--crash-dump-folder` | Folder path to dump requests from the last 5 min before a crash (if any). If not specified, crash dumping is disabled. | `None` | Type: str |
 | `--show-time-cost` | Show time cost of custom marks. | `False` | bool flag (set to enable) |
 | `--enable-metrics` | Enable log prometheus metrics. | `False` | bool flag (set to enable) |
+| `--enable-mfu-metrics` | Enable estimated MFU-related prometheus metrics. | `False` | bool flag (set to enable) |
 | `--enable-metrics-for-all-schedulers` | Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp_attention is enabled, as otherwise all metrics appear to come from TP 0. | `False` | bool flag (set to enable) |
 | `--tokenizer-metrics-custom-labels-header` | Specify the HTTP header for passing custom labels for tokenizer metrics. | `x-custom-labels` | Type: str |
 | `--tokenizer-metrics-allowed-custom-labels` | The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., {'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set. | `None` | List[str] |

docs/references/production_metrics.md

@@ -142,7 +142,8 @@ This section describes how to set up the monitoring stack (Prometheus + Grafana)
     python -m sglang.launch_server \
       --model-path <your_model_path> \
       --port 30000 \
-      --enable-metrics
+      --enable-metrics \
+      --enable-mfu-metrics
     ```
     Replace `<your_model_path>` with the actual path to your model (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct`). Ensure the server is accessible from the monitoring stack (you might need `--host 0.0.0.0` if running in Docker). By default, the metrics endpoint will be available at `http://<sglang_server_host>:30000/metrics`.
 
@@ -229,3 +230,38 @@ python3 -m sglang.bench_serving \
 to generate some requests.
 
 Then you should be able to see the metrics in the Grafana dashboard.
+
+## Estimated Performance Metrics (MFU-related)
+
+SGLang exports the following estimated per-GPU counters that can be used to derive
+Model FLOPs Utilization (MFU)-related signals:
+
+- `sglang:estimated_flops_per_gpu_total`: Estimated floating-point operations.
+- `sglang:estimated_read_bytes_per_gpu_total`: Estimated bytes read from memory.
+- `sglang:estimated_write_bytes_per_gpu_total`: Estimated bytes written to memory.
+
+These metrics are available when both `--enable-metrics` and
+`--enable-mfu-metrics` are enabled.
+
+These are cumulative counters. Use Prometheus `rate(...)` to get per-second values.
+
+### PromQL examples
+
+Average TFLOPS per GPU:
+
+```promql
+rate(sglang:estimated_flops_per_gpu_total[1m]) / 1e12
+```
+
+Average estimated memory bandwidth in GB/s:
+
+```promql
+(rate(sglang:estimated_read_bytes_per_gpu_total[1m]) +
+ rate(sglang:estimated_write_bytes_per_gpu_total[1m])) / 1e9
+```
+
+### Notes
+
+- These metrics are estimates intended for observability and trend analysis.
+- Estimated memory bytes reflect modeled traffic and are not a direct hardware
+  counter from GPU profilers.