diff --git a/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni.mdx b/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni.mdx index 89926ebac..492782b6b 100644 --- a/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni.mdx +++ b/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni.mdx @@ -478,7 +478,7 @@ Nemotron 3 Nano Omni achieves **9x higher throughput** than other open omni mode ### 5.2 Speed Benchmark **Test Environment:** -- Hardware: H100 (4×) +- Hardware: B200 (8×) - Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning - Tensor Parallelism: 4 - SGLang Version: main branch @@ -492,6 +492,7 @@ sglang serve \ --tp 4 \ --max-running-requests 1024 \ --host 0.0.0.0 \ + --attention-backend flashinfer \ --port 30000 ``` @@ -510,12 +511,52 @@ python3 -m sglang.bench_serving \ --max-concurrency 256 ``` +- **Test Results:** + +```text Output +============ Serving Benchmark Result ============ +Backend: sglang +Traffic request rate: inf +Max request concurrency: 256 +Successful requests: 4096 +Benchmark duration (s): 206.52 +Total input tokens: 2081726 +Total input text tokens: 2081726 +Total generated tokens: 2087288 +Total generated tokens (retokenized): 1945477 +Request throughput (req/s): 19.83 +Input token throughput (tok/s): 10080.25 +Output token throughput (tok/s): 10107.18 +Peak output token throughput (tok/s): 20199.00 +Peak concurrent requests: 291 +Total token throughput (tok/s): 20187.44 +Concurrency: 250.83 +----------------End-to-End Latency---------------- +Mean E2E Latency (ms): 12646.47 +Median E2E Latency (ms): 12371.84 +P90 E2E Latency (ms): 22889.81 +P99 E2E Latency (ms): 26528.70 +---------------Time to First Token---------------- +Mean TTFT (ms): 220.66 +Median TTFT (ms): 97.67 +P99 TTFT (ms): 2068.63 +-----Time per Output Token (excl. 1st token)------ +Mean TPOT (ms): 24.98 +Median TPOT (ms): 24.36 +P99 TPOT (ms): 44.97 +---------------Inter-Token Latency---------------- +Mean ITL (ms): 24.43 +Median ITL (ms): 10.91 +P95 ITL (ms): 62.68 +P99 ITL (ms): 100.60 +Max ITL (ms): 2171.93 +================================================== +``` + ### 5.3 Accuracy Benchmark -#### 5.3.1 GSM8K Benchmark - **Environment** -- Hardware: H100 (4×) +- Hardware: B200 (8×) - Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning - Tensor Parallelism: 4 - SGLang Version: main branch @@ -526,17 +567,91 @@ sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning \ --trust-remote-code \ --tp 4 \ + --attention-backend flashinfer \ --reasoning-parser deepseek-r1 ``` +#### 5.3.1 GSM8K Benchmark + **Run Benchmark** ```shell Command python3 benchmark/gsm8k/bench_sglang.py --port 30000 ``` +**Test Results:** +```text Output +Accuracy: 0.830 +Invalid: 0.000 +Latency: 13.970 s +Output throughput: 1611.623 token/s +``` + #### 5.3.2 MMLU Benchmark **Run Benchmark** ```shell Command python3 benchmark/mmlu/bench_sglang.py --port 30000 ``` + +**Test Results:** +```text Output +subject: abstract_algebra, #q:100, acc: 0.510 +subject: anatomy, #q:135, acc: 0.711 +subject: astronomy, #q:152, acc: 0.829 +subject: business_ethics, #q:100, acc: 0.760 +subject: clinical_knowledge, #q:265, acc: 0.781 +subject: college_biology, #q:144, acc: 0.854 +subject: college_chemistry, #q:100, acc: 0.560 +subject: college_computer_science, #q:100, acc: 0.700 +subject: college_mathematics, #q:100, acc: 0.590 +subject: college_medicine, #q:173, acc: 0.775 +subject: college_physics, #q:102, acc: 0.559 +subject: computer_security, #q:100, acc: 0.750 +subject: conceptual_physics, #q:235, acc: 0.821 +subject: econometrics, #q:114, acc: 0.605 +subject: electrical_engineering, #q:145, acc: 0.759 +subject: elementary_mathematics, #q:378, acc: 0.638 +subject: formal_logic, #q:126, acc: 0.524 +subject: global_facts, #q:100, acc: 0.400 +subject: high_school_biology, #q:310, acc: 0.906 +subject: high_school_chemistry, #q:203, acc: 0.759 +subject: high_school_computer_science, #q:100, acc: 0.860 +subject: high_school_european_history, #q:165, acc: 0.812 +subject: high_school_geography, #q:198, acc: 0.889 +subject: high_school_government_and_politics, #q:193, acc: 0.933 +subject: high_school_macroeconomics, #q:390, acc: 0.785 +subject: high_school_mathematics, #q:270, acc: 0.496 +subject: high_school_microeconomics, #q:238, acc: 0.887 +subject: high_school_physics, #q:151, acc: 0.675 +subject: high_school_psychology, #q:545, acc: 0.895 +subject: high_school_statistics, #q:216, acc: 0.731 +subject: high_school_us_history, #q:204, acc: 0.858 +subject: high_school_world_history, #q:237, acc: 0.873 +subject: human_aging, #q:223, acc: 0.740 +subject: human_sexuality, #q:131, acc: 0.855 +subject: international_law, #q:121, acc: 0.851 +subject: jurisprudence, #q:108, acc: 0.815 +subject: logical_fallacies, #q:163, acc: 0.847 +subject: machine_learning, #q:112, acc: 0.598 +subject: management, #q:103, acc: 0.864 +subject: marketing, #q:234, acc: 0.910 +subject: medical_genetics, #q:100, acc: 0.880 +subject: miscellaneous, #q:783, acc: 0.881 +subject: moral_disputes, #q:346, acc: 0.780 +subject: moral_scenarios, #q:895, acc: 0.543 +subject: nutrition, #q:306, acc: 0.814 +subject: philosophy, #q:311, acc: 0.733 +subject: prehistory, #q:324, acc: 0.852 +subject: professional_accounting, #q:282, acc: 0.553 +subject: professional_law, #q:1534, acc: 0.565 +subject: professional_medicine, #q:272, acc: 0.779 +subject: professional_psychology, #q:612, acc: 0.760 +subject: public_relations, #q:110, acc: 0.709 +subject: security_studies, #q:245, acc: 0.759 +subject: sociology, #q:201, acc: 0.831 +subject: us_foreign_policy, #q:100, acc: 0.910 +subject: virology, #q:166, acc: 0.560 +subject: world_religions, #q:171, acc: 0.807 +Total latency: 67.512 +Average accuracy: 0.737 +```