minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[model-gateway] change rust package name to sgl-model-gateway instead (#14283)

Browse Source
This commit is contained in:
Simo Lin
2025-12-02 00:06:54 -08:00
committed by GitHub
parent 64092c8b55
commit 51a86ce69e
56 changed files with 167 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sgl-router/Cargo.toml
+3 -4
View File
@@ -1,5 +1,5 @@
[package] [package]
name = "sglang-router" name = "sgl-model-gateway"
version = "0.2.3" version = "0.2.3"
edition = "2021" edition = "2021"
@@ -11,12 +11,11 @@ vendored-openssl = ["openssl/vendored"]
unused_qualifications = "warn" unused_qualifications = "warn"
[lib] [lib]
name = "sglang_router_rs" name = "sgl_model_gateway"
# Pure Rust library (rlib for the binary and other Rust dependents)
crate-type = ["rlib"] crate-type = ["rlib"]
[[bin]] [[bin]]
name = "sglang-router" name = "sgl-model-gateway"
path = "src/main.rs" path = "src/main.rs"
[[bin]] [[bin]]
sgl-router/README.md
+20 -12
View File
@@ -93,20 +93,28 @@ pip install --force-reinstall dist/*.whl
After installation, verify the installation and check version information: After installation, verify the installation and check version information:
```bash ```bash
# Short version info (Rust binary) # Simple version (Rust binary)
./target/release/sglang-router -v ./target/release/sgl-model-gateway --version
# or use aliases
./target/release/smg --version
./target/release/amg --version
# Full version info with build details (Rust binary) # Full version info with build details
./target/release/sglang-router --version ./target/release/sgl-model-gateway --version-verbose
# Python CLI
amg --version
amg --version-verbose
python3 -m sglang_router --version
``` ```
The `-v` flag displays a concise version string, while `--version` (or `-V`) shows comprehensive build information including Git commit, build time, compiler versions, and platform details. The `--version` (or `-V`) flag displays the version string. Use `--version-verbose` for comprehensive build information including Git commit, build time, compiler versions, and platform details.
## Quick Start ## Quick Start
### Regular HTTP Routing ### Regular HTTP Routing
- **Rust binary** - **Rust binary**
```bash ```bash
./target/release/sglang-router \ ./target/release/sgl-model-gateway \
--worker-urls http://worker1:8000 http://worker2:8000 \ --worker-urls http://worker1:8000 http://worker2:8000 \
--policy cache_aware --policy cache_aware
``` ```
@@ -121,7 +129,7 @@ The `-v` flag displays a concise version string, while `--version` (or `-V`) sho
### Prefill/Decode Disaggregation (PD) ### Prefill/Decode Disaggregation (PD)
- **Rust binary** - **Rust binary**
```bash ```bash
./target/release/sglang-router \ ./target/release/sgl-model-gateway \
--pd-disaggregation \ --pd-disaggregation \
--prefill http://prefill1:30001 9001 \ --prefill http://prefill1:30001 9001 \
--prefill http://prefill2:30002 \ --prefill http://prefill2:30002 \
@@ -146,7 +154,7 @@ Prefill entries accept an optional bootstrap port. PD mode merges prefill metada
### Multi-Model Inference Gateway ### Multi-Model Inference Gateway
Enable IGW mode to route multiple models through a single router while applying per-model policies: Enable IGW mode to route multiple models through a single router while applying per-model policies:
```bash ```bash
./target/release/sglang-router \ ./target/release/sgl-model-gateway \
--enable-igw \ --enable-igw \
--policy cache_aware \ --policy cache_aware \
--max-concurrent-requests 512 --max-concurrent-requests 512
@@ -194,7 +202,7 @@ Add more workers with the same API; include optional `labels` (for per-model pol
### gRPC Routing ### gRPC Routing
- **Rust binary** - **Rust binary**
```bash ```bash
./target/release/sglang-router \ ./target/release/sgl-model-gateway \
--worker-urls grpc://worker-grpc-0:31001 grpc://worker-grpc-1:31002 \ --worker-urls grpc://worker-grpc-0:31001 grpc://worker-grpc-1:31002 \
--tokenizer-path /path/to/tokenizer.json \ --tokenizer-path /path/to/tokenizer.json \
--reasoning-parser deepseek-r1 \ --reasoning-parser deepseek-r1 \
@@ -229,7 +237,7 @@ python3 -m sglang_router.launch_router \
**Notes** **Notes**
- OpenAI backend mode acts as a proxy to a single remote endpoint; load balancing is not applied. - OpenAI backend mode acts as a proxy to a single remote endpoint; load balancing is not applied.
- Provide exactly one `--worker-urls` entry per router instance. - Provide exactly one `--worker-urls` entry per router instance.
- The Rust binary supports the same flags (`./target/release/sglang-router --backend openai ...`). - The Rust binary supports the same flags (`./target/release/sgl-model-gateway --backend openai ...`).
### MCP Integration ### MCP Integration
The SGL Model Gateway provides native Model Context Protocol (MCP) client integration, enabling tool calling across STDIO, SSE, and Streamable transports. MCP servers are configured via a YAML configuration file and registered at startup through the workflow engine. The SGL Model Gateway provides native Model Context Protocol (MCP) client integration, enabling tool calling across STDIO, SSE, and Streamable transports. MCP servers are configured via a YAML configuration file and registered at startup through the workflow engine.
@@ -237,7 +245,7 @@ The SGL Model Gateway provides native Model Context Protocol (MCP) client integr
#### Basic Usage #### Basic Usage
```bash ```bash
# Rust binary # Rust binary
./target/release/sglang-router \ ./target/release/sgl-model-gateway \
--mcp-config-path /path/to/mcp-config.yaml \ --mcp-config-path /path/to/mcp-config.yaml \
--worker-urls http://worker1:8000 --worker-urls http://worker1:8000
@@ -407,7 +415,7 @@ All administrative routes inherit router API-key protection when `--api-key` is
### Service Discovery ### Service Discovery
Enable Kubernetes discovery to reconcile workers automatically: Enable Kubernetes discovery to reconcile workers automatically:
```bash ```bash
./target/release/sglang-router \ ./target/release/sgl-model-gateway \
--service-discovery \ --service-discovery \
--selector app=sglang-worker role=inference \ --selector app=sglang-worker role=inference \
--service-discovery-namespace sglang-system \ --service-discovery-namespace sglang-system \
sgl-router/benches/request_processing.rs
+1 -1
View File
@@ -2,7 +2,7 @@ use std::time::Instant;
use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput};
use serde_json::{from_str, to_string, to_value, to_vec}; use serde_json::{from_str, to_string, to_value, to_vec};
use sglang_router_rs::{ use sgl_model_gateway::{
core::{BasicWorker, BasicWorkerBuilder, Worker, WorkerType}, core::{BasicWorker, BasicWorkerBuilder, Worker, WorkerType},
protocols::{ protocols::{
chat::{ChatCompletionRequest, ChatMessage, MessageContent}, chat::{ChatCompletionRequest, ChatMessage, MessageContent},
sgl-router/benches/tokenizer_benchmark.rs
+2 -2
View File
@@ -13,7 +13,7 @@ use std::{
}; };
use criterion::{black_box, criterion_group, BenchmarkId, Criterion, Throughput}; use criterion::{black_box, criterion_group, BenchmarkId, Criterion, Throughput};
use sglang_router_rs::tokenizer::{ use sgl_model_gateway::tokenizer::{
cache::{CacheConfig, CachedTokenizer}, cache::{CacheConfig, CachedTokenizer},
huggingface::HuggingFaceTokenizer, huggingface::HuggingFaceTokenizer,
sequence::Sequence, sequence::Sequence,
@@ -31,7 +31,7 @@ fn get_tokenizer_path() -> &'static PathBuf {
// with special: true, normalized: false - perfect for demonstrating L1 cache // with special: true, normalized: false - perfect for demonstrating L1 cache
let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime"); let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime");
let tokenizer_dir = rt.block_on(async { let tokenizer_dir = rt.block_on(async {
sglang_router_rs::tokenizer::hub::download_tokenizer_from_hf( sgl_model_gateway::tokenizer::hub::download_tokenizer_from_hf(
"Qwen/Qwen3-4B-Instruct-2507", "Qwen/Qwen3-4B-Instruct-2507",
) )
.await .await
sgl-router/benches/tool_parser_benchmark.rs
+1 -1
View File
@@ -19,7 +19,7 @@ use std::{
use criterion::{black_box, criterion_group, BenchmarkId, Criterion, Throughput}; use criterion::{black_box, criterion_group, BenchmarkId, Criterion, Throughput};
use serde_json::json; use serde_json::json;
use sglang_router_rs::{ use sgl_model_gateway::{
protocols::common::{Function, Tool}, protocols::common::{Function, Tool},
tool_parser::{JsonParser, ParserFactory as ToolParserFactory, ToolParser}, tool_parser::{JsonParser, ParserFactory as ToolParserFactory, ToolParser},
}; };
sgl-router/bindings/golang/Cargo.toml
+5 -6
View File
@@ -1,10 +1,10 @@
[package] [package]
name = "sglang-router-golang" name = "sgl-model-gateway-golang"
version = "0.2.2" version = "0.2.3"
edition = "2021" edition = "2021"
[lib] [lib]
name = "sglang_router_rs" name = "sgl_model_gateway_go"
crate-type = ["cdylib"] crate-type = ["cdylib"]
[dependencies] [dependencies]
@@ -18,14 +18,13 @@ once_cell = "1.21.3"
futures-util = "0.3" futures-util = "0.3"
tracing = "0.1" tracing = "0.1"
[dependencies.sglang-router] [dependencies.sgl-model-gateway]
path = "../.." path = "../.."
default-features = true default-features = true
package = "sglang-router"
[features] [features]
default = [] default = []
vendored-openssl = ["sglang-router/vendored-openssl"] vendored-openssl = ["sgl-model-gateway/vendored-openssl"]
[profile.release] [profile.release]
opt-level = "z" # Optimize for size opt-level = "z" # Optimize for size
sgl-router/bindings/golang/src/client.rs
+5 -5
View File
@@ -8,11 +8,11 @@ use tokio::runtime::Runtime;
use once_cell::sync::Lazy; use once_cell::sync::Lazy;
use uuid::Uuid; use uuid::Uuid;
use sglang_router::tokenizer::create_tokenizer_from_file; use sgl_model_gateway::tokenizer::create_tokenizer_from_file;
use sglang_router::tokenizer::traits::Tokenizer; use sgl_model_gateway::tokenizer::traits::Tokenizer;
use sglang_router::grpc_client::sglang_scheduler::SglangSchedulerClient; use sgl_model_gateway::grpc_client::sglang_scheduler::SglangSchedulerClient;
use sglang_router::protocols::chat::ChatCompletionRequest; use sgl_model_gateway::protocols::chat::ChatCompletionRequest;
use sglang_router::routers::grpc::utils::{process_chat_messages, generate_tool_constraints}; use sgl_model_gateway::routers::grpc::utils::{process_chat_messages, generate_tool_constraints};
use super::error::{SglErrorCode, set_error_message}; use super::error::{SglErrorCode, set_error_message};
use super::grpc_converter::sgl_grpc_response_converter_create; use super::grpc_converter::sgl_grpc_response_converter_create;
sgl-router/bindings/golang/src/grpc_converter.rs
+17 -17
View File
@@ -9,12 +9,12 @@ use serde_json::Value;
use tokio::runtime::Runtime; use tokio::runtime::Runtime;
use once_cell::sync::Lazy; use once_cell::sync::Lazy;
use sglang_router::tokenizer::traits::Tokenizer; use sgl_model_gateway::tokenizer::traits::Tokenizer;
use sglang_router::tokenizer::stream::DecodeStream; use sgl_model_gateway::tokenizer::stream::DecodeStream;
use sglang_router::tool_parser::ToolParser; use sgl_model_gateway::tool_parser::ToolParser;
use sglang_router::protocols::common::{Tool, ToolChoice, ToolChoiceValue, ToolCallDelta, FunctionCallDelta, Usage, StringOrArray}; use sgl_model_gateway::protocols::common::{Tool, ToolChoice, ToolChoiceValue, ToolCallDelta, FunctionCallDelta, Usage, StringOrArray};
use sglang_router::tokenizer::stop::StopSequenceDecoder; use sgl_model_gateway::tokenizer::stop::StopSequenceDecoder;
use sglang_router::grpc_client::sglang_proto as proto; use sgl_model_gateway::grpc_client::sglang_proto as proto;
use super::error::{SglErrorCode, set_error_message, clear_error_message}; use super::error::{SglErrorCode, set_error_message, clear_error_message};
use super::tokenizer::TokenizerHandle; use super::tokenizer::TokenizerHandle;
@@ -22,9 +22,9 @@ use super::utils::generate_tool_call_id;
/// Global parser factory (initialized once) /// Global parser factory (initialized once)
// Use the re-exported ParserFactory from tool_parser module // Use the re-exported ParserFactory from tool_parser module
static PARSER_FACTORY: Lazy<sglang_router::tool_parser::ParserFactory> = Lazy::new(|| { static PARSER_FACTORY: Lazy<sgl_model_gateway::tool_parser::ParserFactory> = Lazy::new(|| {
// ParserFactory is re-exported from tool_parser::factory, so we can use it directly // ParserFactory is re-exported from tool_parser::factory, so we can use it directly
sglang_router::tool_parser::ParserFactory::default() sgl_model_gateway::tool_parser::ParserFactory::default()
}); });
/// Global tokio runtime for async operations /// Global tokio runtime for async operations
@@ -151,7 +151,7 @@ pub unsafe extern "C" fn sgl_grpc_response_converter_create(
// Create stop decoder if needed // Create stop decoder if needed
let stop_decoder = if stop.is_some() || stop_token_ids.is_some() { let stop_decoder = if stop.is_some() || stop_token_ids.is_some() {
Some(Arc::new(tokio::sync::Mutex::new( Some(Arc::new(tokio::sync::Mutex::new(
sglang_router::routers::grpc::utils::create_stop_decoder( sgl_model_gateway::routers::grpc::utils::create_stop_decoder(
&tokenizer, &tokenizer,
stop.as_ref(), stop.as_ref(),
stop_token_ids.as_ref(), stop_token_ids.as_ref(),
@@ -389,9 +389,9 @@ pub(crate) async fn convert_proto_chunk_to_openai(
request_id: &str, request_id: &str,
created: u64, created: u64,
system_fingerprint: Option<&str>, system_fingerprint: Option<&str>,
) -> Result<Option<sglang_router::protocols::chat::ChatCompletionStreamResponse>, String> { ) -> Result<Option<sgl_model_gateway::protocols::chat::ChatCompletionStreamResponse>, String> {
use sglang_router::grpc_client::sglang_proto::generate_response::Response::*; use sgl_model_gateway::grpc_client::sglang_proto::generate_response::Response::*;
use sglang_router::protocols::chat::{ChatCompletionStreamResponse, ChatMessageDelta, ChatStreamChoice}; use sgl_model_gateway::protocols::chat::{ChatCompletionStreamResponse, ChatMessageDelta, ChatStreamChoice};
match proto_response.response { match proto_response.response {
Some(Chunk(chunk)) => { Some(Chunk(chunk)) => {
@@ -427,19 +427,19 @@ pub(crate) async fn convert_proto_chunk_to_openai(
let mut text = String::new(); let mut text = String::new();
for &token_id in &chunk.token_ids { for &token_id in &chunk.token_ids {
match decoder_guard.process_token(token_id).unwrap_or_else(|_| { match decoder_guard.process_token(token_id).unwrap_or_else(|_| {
sglang_router::tokenizer::stop::SequenceDecoderOutput::Held sgl_model_gateway::tokenizer::stop::SequenceDecoderOutput::Held
}) { }) {
sglang_router::tokenizer::stop::SequenceDecoderOutput::Text(t) => { sgl_model_gateway::tokenizer::stop::SequenceDecoderOutput::Text(t) => {
text.push_str(&t); text.push_str(&t);
} }
sglang_router::tokenizer::stop::SequenceDecoderOutput::StoppedWithText(t) => { sgl_model_gateway::tokenizer::stop::SequenceDecoderOutput::StoppedWithText(t) => {
text.push_str(&t); text.push_str(&t);
break; break;
} }
sglang_router::tokenizer::stop::SequenceDecoderOutput::Stopped => { sgl_model_gateway::tokenizer::stop::SequenceDecoderOutput::Stopped => {
break; break;
} }
sglang_router::tokenizer::stop::SequenceDecoderOutput::Held => {} sgl_model_gateway::tokenizer::stop::SequenceDecoderOutput::Held => {}
} }
} }
text text
sgl-router/bindings/golang/src/stream.rs
+1 -1
View File
@@ -23,7 +23,7 @@ use tokio::runtime::Runtime;
use once_cell::sync::Lazy; use once_cell::sync::Lazy;
use futures_util::StreamExt; use futures_util::StreamExt;
use sglang_router::grpc_client::{sglang_proto as proto, sglang_scheduler::{SglangSchedulerClient, AbortOnDropStream}}; use sgl_model_gateway::grpc_client::{sglang_proto as proto, sglang_scheduler::{SglangSchedulerClient, AbortOnDropStream}};
use super::error::{SglErrorCode, set_error_message}; use super::error::{SglErrorCode, set_error_message};
use super::grpc_converter::{GrpcResponseConverterHandle, convert_proto_chunk_to_openai}; use super::grpc_converter::{GrpcResponseConverterHandle, convert_proto_chunk_to_openai};
sgl-router/bindings/golang/src/tokenizer.rs
+1 -1
View File
@@ -6,7 +6,7 @@ use std::ptr;
use std::sync::Arc; use std::sync::Arc;
use serde_json::Value; use serde_json::Value;
use sglang_router::tokenizer::{ use sgl_model_gateway::tokenizer::{
create_tokenizer_from_file, create_tokenizer_from_file,
traits::Tokenizer as TokenizerTrait, traits::Tokenizer as TokenizerTrait,
chat_template::ChatTemplateParams, chat_template::ChatTemplateParams,
sgl-router/bindings/golang/src/tool_parser.rs
+2 -2
View File
@@ -9,8 +9,8 @@ use serde_json::{json, Value};
use tokio::runtime::Runtime; use tokio::runtime::Runtime;
use once_cell::sync::Lazy; use once_cell::sync::Lazy;
use sglang_router::tool_parser::{ParserFactory, ToolParser}; use sgl_model_gateway::tool_parser::{ParserFactory, ToolParser};
use sglang_router::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
use super::error::{SglErrorCode, set_error_message, clear_error_message}; use super::error::{SglErrorCode, set_error_message, clear_error_message};
use super::utils::generate_tool_call_id; use super::utils::generate_tool_call_id;
sgl-router/bindings/python/Cargo.toml
+4 -5
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "sglang-router-python" name = "sgl-model-gateway-python"
version = "0.2.2" version = "0.2.3"
edition = "2021" edition = "2021"
[lib] [lib]
@@ -11,14 +11,13 @@ crate-type = ["cdylib"]
pyo3 = { version = "0.27.1", features = ["extension-module", "abi3-py38"] } pyo3 = { version = "0.27.1", features = ["extension-module", "abi3-py38"] }
tokio = { version = "1.42.0", features = ["full"] } tokio = { version = "1.42.0", features = ["full"] }
[dependencies.sglang-router] [dependencies.sgl-model-gateway]
path = "../.." path = "../.."
default-features = true default-features = true
package = "sglang-router"
[features] [features]
default = ["pyo3/extension-module"] default = ["pyo3/extension-module"]
vendored-openssl = ["sglang-router/vendored-openssl"] vendored-openssl = ["sgl-model-gateway/vendored-openssl"]
[profile.ci] [profile.ci]
inherits = "release" inherits = "release"
sgl-router/bindings/python/src/lib.rs
+1 -1
View File
@@ -1,5 +1,5 @@
use pyo3::prelude::*; use pyo3::prelude::*;
use sglang_router::*; use sgl_model_gateway::*;
use std::collections::HashMap; use std::collections::HashMap;
// Define the enums with PyO3 bindings // Define the enums with PyO3 bindings
sgl-router/src/core/model_card.rs
+1 -1
View File
@@ -90,7 +90,7 @@ impl std::fmt::Display for ProviderType {
/// # Example /// # Example
/// ///
/// ``` /// ```
/// use sglang_router_rs::core::{ModelCard, ModelType, ProviderType}; /// use sgl_model_gateway::core::{ModelCard, ModelType, ProviderType};
/// ///
/// let card = ModelCard::new("meta-llama/Llama-3.1-8B-Instruct") /// let card = ModelCard::new("meta-llama/Llama-3.1-8B-Instruct")
/// .with_display_name("Llama 3.1 8B Instruct") /// .with_display_name("Llama 3.1 8B Instruct")
sgl-router/src/logging.rs
+2 -2
View File
@@ -28,7 +28,7 @@ impl Default for LoggingConfig {
log_dir: None, log_dir: None,
colorize: true, colorize: true,
log_file_name: "sgl-router".to_string(), log_file_name: "sgl-router".to_string(),
log_targets: Some(vec!["sglang_router_rs".to_string()]), log_targets: Some(vec!["sgl_model_gateway".to_string()]),
} }
} }
} }
@@ -63,7 +63,7 @@ pub fn init_logging(config: LoggingConfig) -> LogGuard {
}) })
.collect::<String>() .collect::<String>()
} else { } else {
format!("sglang_router_rs={}", level_filter) format!("sgl_model_gateway={}", level_filter)
}; };
EnvFilter::new(filter_string) EnvFilter::new(filter_string)
sgl-router/src/main.rs
+1 -1
View File
@@ -1,7 +1,7 @@
use std::collections::HashMap; use std::collections::HashMap;
use clap::{ArgAction, Parser, Subcommand, ValueEnum}; use clap::{ArgAction, Parser, Subcommand, ValueEnum};
use sglang_router_rs::{ use sgl_model_gateway::{
config::{ config::{
CircuitBreakerConfig, ConfigError, ConfigResult, DiscoveryConfig, HealthCheckConfig, CircuitBreakerConfig, ConfigError, ConfigResult, DiscoveryConfig, HealthCheckConfig,
HistoryBackend, MetricsConfig, OracleConfig, PolicyConfig, PostgresConfig, RetryConfig, HistoryBackend, MetricsConfig, OracleConfig, PolicyConfig, PostgresConfig, RetryConfig,
sgl-router/src/middleware.rs
+7 -7
View File
@@ -222,7 +222,7 @@ impl<B> OnRequest<B> for RequestLogger {
// Log the request start // Log the request start
info!( info!(
target: "sglang_router_rs::request", target: "sgl_model_gateway::request",
"started processing request" "started processing request"
); );
} }
@@ -254,17 +254,17 @@ impl<B> OnResponse<B> for ResponseLogger {
let _enter = span.enter(); let _enter = span.enter();
if status.is_server_error() { if status.is_server_error() {
error!( error!(
target: "sglang_router_rs::response", target: "sgl_model_gateway::response",
"request failed with server error" "request failed with server error"
); );
} else if status.is_client_error() { } else if status.is_client_error() {
warn!( warn!(
target: "sglang_router_rs::response", target: "sgl_model_gateway::response",
"request failed with client error" "request failed with client error"
); );
} else { } else {
info!( info!(
target: "sglang_router_rs::response", target: "sgl_model_gateway::response",
"finished processing request" "finished processing request"
); );
} }
@@ -303,7 +303,7 @@ pub struct RequestLogEntry {
pub fn log_request(entry: RequestLogEntry) { pub fn log_request(entry: RequestLogEntry) {
if entry.status >= 500 { if entry.status >= 500 {
tracing::error!( tracing::error!(
target: "sglang_router_rs::http", target: "sgl_model_gateway::http",
request_id = %entry.request_id, request_id = %entry.request_id,
method = %entry.method, method = %entry.method,
uri = %entry.uri, uri = %entry.uri,
@@ -316,7 +316,7 @@ pub fn log_request(entry: RequestLogEntry) {
); );
} else if entry.status >= 400 { } else if entry.status >= 400 {
tracing::warn!( tracing::warn!(
target: "sglang_router_rs::http", target: "sgl_model_gateway::http",
request_id = %entry.request_id, request_id = %entry.request_id,
method = %entry.method, method = %entry.method,
uri = %entry.uri, uri = %entry.uri,
@@ -328,7 +328,7 @@ pub fn log_request(entry: RequestLogEntry) {
); );
} else { } else {
tracing::info!( tracing::info!(
target: "sglang_router_rs::http", target: "sgl_model_gateway::http",
request_id = %entry.request_id, request_id = %entry.request_id,
method = %entry.method, method = %entry.method,
uri = %entry.uri, uri = %entry.uri,
sgl-router/src/routers/grpc/harmony/mod.rs
+1 -1
View File
@@ -18,7 +18,7 @@
//! ## Usage //! ## Usage
//! //!
//! ```ignore //! ```ignore
//! use sglang_router_rs::routers::grpc::harmony::{HarmonyDetector, HarmonyBuilder}; //! use sgl_model_gateway::routers::grpc::harmony::{HarmonyDetector, HarmonyBuilder};
//! //!
//! // Detect if model supports Harmony //! // Detect if model supports Harmony
//! if HarmonyDetector::is_harmony_model("gpt-4o") { //! if HarmonyDetector::is_harmony_model("gpt-4o") {
sgl-router/src/tokenizer/README.md
+2 -2
View File
@@ -138,7 +138,7 @@ as of `sgl-router/src/tokenizer/*`.
## Usage Examples ## Usage Examples
```rust ```rust
use std::sync::Arc; use std::sync::Arc;
use sglang_router_rs::tokenizer::{ use sgl_model_gateway::tokenizer::{
create_tokenizer, SequenceDecoderOutput, StopSequenceDecoderBuilder, Tokenizer, create_tokenizer, SequenceDecoderOutput, StopSequenceDecoderBuilder, Tokenizer,
}; };
@@ -172,7 +172,7 @@ for &token in encoding.token_ids() {
```rust ```rust
// Apply a chat template when one is bundled with the tokenizer // Apply a chat template when one is bundled with the tokenizer
use sglang_router_rs::tokenizer::{chat_template::ChatTemplateParams, HuggingFaceTokenizer}; use sgl_model_gateway::tokenizer::{chat_template::ChatTemplateParams, HuggingFaceTokenizer};
let mut hf = HuggingFaceTokenizer::from_file_with_chat_template( let mut hf = HuggingFaceTokenizer::from_file_with_chat_template(
"./tokenizer.json", "./tokenizer.json",
sgl-router/tests/api_endpoints_test.rs
+2 -2
View File
@@ -10,7 +10,7 @@ use axum::{
use common::mock_worker::{HealthStatus, MockWorker, MockWorkerConfig, WorkerType}; use common::mock_worker::{HealthStatus, MockWorker, MockWorkerConfig, WorkerType};
use reqwest::Client; use reqwest::Client;
use serde_json::json; use serde_json::json;
use sglang_router_rs::{ use sgl_model_gateway::{
app_context::AppContext, app_context::AppContext,
config::{RouterConfig, RoutingMode}, config::{RouterConfig, RoutingMode},
core::Job, core::Job,
@@ -1028,7 +1028,7 @@ mod responses_endpoint_tests {
let app = ctx.create_app().await; let app = ctx.create_app().await;
// Directly store a response in the storage to test the retrieval endpoint // Directly store a response in the storage to test the retrieval endpoint
use sglang_router_rs::data_connector::{ResponseId, StoredResponse}; use sgl_model_gateway::data_connector::{ResponseId, StoredResponse};
let mut stored_response = StoredResponse::new(None); let mut stored_response = StoredResponse::new(None);
stored_response.id = ResponseId::from("resp_test_input_items"); stored_response.id = ResponseId::from("resp_test_input_items");
stored_response.input = json!([ stored_response.input = json!([
sgl-router/tests/cache_aware_backward_compat_test.rs
+1 -1
View File
@@ -1,6 +1,6 @@
use std::{collections::HashMap, sync::Arc}; use std::{collections::HashMap, sync::Arc};
use sglang_router_rs::{ use sgl_model_gateway::{
core::{BasicWorkerBuilder, Worker, WorkerType}, core::{BasicWorkerBuilder, Worker, WorkerType},
policies::{CacheAwareConfig, CacheAwarePolicy, LoadBalancingPolicy}, policies::{CacheAwareConfig, CacheAwarePolicy, LoadBalancingPolicy},
}; };
sgl-router/tests/chat_template_format_detection.rs
+1 -1
View File
@@ -1,4 +1,4 @@
use sglang_router_rs::{ use sgl_model_gateway::{
protocols::chat::{ChatMessage, MessageContent}, protocols::chat::{ChatMessage, MessageContent},
tokenizer::chat_template::{ tokenizer::chat_template::{
detect_chat_template_content_format, ChatTemplateContentFormat, ChatTemplateParams, detect_chat_template_content_format, ChatTemplateContentFormat, ChatTemplateParams,
sgl-router/tests/chat_template_integration.rs
+1 -1
View File
@@ -1,4 +1,4 @@
use sglang_router_rs::{ use sgl_model_gateway::{
protocols::{ protocols::{
chat::{ChatMessage, MessageContent}, chat::{ChatMessage, MessageContent},
common::{ContentPart, ImageUrl}, common::{ContentPart, ImageUrl},
sgl-router/tests/chat_template_loading.rs
+2 -2
View File
@@ -2,7 +2,7 @@
mod tests { mod tests {
use std::fs; use std::fs;
use sglang_router_rs::{ use sgl_model_gateway::{
protocols::chat::{ChatMessage, MessageContent}, protocols::chat::{ChatMessage, MessageContent},
tokenizer::{chat_template::ChatTemplateParams, huggingface::HuggingFaceTokenizer}, tokenizer::{chat_template::ChatTemplateParams, huggingface::HuggingFaceTokenizer},
}; };
@@ -78,7 +78,7 @@ mod tests {
.map(|msg| serde_json::to_value(msg).unwrap()) .map(|msg| serde_json::to_value(msg).unwrap())
.collect(); .collect();
use sglang_router_rs::tokenizer::chat_template::ChatTemplateParams; use sgl_model_gateway::tokenizer::chat_template::ChatTemplateParams;
let params = ChatTemplateParams { let params = ChatTemplateParams {
add_generation_prompt: true, add_generation_prompt: true,
..Default::default() ..Default::default()
sgl-router/tests/common/mod.rs
+9 -9
View File
@@ -14,7 +14,7 @@ use std::{
}; };
use serde_json::json; use serde_json::json;
use sglang_router_rs::{ use sgl_model_gateway::{
app_context::AppContext, app_context::AppContext,
config::RouterConfig, config::RouterConfig,
core::{LoadMonitor, WorkerRegistry}, core::{LoadMonitor, WorkerRegistry},
@@ -90,8 +90,8 @@ pub async fn create_test_context(config: RouterConfig) -> Arc<AppContext> {
// Initialize JobQueue after AppContext is created // Initialize JobQueue after AppContext is created
let weak_context = Arc::downgrade(&app_context); let weak_context = Arc::downgrade(&app_context);
let job_queue = sglang_router_rs::core::JobQueue::new( let job_queue = sgl_model_gateway::core::JobQueue::new(
sglang_router_rs::core::JobQueueConfig::default(), sgl_model_gateway::core::JobQueueConfig::default(),
weak_context, weak_context,
); );
app_context app_context
@@ -100,7 +100,7 @@ pub async fn create_test_context(config: RouterConfig) -> Arc<AppContext> {
.expect("JobQueue should only be initialized once"); .expect("JobQueue should only be initialized once");
// Initialize WorkflowEngine and register workflows // Initialize WorkflowEngine and register workflows
use sglang_router_rs::core::workflow::{ use sgl_model_gateway::core::workflow::{
create_worker_registration_workflow, create_worker_removal_workflow, WorkflowEngine, create_worker_registration_workflow, create_worker_removal_workflow, WorkflowEngine,
}; };
let engine = Arc::new(WorkflowEngine::new()); let engine = Arc::new(WorkflowEngine::new());
@@ -112,7 +112,7 @@ pub async fn create_test_context(config: RouterConfig) -> Arc<AppContext> {
.expect("WorkflowEngine should only be initialized once"); .expect("WorkflowEngine should only be initialized once");
// Initialize MCP manager with empty config // Initialize MCP manager with empty config
use sglang_router_rs::mcp::{McpConfig, McpManager}; use sgl_model_gateway::mcp::{McpConfig, McpManager};
let empty_config = McpConfig { let empty_config = McpConfig {
servers: vec![], servers: vec![],
pool: Default::default(), pool: Default::default(),
@@ -137,7 +137,7 @@ pub async fn create_test_context_with_mcp_config(
config: RouterConfig, config: RouterConfig,
mcp_config_path: &str, mcp_config_path: &str,
) -> Arc<AppContext> { ) -> Arc<AppContext> {
use sglang_router_rs::mcp::{McpConfig, McpManager}; use sgl_model_gateway::mcp::{McpConfig, McpManager};
let client = reqwest::Client::new(); let client = reqwest::Client::new();
@@ -201,8 +201,8 @@ pub async fn create_test_context_with_mcp_config(
// Initialize JobQueue after AppContext is created // Initialize JobQueue after AppContext is created
let weak_context = Arc::downgrade(&app_context); let weak_context = Arc::downgrade(&app_context);
let job_queue = sglang_router_rs::core::JobQueue::new( let job_queue = sgl_model_gateway::core::JobQueue::new(
sglang_router_rs::core::JobQueueConfig::default(), sgl_model_gateway::core::JobQueueConfig::default(),
weak_context, weak_context,
); );
app_context app_context
@@ -211,7 +211,7 @@ pub async fn create_test_context_with_mcp_config(
.expect("JobQueue should only be initialized once"); .expect("JobQueue should only be initialized once");
// Initialize WorkflowEngine and register workflows // Initialize WorkflowEngine and register workflows
use sglang_router_rs::core::workflow::{ use sgl_model_gateway::core::workflow::{
create_worker_registration_workflow, create_worker_removal_workflow, WorkflowEngine, create_worker_registration_workflow, create_worker_removal_workflow, WorkflowEngine,
}; };
let engine = Arc::new(WorkflowEngine::new()); let engine = Arc::new(WorkflowEngine::new());
sgl-router/tests/common/test_app.rs
+1 -1
View File
@@ -2,7 +2,7 @@ use std::sync::{Arc, OnceLock};
use axum::Router; use axum::Router;
use reqwest::Client; use reqwest::Client;
use sglang_router_rs::{ use sgl_model_gateway::{
app_context::AppContext, app_context::AppContext,
config::RouterConfig, config::RouterConfig,
core::{LoadMonitor, WorkerRegistry}, core::{LoadMonitor, WorkerRegistry},
sgl-router/tests/mcp_test.rs
+1 -1
View File
@@ -13,7 +13,7 @@ use std::collections::HashMap;
use common::mock_mcp_server::MockMCPServer; use common::mock_mcp_server::MockMCPServer;
use serde_json::json; use serde_json::json;
use sglang_router_rs::mcp::{McpConfig, McpError, McpManager, McpServerConfig, McpTransport}; use sgl_model_gateway::mcp::{McpConfig, McpError, McpManager, McpServerConfig, McpTransport};
/// Create a new mock server for testing (each test gets its own) /// Create a new mock server for testing (each test gets its own)
async fn create_mock_server() -> MockMCPServer { async fn create_mock_server() -> MockMCPServer {
sgl-router/tests/metrics_aggregator_test.rs
+1 -1
View File
@@ -1,4 +1,4 @@
use sglang_router_rs::core::metrics_aggregator::{aggregate_metrics, MetricPack}; use sgl_model_gateway::core::metrics_aggregator::{aggregate_metrics, MetricPack};
#[test] #[test]
fn test_aggregate_simple() { fn test_aggregate_simple() {
sgl-router/tests/policy_registry_integration.rs
+3 -3
View File
@@ -2,7 +2,7 @@
use std::{collections::HashMap, sync::Arc}; use std::{collections::HashMap, sync::Arc};
use sglang_router_rs::{ use sgl_model_gateway::{
config::PolicyConfig, core::WorkerRegistry, policies::PolicyRegistry, config::PolicyConfig, core::WorkerRegistry, policies::PolicyRegistry,
protocols::worker_spec::WorkerConfigRequest, routers::router_manager::RouterManager, protocols::worker_spec::WorkerConfigRequest, routers::router_manager::RouterManager,
}; };
@@ -115,7 +115,7 @@ async fn test_policy_registry_with_router_manager() {
#[test] #[test]
fn test_policy_registry_cleanup() { fn test_policy_registry_cleanup() {
use sglang_router_rs::{config::PolicyConfig, policies::PolicyRegistry}; use sgl_model_gateway::{config::PolicyConfig, policies::PolicyRegistry};
let registry = PolicyRegistry::new(PolicyConfig::RoundRobin); let registry = PolicyRegistry::new(PolicyConfig::RoundRobin);
@@ -142,7 +142,7 @@ fn test_policy_registry_cleanup() {
#[test] #[test]
fn test_policy_registry_multiple_models() { fn test_policy_registry_multiple_models() {
use sglang_router_rs::{config::PolicyConfig, policies::PolicyRegistry}; use sgl_model_gateway::{config::PolicyConfig, policies::PolicyRegistry};
let registry = PolicyRegistry::new(PolicyConfig::RoundRobin); let registry = PolicyRegistry::new(PolicyConfig::RoundRobin);
sgl-router/tests/request_formats_test.rs
+1 -1
View File
@@ -5,7 +5,7 @@ use std::sync::Arc;
use common::mock_worker::{HealthStatus, MockWorker, MockWorkerConfig, WorkerType}; use common::mock_worker::{HealthStatus, MockWorker, MockWorkerConfig, WorkerType};
use reqwest::Client; use reqwest::Client;
use serde_json::json; use serde_json::json;
use sglang_router_rs::{ use sgl_model_gateway::{
config::{RouterConfig, RoutingMode}, config::{RouterConfig, RoutingMode},
routers::{RouterFactory, RouterTrait}, routers::{RouterFactory, RouterTrait},
}; };
sgl-router/tests/responses_api_test.rs
+4 -4
View File
@@ -1,7 +1,7 @@
// Integration test for Responses API // Integration test for Responses API
use axum::http::StatusCode; use axum::http::StatusCode;
use sglang_router_rs::protocols::{ use sgl_model_gateway::protocols::{
common::{GenerationRequest, ToolChoice, ToolChoiceValue, UsageInfo}, common::{GenerationRequest, ToolChoice, ToolChoiceValue, UsageInfo},
responses::{ responses::{
ReasoningEffort, ResponseInput, ResponseReasoningParam, ResponseTool, ResponseToolType, ReasoningEffort, ResponseInput, ResponseReasoningParam, ResponseTool, ResponseToolType,
@@ -14,7 +14,7 @@ use common::{
mock_mcp_server::MockMCPServer, mock_mcp_server::MockMCPServer,
mock_worker::{HealthStatus, MockWorker, MockWorkerConfig, WorkerType}, mock_worker::{HealthStatus, MockWorker, MockWorkerConfig, WorkerType},
}; };
use sglang_router_rs::{config::RouterConfig, routers::RouterFactory}; use sgl_model_gateway::{config::RouterConfig, routers::RouterFactory};
#[tokio::test] #[tokio::test]
async fn test_non_streaming_mcp_minimal_e2e_with_persistence() { async fn test_non_streaming_mcp_minimal_e2e_with_persistence() {
@@ -393,7 +393,7 @@ fn test_usage_conversion() {
completion_tokens: 25, completion_tokens: 25,
total_tokens: 40, total_tokens: 40,
reasoning_tokens: Some(8), reasoning_tokens: Some(8),
prompt_tokens_details: Some(sglang_router_rs::protocols::common::PromptTokenUsageInfo { prompt_tokens_details: Some(sgl_model_gateway::protocols::common::PromptTokenUsageInfo {
cached_tokens: 3, cached_tokens: 3,
}), }),
}; };
@@ -781,7 +781,7 @@ async fn test_max_tool_calls_limit() {
async fn setup_streaming_mcp_test() -> ( async fn setup_streaming_mcp_test() -> (
MockMCPServer, MockMCPServer,
MockWorker, MockWorker,
Box<dyn sglang_router_rs::routers::RouterTrait>, Box<dyn sgl_model_gateway::routers::RouterTrait>,
tempfile::TempDir, tempfile::TempDir,
) { ) {
let mcp = MockMCPServer::start().await.expect("start mcp"); let mcp = MockMCPServer::start().await.expect("start mcp");
sgl-router/tests/spec/chat_completion.rs
+1 -1
View File
@@ -1,5 +1,5 @@
use serde_json::json; use serde_json::json;
use sglang_router_rs::protocols::{ use sgl_model_gateway::protocols::{
chat::{ChatCompletionRequest, ChatMessage, MessageContent}, chat::{ChatCompletionRequest, ChatMessage, MessageContent},
common::{ common::{
Function, FunctionCall, FunctionChoice, StreamOptions, Tool, ToolChoice, ToolChoiceValue, Function, FunctionCall, FunctionChoice, StreamOptions, Tool, ToolChoice, ToolChoiceValue,
sgl-router/tests/spec/chat_message.rs
+1 -1
View File
@@ -1,5 +1,5 @@
use serde_json::json; use serde_json::json;
use sglang_router_rs::protocols::chat::{ChatMessage, MessageContent}; use sgl_model_gateway::protocols::chat::{ChatMessage, MessageContent};
#[test] #[test]
fn test_chat_message_tagged_by_role_system() { fn test_chat_message_tagged_by_role_system() {
sgl-router/tests/spec/embedding.rs
+1 -1
View File
@@ -1,5 +1,5 @@
use serde_json::{from_str, json, to_string}; use serde_json::{from_str, json, to_string};
use sglang_router_rs::protocols::{common::GenerationRequest, embedding::EmbeddingRequest}; use sgl_model_gateway::protocols::{common::GenerationRequest, embedding::EmbeddingRequest};
#[test] #[test]
fn test_embedding_request_serialization_string_input() { fn test_embedding_request_serialization_string_input() {
sgl-router/tests/spec/rerank.rs
+1 -1
View File
@@ -1,7 +1,7 @@
use std::collections::HashMap; use std::collections::HashMap;
use serde_json::{from_str, to_string, Number, Value}; use serde_json::{from_str, to_string, Number, Value};
use sglang_router_rs::protocols::{ use sgl_model_gateway::protocols::{
common::{GenerationRequest, StringOrArray, UsageInfo}, common::{GenerationRequest, StringOrArray, UsageInfo},
rerank::{RerankRequest, RerankResponse, RerankResult, V1RerankReqInput}, rerank::{RerankRequest, RerankResponse, RerankResult, V1RerankReqInput},
}; };
sgl-router/tests/spec/responses.rs
+9 -9
View File
@@ -1,5 +1,5 @@
use serde_json::json; use serde_json::json;
use sglang_router_rs::protocols::{ use sgl_model_gateway::protocols::{
common::{Function, StringOrArray, ToolChoice, ToolChoiceValue}, common::{Function, StringOrArray, ToolChoice, ToolChoiceValue},
responses::{ responses::{
IncludeField, ResponseInput, ResponseInputOutputItem, ResponseTool, ResponseToolType, IncludeField, ResponseInput, ResponseInputOutputItem, ResponseTool, ResponseToolType,
@@ -963,7 +963,7 @@ fn test_validate_input_items_structure() {
/// Test tool_choice defaults to auto when tools are present /// Test tool_choice defaults to auto when tools are present
#[test] #[test]
fn test_normalize_tool_choice_auto() { fn test_normalize_tool_choice_auto() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
@@ -1004,7 +1004,7 @@ fn test_normalize_tool_choice_auto() {
/// Test tool_choice defaults to none when tools array is empty /// Test tool_choice defaults to none when tools array is empty
#[test] #[test]
fn test_normalize_tool_choice_none() { fn test_normalize_tool_choice_none() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
@@ -1031,7 +1031,7 @@ fn test_normalize_tool_choice_none() {
/// Test tool_choice is not overridden if already set /// Test tool_choice is not overridden if already set
#[test] #[test]
fn test_normalize_tool_choice_no_override() { fn test_normalize_tool_choice_no_override() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
@@ -1068,7 +1068,7 @@ fn test_normalize_tool_choice_no_override() {
/// Test parallel_tool_calls defaults to true when tools are present /// Test parallel_tool_calls defaults to true when tools are present
#[test] #[test]
fn test_normalize_parallel_tool_calls() { fn test_normalize_parallel_tool_calls() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
@@ -1107,7 +1107,7 @@ fn test_normalize_parallel_tool_calls() {
/// Test parallel_tool_calls is not set when tools are absent /// Test parallel_tool_calls is not set when tools are absent
#[test] #[test]
fn test_normalize_parallel_tool_calls_no_tools() { fn test_normalize_parallel_tool_calls_no_tools() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
@@ -1127,7 +1127,7 @@ fn test_normalize_parallel_tool_calls_no_tools() {
/// Test parallel_tool_calls is not overridden if already set /// Test parallel_tool_calls is not overridden if already set
#[test] #[test]
fn test_normalize_parallel_tool_calls_no_override() { fn test_normalize_parallel_tool_calls_no_override() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
@@ -1162,7 +1162,7 @@ fn test_normalize_parallel_tool_calls_no_override() {
/// Test store defaults to true /// Test store defaults to true
#[test] #[test]
fn test_normalize_store_default() { fn test_normalize_store_default() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
@@ -1182,7 +1182,7 @@ fn test_normalize_store_default() {
/// Test store is not overridden if already set /// Test store is not overridden if already set
#[test] #[test]
fn test_normalize_store_no_override() { fn test_normalize_store_no_override() {
use sglang_router_rs::protocols::validated::Normalizable; use sgl_model_gateway::protocols::validated::Normalizable;
let mut request = ResponsesRequest { let mut request = ResponsesRequest {
input: ResponseInput::Text("test".to_string()), input: ResponseInput::Text("test".to_string()),
sgl-router/tests/streaming_tests.rs
+1 -1
View File
@@ -6,7 +6,7 @@ use common::mock_worker::{HealthStatus, MockWorker, MockWorkerConfig, WorkerType
use futures_util::StreamExt; use futures_util::StreamExt;
use reqwest::Client; use reqwest::Client;
use serde_json::json; use serde_json::json;
use sglang_router_rs::{ use sgl_model_gateway::{
config::{RouterConfig, RoutingMode}, config::{RouterConfig, RoutingMode},
routers::{RouterFactory, RouterTrait}, routers::{RouterFactory, RouterTrait},
}; };
sgl-router/tests/test_openai_routing.rs
+6 -4
View File
@@ -17,7 +17,7 @@ use axum::{
Json, Router, Json, Router,
}; };
use serde_json::json; use serde_json::json;
use sglang_router_rs::{ use sgl_model_gateway::{
config::{ config::{
ConfigError, ConfigValidator, HistoryBackend, OracleConfig, RouterConfig, RoutingMode, ConfigError, ConfigValidator, HistoryBackend, OracleConfig, RouterConfig, RoutingMode,
}, },
@@ -577,12 +577,14 @@ async fn test_router_factory_openai_mode() {
worker_urls: vec!["https://api.openai.com".to_string()], worker_urls: vec!["https://api.openai.com".to_string()],
}; };
let router_config = let router_config = RouterConfig::new(
RouterConfig::new(routing_mode, sglang_router_rs::config::PolicyConfig::Random); routing_mode,
sgl_model_gateway::config::PolicyConfig::Random,
);
let app_context = common::create_test_context(router_config).await; let app_context = common::create_test_context(router_config).await;
let router = sglang_router_rs::routers::RouterFactory::create_router(&app_context).await; let router = sgl_model_gateway::routers::RouterFactory::create_router(&app_context).await;
assert!( assert!(
router.is_ok(), router.is_ok(),
"Router factory should create OpenAI router successfully" "Router factory should create OpenAI router successfully"
sgl-router/tests/test_pd_routing.rs
+4 -4
View File
@@ -1,7 +1,7 @@
#[cfg(test)] #[cfg(test)]
mod test_pd_routing { mod test_pd_routing {
use serde_json::json; use serde_json::json;
use sglang_router_rs::{ use sgl_model_gateway::{
app_context::AppContext, app_context::AppContext,
config::{PolicyConfig, RouterConfig, RoutingMode}, config::{PolicyConfig, RouterConfig, RoutingMode},
core::{BasicWorkerBuilder, Worker, WorkerType}, core::{BasicWorkerBuilder, Worker, WorkerType},
@@ -38,7 +38,7 @@ mod test_pd_routing {
#[test] #[test]
fn test_worker_types() { fn test_worker_types() {
use sglang_router_rs::core::{BasicWorkerBuilder, Worker, WorkerType}; use sgl_model_gateway::core::{BasicWorkerBuilder, Worker, WorkerType};
let prefill_worker: Box<dyn Worker> = Box::new( let prefill_worker: Box<dyn Worker> = Box::new(
BasicWorkerBuilder::new("http://prefill:8080") BasicWorkerBuilder::new("http://prefill:8080")
@@ -214,7 +214,7 @@ mod test_pd_routing {
let app_context = { let app_context = {
use std::sync::{Arc, OnceLock}; use std::sync::{Arc, OnceLock};
use sglang_router_rs::{ use sgl_model_gateway::{
core::{LoadMonitor, WorkerRegistry}, core::{LoadMonitor, WorkerRegistry},
data_connector::{ data_connector::{
MemoryConversationItemStorage, MemoryConversationStorage, MemoryConversationItemStorage, MemoryConversationStorage,
@@ -673,7 +673,7 @@ mod test_pd_routing {
#[test] #[test]
fn test_bootstrap_injection_with_benchmark_requests() { fn test_bootstrap_injection_with_benchmark_requests() {
use sglang_router_rs::core::{BasicWorkerBuilder, Worker, WorkerType}; use sgl_model_gateway::core::{BasicWorkerBuilder, Worker, WorkerType};
let mut benchmark_request = json!({ let mut benchmark_request = json!({
"input_ids": vec![vec![1, 2, 3, 4]; 16], // Batch size 16 "input_ids": vec![vec![1, 2, 3, 4]; 16], // Batch size 16
sgl-router/tests/tokenizer_cache_correctness_test.rs
+1 -1
View File
@@ -9,7 +9,7 @@ use std::{
sync::{Arc, OnceLock}, sync::{Arc, OnceLock},
}; };
use sglang_router_rs::tokenizer::{ use sgl_model_gateway::tokenizer::{
cache::{CacheConfig, CachedTokenizer}, cache::{CacheConfig, CachedTokenizer},
hub::download_tokenizer_from_hf, hub::download_tokenizer_from_hf,
huggingface::HuggingFaceTokenizer, huggingface::HuggingFaceTokenizer,
sgl-router/tests/tokenizer_integration.rs
+6 -6
View File
@@ -7,7 +7,7 @@ mod common;
use std::sync::Arc; use std::sync::Arc;
use common::{ensure_tokenizer_cached, EXPECTED_HASHES, TEST_PROMPTS}; use common::{ensure_tokenizer_cached, EXPECTED_HASHES, TEST_PROMPTS};
use sglang_router_rs::tokenizer::{ use sgl_model_gateway::tokenizer::{
factory, huggingface::HuggingFaceTokenizer, sequence::Sequence, stop::*, stream::DecodeStream, factory, huggingface::HuggingFaceTokenizer, sequence::Sequence, stop::*, stream::DecodeStream,
traits::*, traits::*,
}; };
@@ -269,7 +269,7 @@ fn test_batch_encoding() {
#[test] #[test]
fn test_special_tokens() { fn test_special_tokens() {
use sglang_router_rs::tokenizer::traits::Tokenizer as TokenizerTrait; use sgl_model_gateway::tokenizer::traits::Tokenizer as TokenizerTrait;
let tokenizer_path = ensure_tokenizer_cached(); let tokenizer_path = ensure_tokenizer_cached();
let tokenizer = HuggingFaceTokenizer::from_file(tokenizer_path.to_str().unwrap()) let tokenizer = HuggingFaceTokenizer::from_file(tokenizer_path.to_str().unwrap())
@@ -398,7 +398,7 @@ fn test_load_chat_template_from_local_file() {
#[tokio::test] #[tokio::test]
async fn test_tinyllama_embedded_template() { async fn test_tinyllama_embedded_template() {
use sglang_router_rs::tokenizer::hub::download_tokenizer_from_hf; use sgl_model_gateway::tokenizer::hub::download_tokenizer_from_hf;
// Skip in CI without HF_TOKEN // Skip in CI without HF_TOKEN
@@ -434,7 +434,7 @@ async fn test_tinyllama_embedded_template() {
#[tokio::test] #[tokio::test]
async fn test_qwen3_next_embedded_template() { async fn test_qwen3_next_embedded_template() {
use sglang_router_rs::tokenizer::hub::download_tokenizer_from_hf; use sgl_model_gateway::tokenizer::hub::download_tokenizer_from_hf;
// Test 3: Qwen3-Next has chat template in tokenizer_config.json // Test 3: Qwen3-Next has chat template in tokenizer_config.json
match download_tokenizer_from_hf("Qwen/Qwen3-Next-80B-A3B-Instruct").await { match download_tokenizer_from_hf("Qwen/Qwen3-Next-80B-A3B-Instruct").await {
@@ -466,7 +466,7 @@ async fn test_qwen3_next_embedded_template() {
#[tokio::test] #[tokio::test]
async fn test_qwen3_vl_json_template_priority() { async fn test_qwen3_vl_json_template_priority() {
use sglang_router_rs::tokenizer::hub::download_tokenizer_from_hf; use sgl_model_gateway::tokenizer::hub::download_tokenizer_from_hf;
// Test 4: Qwen3-VL has both tokenizer_config.json template and chat_template.json // Test 4: Qwen3-VL has both tokenizer_config.json template and chat_template.json
// Should prioritize chat_template.json // Should prioritize chat_template.json
@@ -508,7 +508,7 @@ async fn test_qwen3_vl_json_template_priority() {
#[tokio::test] #[tokio::test]
async fn test_llava_separate_jinja_template() { async fn test_llava_separate_jinja_template() {
use sglang_router_rs::tokenizer::hub::download_tokenizer_from_hf; use sgl_model_gateway::tokenizer::hub::download_tokenizer_from_hf;
// Test 5: llava has chat_template.jinja as a separate file, not in tokenizer_config.json // Test 5: llava has chat_template.jinja as a separate file, not in tokenizer_config.json
match download_tokenizer_from_hf("llava-hf/llava-1.5-7b-hf").await { match download_tokenizer_from_hf("llava-hf/llava-1.5-7b-hf").await {
sgl-router/tests/tool_parser_deepseek.rs
+1 -1
View File
@@ -1,6 +1,6 @@
//! DeepSeek V3 Parser Integration Tests //! DeepSeek V3 Parser Integration Tests
use sglang_router_rs::tool_parser::{DeepSeekParser, ToolParser}; use sgl_model_gateway::tool_parser::{DeepSeekParser, ToolParser};
mod common; mod common;
use common::create_test_tools; use common::create_test_tools;
sgl-router/tests/tool_parser_edge_cases.rs
+1 -1
View File
@@ -2,7 +2,7 @@
//! //!
//! Tests for malformed input, edge cases, and error recovery //! Tests for malformed input, edge cases, and error recovery
use sglang_router_rs::tool_parser::{ use sgl_model_gateway::tool_parser::{
JsonParser, MistralParser, PythonicParser, QwenParser, ToolParser, JsonParser, MistralParser, PythonicParser, QwenParser, ToolParser,
}; };
sgl-router/tests/tool_parser_fallback.rs
+1 -1
View File
@@ -3,7 +3,7 @@
//! When tool call parsing fails, the original text should be preserved as normal text //! When tool call parsing fails, the original text should be preserved as normal text
//! rather than being lost. This ensures graceful degradation. //! rather than being lost. This ensures graceful degradation.
use sglang_router_rs::tool_parser::{ use sgl_model_gateway::tool_parser::{
DeepSeekParser, JsonParser, LlamaParser, MistralParser, QwenParser, ToolParser, DeepSeekParser, JsonParser, LlamaParser, MistralParser, QwenParser, ToolParser,
}; };
sgl-router/tests/tool_parser_glm4_moe.rs
+1 -1
View File
@@ -1,6 +1,6 @@
//! GLM-4 MoE Parser Integration Tests //! GLM-4 MoE Parser Integration Tests
use sglang_router_rs::tool_parser::{Glm4MoeParser, ToolParser}; use sgl_model_gateway::tool_parser::{Glm4MoeParser, ToolParser};
mod common; mod common;
use common::create_test_tools; use common::create_test_tools;
sgl-router/tests/tool_parser_json.rs
+14 -14
View File
@@ -3,7 +3,7 @@
//! Tests for the JSON parser which handles OpenAI, Claude, and generic JSON formats //! Tests for the JSON parser which handles OpenAI, Claude, and generic JSON formats
use serde_json::json; use serde_json::json;
use sglang_router_rs::tool_parser::{JsonParser, ToolParser}; use sgl_model_gateway::tool_parser::{JsonParser, ToolParser};
mod common; mod common;
use common::{create_test_tools, streaming_helpers::*}; use common::{create_test_tools, streaming_helpers::*};
@@ -166,7 +166,7 @@ async fn test_json_format_detection() {
// Streaming tests for JSON array format // Streaming tests for JSON array format
#[tokio::test] #[tokio::test]
async fn test_json_array_streaming_required_mode() { async fn test_json_array_streaming_required_mode() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test that simulates the exact streaming pattern from required mode // Test that simulates the exact streaming pattern from required mode
let mut parser = JsonParser::new(); let mut parser = JsonParser::new();
@@ -174,7 +174,7 @@ async fn test_json_array_streaming_required_mode() {
// Define test tools // Define test tools
let tools = vec![Tool { let tools = vec![Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: serde_json::json!({}), parameters: serde_json::json!({}),
@@ -240,7 +240,7 @@ async fn test_json_array_streaming_required_mode() {
#[tokio::test] #[tokio::test]
async fn test_json_array_multiple_tools_streaming() { async fn test_json_array_multiple_tools_streaming() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test with multiple tools in array // Test with multiple tools in array
let mut parser = JsonParser::new(); let mut parser = JsonParser::new();
@@ -248,7 +248,7 @@ async fn test_json_array_multiple_tools_streaming() {
let tools = vec![ let tools = vec![
Tool { Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: serde_json::json!({}), parameters: serde_json::json!({}),
@@ -257,7 +257,7 @@ async fn test_json_array_multiple_tools_streaming() {
}, },
Tool { Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_news".to_string(), name: "get_news".to_string(),
description: Some("Get news".to_string()), description: Some("Get news".to_string()),
parameters: serde_json::json!({}), parameters: serde_json::json!({}),
@@ -305,14 +305,14 @@ async fn test_json_array_multiple_tools_streaming() {
#[tokio::test] #[tokio::test]
async fn test_json_array_closing_bracket_separate_chunk() { async fn test_json_array_closing_bracket_separate_chunk() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test case where the closing ] comes as a separate chunk // Test case where the closing ] comes as a separate chunk
let mut parser = JsonParser::new(); let mut parser = JsonParser::new();
let tools = vec![Tool { let tools = vec![Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: json!({}), parameters: json!({}),
@@ -366,14 +366,14 @@ async fn test_json_array_closing_bracket_separate_chunk() {
#[tokio::test] #[tokio::test]
async fn test_json_single_object_with_trailing_text() { async fn test_json_single_object_with_trailing_text() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test single object format (no array) with trailing text // Test single object format (no array) with trailing text
let mut parser = JsonParser::new(); let mut parser = JsonParser::new();
let tools = vec![Tool { let tools = vec![Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: serde_json::json!({}), parameters: serde_json::json!({}),
@@ -420,14 +420,14 @@ async fn test_json_single_object_with_trailing_text() {
#[tokio::test] #[tokio::test]
async fn test_json_single_object_with_bracket_in_text() { async fn test_json_single_object_with_bracket_in_text() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test that ] in normal text is NOT stripped for single object format // Test that ] in normal text is NOT stripped for single object format
let mut parser = JsonParser::new(); let mut parser = JsonParser::new();
let tools = vec![Tool { let tools = vec![Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: serde_json::json!({}), parameters: serde_json::json!({}),
@@ -472,14 +472,14 @@ async fn test_json_single_object_with_bracket_in_text() {
#[tokio::test] #[tokio::test]
async fn test_json_array_bracket_in_text_after_tools() { async fn test_json_array_bracket_in_text_after_tools() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test that ] in normal text AFTER array tools is preserved // Test that ] in normal text AFTER array tools is preserved
let mut parser = JsonParser::new(); let mut parser = JsonParser::new();
let tools = vec![Tool { let tools = vec![Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: serde_json::json!({}), parameters: serde_json::json!({}),
sgl-router/tests/tool_parser_kimik2.rs
+1 -1
View File
@@ -1,6 +1,6 @@
//! Kimi K2 Parser Integration Tests //! Kimi K2 Parser Integration Tests
use sglang_router_rs::tool_parser::{KimiK2Parser, ToolParser}; use sgl_model_gateway::tool_parser::{KimiK2Parser, ToolParser};
mod common; mod common;
use common::create_test_tools; use common::create_test_tools;
sgl-router/tests/tool_parser_llama.rs
+1 -1
View File
@@ -2,7 +2,7 @@
//! //!
//! Tests for the Llama parser which handles <|python_tag|> format and plain JSON //! Tests for the Llama parser which handles <|python_tag|> format and plain JSON
use sglang_router_rs::tool_parser::{LlamaParser, ToolParser}; use sgl_model_gateway::tool_parser::{LlamaParser, ToolParser};
mod common; mod common;
use common::{create_test_tools, streaming_helpers::*}; use common::{create_test_tools, streaming_helpers::*};
sgl-router/tests/tool_parser_minimax_m2.rs
+1 -1
View File
@@ -1,6 +1,6 @@
//! MiniMax M2 Parser Integration Tests //! MiniMax M2 Parser Integration Tests
use sglang_router_rs::tool_parser::{MinimaxM2Parser, ToolParser}; use sgl_model_gateway::tool_parser::{MinimaxM2Parser, ToolParser};
mod common; mod common;
use common::create_test_tools; use common::create_test_tools;
sgl-router/tests/tool_parser_mistral.rs
+5 -5
View File
@@ -3,7 +3,7 @@
//! Tests for the Mistral parser which handles [TOOL_CALLS] format //! Tests for the Mistral parser which handles [TOOL_CALLS] format
use serde_json::json; use serde_json::json;
use sglang_router_rs::tool_parser::{MistralParser, ToolParser}; use sgl_model_gateway::tool_parser::{MistralParser, ToolParser};
#[tokio::test] #[tokio::test]
async fn test_mistral_single_tool() { async fn test_mistral_single_tool() {
@@ -158,14 +158,14 @@ Let me execute these searches for you."#;
#[tokio::test] #[tokio::test]
async fn test_mistral_streaming_closing_bracket() { async fn test_mistral_streaming_closing_bracket() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test that closing ] is stripped for Mistral array format // Test that closing ] is stripped for Mistral array format
let mut parser = MistralParser::new(); let mut parser = MistralParser::new();
let tools = vec![Tool { let tools = vec![Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: json!({}), parameters: json!({}),
@@ -218,14 +218,14 @@ async fn test_mistral_streaming_closing_bracket() {
#[tokio::test] #[tokio::test]
async fn test_mistral_streaming_bracket_in_text_after_tools() { async fn test_mistral_streaming_bracket_in_text_after_tools() {
use sglang_router_rs::protocols::common::Tool; use sgl_model_gateway::protocols::common::Tool;
// Test that ] in normal text AFTER tool calls is preserved // Test that ] in normal text AFTER tool calls is preserved
let mut parser = MistralParser::new(); let mut parser = MistralParser::new();
let tools = vec![Tool { let tools = vec![Tool {
tool_type: "function".to_string(), tool_type: "function".to_string(),
function: sglang_router_rs::protocols::common::Function { function: sgl_model_gateway::protocols::common::Function {
name: "get_weather".to_string(), name: "get_weather".to_string(),
description: Some("Get weather".to_string()), description: Some("Get weather".to_string()),
parameters: json!({}), parameters: json!({}),
sgl-router/tests/tool_parser_mixed_edge_cases.rs
+1 -1
View File
@@ -3,7 +3,7 @@
//! Tests for edge cases across parsers and mixed format scenarios //! Tests for edge cases across parsers and mixed format scenarios
use serde_json::json; use serde_json::json;
use sglang_router_rs::tool_parser::{ use sgl_model_gateway::tool_parser::{
JsonParser, LlamaParser, MistralParser, PythonicParser, QwenParser, ToolParser, JsonParser, LlamaParser, MistralParser, PythonicParser, QwenParser, ToolParser,
}; };
sgl-router/tests/tool_parser_partial_json.rs
+1 -1
View File
@@ -2,7 +2,7 @@
//! //!
//! Tests for the partial JSON parser with allow_partial_strings flag behavior //! Tests for the partial JSON parser with allow_partial_strings flag behavior
use sglang_router_rs::tool_parser::partial_json::PartialJson; use sgl_model_gateway::tool_parser::partial_json::PartialJson;
#[test] #[test]
fn test_partial_string_flag_disallows_incomplete_strings() { fn test_partial_string_flag_disallows_incomplete_strings() {
sgl-router/tests/tool_parser_pythonic.rs
+1 -1
View File
@@ -3,7 +3,7 @@
//! Tests for the Pythonic parser which handles Python function call syntax //! Tests for the Pythonic parser which handles Python function call syntax
use serde_json::json; use serde_json::json;
use sglang_router_rs::tool_parser::{PythonicParser, ToolParser}; use sgl_model_gateway::tool_parser::{PythonicParser, ToolParser};
mod common; mod common;
use common::create_test_tools; use common::create_test_tools;
sgl-router/tests/tool_parser_qwen.rs
+1 -1
View File
@@ -3,7 +3,7 @@
//! Tests for the Qwen parser which handles <tool_call>...</tool_call> format //! Tests for the Qwen parser which handles <tool_call>...</tool_call> format
use serde_json::json; use serde_json::json;
use sglang_router_rs::tool_parser::{QwenParser, ToolParser}; use sgl_model_gateway::tool_parser::{QwenParser, ToolParser};
mod common; mod common;
use common::{create_test_tools, streaming_helpers::*}; use common::{create_test_tools, streaming_helpers::*};
sgl-router/tests/tool_parser_step3.rs
+1 -1
View File
@@ -1,6 +1,6 @@
//! Step3 Parser Integration Tests //! Step3 Parser Integration Tests
use sglang_router_rs::tool_parser::{Step3Parser, ToolParser}; use sgl_model_gateway::tool_parser::{Step3Parser, ToolParser};
mod common; mod common;
use common::create_test_tools; use common::create_test_tools;
sgl-router/tests/workflow_test.rs
+1 -1
View File
@@ -8,7 +8,7 @@ use std::{
time::Duration, time::Duration,
}; };
use sglang_router_rs::core::workflow::*; use sgl_model_gateway::core::workflow::*;
use tokio::time::sleep; use tokio::time::sleep;
// Test step that counts invocations // Test step that counts invocations