Add DeepReinforce Ornith-1.0 to cookbook (#29404)
Co-authored-by: averyjones4 <averyjones4@users.noreply.github.com> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
averyjones4
Zijie Xia
Claude Opus 4.8
parent
3d2e7cc601
commit
631213c3bf
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
@@ -0,0 +1,224 @@
|
|||||||
|
---
|
||||||
|
title: Ornith-1.0
|
||||||
|
description: "Deploy DeepReinforce Ornith-1.0 with SGLang - a self-improving agentic-coding model family with 397B, 35B, and 9B checkpoints plus FP8 and GGUF variants."
|
||||||
|
tag: NEW
|
||||||
|
---
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
<a id="install" />
|
||||||
|
|
||||||
|
<Accordion title="Install SGLang">
|
||||||
|
|
||||||
|
Ornith-1.0 model cards recommend SGLang `>=0.5.9`. The Deploy panel below emits the base serve command; the reasoning and tool-call parsers from the model-card quickstarts (`--reasoning-parser qwen3` for `<think>...</think>` traces, `--tool-call-parser qwen3_coder` for Qwen-style XML tool calls) are added on top via the [Playground](#playground).
|
||||||
|
|
||||||
|
<Tabs>
|
||||||
|
|
||||||
|
<Tab title="Python (pip / uv)">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
pip install --upgrade pip
|
||||||
|
pip install uv
|
||||||
|
uv pip install "sglang>=0.5.9"
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the **Python** output of the command panel below in that environment.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
<Tab title="Docker">
|
||||||
|
|
||||||
|
```bash Command
|
||||||
|
docker pull lmsysorg/sglang:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
|
||||||
|
|
||||||
|
</Tab>
|
||||||
|
|
||||||
|
</Tabs>
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
Pick an Ornith checkpoint to generate the launch command. The non-FP8 397B recipe requires an H200 single node in this matrix. The 397B-FP8 recipe is available on H100 and H200 with TP=8; H100 also supports the 35B and 9B checkpoints. The 35B recipes use tensor parallelism 2 in this matrix. The 9B checkpoint is dense and serves on a single GPU by default; the command panel makes that default explicit with `--tp 1`.
|
||||||
|
|
||||||
|
import { Deployment } from "/src/snippets/_deployment.jsx";
|
||||||
|
import { config } from "/src/snippets/configs/deepreinforce-ai/ornith-1.0.jsx";
|
||||||
|
|
||||||
|
<Deployment config={config} />
|
||||||
|
|
||||||
|
## Playground
|
||||||
|
|
||||||
|
The Playground layers SGLang features on top of whichever cell the Deploy panel is showing — only your overrides change, and any change flips the badge to **Not Verified** until the new configuration is run end-to-end.
|
||||||
|
|
||||||
|
For Ornith-1.0 the knobs are the reasoning and tool-call parsers:
|
||||||
|
|
||||||
|
- **Reasoning Parser** appends `--reasoning-parser qwen3`. Ornith emits `<think>...</think>` traces; with this on, SGLang surfaces them as `message.reasoning_content` instead of leaving the tags inline in `content`.
|
||||||
|
- **Tool Call Parser** appends `--tool-call-parser qwen3_coder`, so Qwen-style XML tool calls are returned as OpenAI-compatible `tool_calls`.
|
||||||
|
|
||||||
|
import { Playground } from "/src/snippets/_playground.jsx";
|
||||||
|
|
||||||
|
<Playground config={config} />
|
||||||
|
|
||||||
|
## 1. Model Introduction
|
||||||
|
|
||||||
|
[Ornith-1.0](https://huggingface.co/collections/deepreinforce-ai/ornith-10) is DeepReinforce's self-improving open-source model family for agentic coding. The model cards describe the family as post-trained on top of Gemma 4 and Qwen 3.5, and the collection currently includes 397B, 35B, and 9B repos plus FP8 and GGUF variants. The model cards report results on Terminal-Bench 2.1, SWE-Bench, NL2Repo, ClawEval, and SWE Atlas benchmarks.
|
||||||
|
|
||||||
|
**Key Features:**
|
||||||
|
|
||||||
|
- **Agentic coding specialization**: the model cards describe Ornith-1.0 as specialized for agentic coding and report coding-agent benchmark results.
|
||||||
|
- **Self-improving training**: the model cards state that Ornith-1.0 uses reinforcement learning to optimize both solution rollouts and the scaffold that drives those rollouts.
|
||||||
|
- **Reasoning model behavior**: assistant responses begin with a `<think>...</think>` reasoning block before the final answer; enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) to split it into `reasoning_content`.
|
||||||
|
- **Tool calling**: emits Qwen-style XML tool calls; enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground).
|
||||||
|
- **Long context**: model-card recipes use `--context-length 262144`.
|
||||||
|
- **MIT license**: the Hugging Face repos are released under MIT.
|
||||||
|
|
||||||
|
**Available Models:**
|
||||||
|
|
||||||
|
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||||
|
<colgroup>
|
||||||
|
<col style={{width: "32%"}} />
|
||||||
|
<col style={{width: "18%"}} />
|
||||||
|
<col style={{width: "18%"}} />
|
||||||
|
<col style={{width: "32%"}} />
|
||||||
|
</colgroup>
|
||||||
|
<thead>
|
||||||
|
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Model</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Format</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Deploy Panel</th>
|
||||||
|
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700}}>Notes</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B">deepreinforce-ai/Ornith-1.0-397B</a></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>H200 only</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>Flagship 397B MoE checkpoint; model-card baseline uses TP=8 on an H200 single node.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B-FP8">deepreinforce-ai/Ornith-1.0-397B-FP8</a></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>FP8 repo in the collection; the deploy command uses this repo id with TP=8.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B">deepreinforce-ai/Ornith-1.0-35B</a></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>35B MoE checkpoint; the deploy command uses TP=2.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-FP8">deepreinforce-ai/Ornith-1.0-35B-FP8</a></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>FP8</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>FP8 repo in the collection; the deploy command uses this repo id with TP=2.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B">deepreinforce-ai/Ornith-1.0-9B</a></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>BF16</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>H100 / H200</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>Dense 9B checkpoint; the model card describes it as designed for efficient single-GPU deployment.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF">deepreinforce-ai/Ornith-1.0-35B-GGUF</a></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>GGUF</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>No</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>Listed for completeness; GGUF targets llama.cpp-style local inference, not the SGLang server recipe here.</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td style={{padding: "9px 12px", fontWeight: 500}}><a href="https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B-GGUF">deepreinforce-ai/Ornith-1.0-9B-GGUF</a></td>
|
||||||
|
<td style={{padding: "9px 12px"}}>GGUF</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>No</td>
|
||||||
|
<td style={{padding: "9px 12px"}}>Listed for completeness; the model card shows llama.cpp and Ollama examples for the GGUF build.</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
**License:** [MIT](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B/blob/main/LICENSE)
|
||||||
|
|
||||||
|
**Resources:** [Hugging Face collection](https://huggingface.co/collections/deepreinforce-ai/ornith-10) · [Ornith blog](https://deep-reinforce.com/ornith_1_0.html)
|
||||||
|
|
||||||
|
## 2. Configuration Tips
|
||||||
|
|
||||||
|
- **Reasoning parser**: Ornith responses include `<think>...</think>`. Enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) so OpenAI-compatible responses expose the reasoning trace as `message.reasoning_content`.
|
||||||
|
- **Tool-call parser**: enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) so `<tool_call>` blocks are returned as OpenAI-compatible tool calls.
|
||||||
|
- **Context length**: the model-card SGLang recipes use `--context-length 262144`. Lower it if you need more memory headroom.
|
||||||
|
- **Tensor parallelism**: the 397B model-card recipes use `--tp 8`; in this single-node matrix, non-FP8 397B is H200-only, while 397B-FP8 is available on both H100 and H200. The 35B deploy commands use `--tp 2`. The 9B model-card recipe is single-GPU by default; the command panel makes that explicit with `--tp 1`. Adjust TP to match your node and memory budget.
|
||||||
|
- **Sampling**: model cards recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20` for normal use. Their reported benchmark setup may use different task-specific sampling parameters.
|
||||||
|
- **Benchmarks**: benchmark numbers in the model cards are reported by DeepReinforce. They are useful for context, but the command panel leaves recipes unverified until exact runs are signed off.
|
||||||
|
|
||||||
|
## 3. Usage Examples
|
||||||
|
|
||||||
|
### 3.1 Basic Chat Completion
|
||||||
|
|
||||||
|
`message.reasoning_content` is only populated when the server was launched with the `--reasoning-parser qwen3` toggle (see the [Playground](#playground)); otherwise the `<think>...</think>` trace stays inline in `message.content`.
|
||||||
|
|
||||||
|
<Accordion title="Python client">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="Ornith-1.0-9B",
|
||||||
|
messages=[
|
||||||
|
{"role": "user", "content": "Write a compact Python function is_prime(n)."}
|
||||||
|
],
|
||||||
|
temperature=0.6,
|
||||||
|
top_p=0.95,
|
||||||
|
max_tokens=1024,
|
||||||
|
extra_body={"top_k": 20},
|
||||||
|
)
|
||||||
|
|
||||||
|
message = response.choices[0].message
|
||||||
|
print("=============== Reasoning ===============")
|
||||||
|
print(message.reasoning_content)
|
||||||
|
print("=============== Answer ==================")
|
||||||
|
print(message.content)
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
|
|
||||||
|
### 3.2 Tool Calling
|
||||||
|
|
||||||
|
Enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) and launch with the resulting command. Then use the standard OpenAI-compatible `tools` field:
|
||||||
|
|
||||||
|
<Accordion title="Tool-call request">
|
||||||
|
|
||||||
|
```python Example
|
||||||
|
from openai import OpenAI
|
||||||
|
|
||||||
|
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||||
|
|
||||||
|
tools = [{
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "run_tests",
|
||||||
|
"description": "Run the project's test suite.",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"target": {"type": "string", "description": "Test target or command"}
|
||||||
|
},
|
||||||
|
"required": ["target"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}]
|
||||||
|
|
||||||
|
response = client.chat.completions.create(
|
||||||
|
model="Ornith-1.0-9B",
|
||||||
|
messages=[{"role": "user", "content": "Run the unit tests for the parser module."}],
|
||||||
|
tools=tools,
|
||||||
|
tool_choice="auto",
|
||||||
|
temperature=0.6,
|
||||||
|
top_p=0.95,
|
||||||
|
max_tokens=2048,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(response.choices[0].message.tool_calls)
|
||||||
|
```
|
||||||
|
|
||||||
|
</Accordion>
|
||||||
@@ -19,6 +19,12 @@ metatags:
|
|||||||
href="/cookbook/autoregressive/DeepSeek/DeepSeek-V4"
|
href="/cookbook/autoregressive/DeepSeek/DeepSeek-V4"
|
||||||
img="/cards/logos/deepseek.png"
|
img="/cards/logos/deepseek.png"
|
||||||
/>
|
/>
|
||||||
|
<Card
|
||||||
|
title="DeepReinforce"
|
||||||
|
mode="card"
|
||||||
|
href="/cookbook/autoregressive/DeepReinforce/Ornith-1.0"
|
||||||
|
img="/cards/logos/deepreinforce.png"
|
||||||
|
/>
|
||||||
<Card
|
<Card
|
||||||
title="Llama"
|
title="Llama"
|
||||||
mode="card"
|
mode="card"
|
||||||
|
|||||||
@@ -994,6 +994,12 @@
|
|||||||
"cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2"
|
"cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"group": "DeepReinforce",
|
||||||
|
"pages": [
|
||||||
|
"cookbook/autoregressive/DeepReinforce/Ornith-1.0"
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"group": "Llama",
|
"group": "Llama",
|
||||||
"pages": [
|
"pages": [
|
||||||
|
|||||||
@@ -0,0 +1,202 @@
|
|||||||
|
// Ornith-1.0 (DeepReinforce) — config-driven cookbook page.
|
||||||
|
// The launch flags follow the SGLang quickstarts published on the model cards; the 9B recipe makes the single-GPU default explicit with --tp 1.
|
||||||
|
// FP8 quantization cells use their FP8 repo ids from the collection with the same flags; their README quickstarts currently point to the non-FP8 repos.
|
||||||
|
// Reasoning / tool-call parsers are exposed as a Playground toggle (qwen3 / qwen3_coder), not baked into the deploy cells.
|
||||||
|
// Cells remain unverified until exact recipes are run and signed off.
|
||||||
|
|
||||||
|
export const config = {
|
||||||
|
modelName: "Ornith-1.0",
|
||||||
|
|
||||||
|
supportedHardware: ["h100", "h200"],
|
||||||
|
|
||||||
|
variants: [
|
||||||
|
{ id: "397b", label: "397B", subtitle: "MoE" },
|
||||||
|
{ id: "35b", label: "35B", subtitle: "MoE" },
|
||||||
|
{ id: "9b", label: "9B", subtitle: "dense" },
|
||||||
|
],
|
||||||
|
|
||||||
|
quantizations: [
|
||||||
|
{ id: "bf16", label: "BF16" },
|
||||||
|
{ id: "fp8", label: "FP8" },
|
||||||
|
],
|
||||||
|
|
||||||
|
strategies: [
|
||||||
|
{ id: "balanced", label: "Balanced" },
|
||||||
|
],
|
||||||
|
|
||||||
|
nodesOptions: [
|
||||||
|
{ id: "single", label: "Single Node" },
|
||||||
|
],
|
||||||
|
|
||||||
|
modelNames: {
|
||||||
|
"397b|bf16": "Ornith-1.0-397B",
|
||||||
|
"397b|fp8": "Ornith-1.0-397B-FP8",
|
||||||
|
"35b|bf16": "Ornith-1.0-35B",
|
||||||
|
"35b|fp8": "Ornith-1.0-35B-FP8",
|
||||||
|
"9b|bf16": "Ornith-1.0-9B",
|
||||||
|
},
|
||||||
|
|
||||||
|
placeholders: {
|
||||||
|
HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" },
|
||||||
|
PORT: { target: "command", label: "Bind port", default: "30000" },
|
||||||
|
CURL_HOST: { target: "curl", label: "Server host", default: "localhost" },
|
||||||
|
CURL_PORT: { target: "curl", label: "Server port", default: "30000" },
|
||||||
|
},
|
||||||
|
|
||||||
|
curl: `curl http://{{CURL_HOST}}:{{CURL_PORT}}/v1/chat/completions \\
|
||||||
|
-H 'Content-Type: application/json' \\
|
||||||
|
-d '{ "model": "{{MODEL_NAME}}", "messages": [{"role":"user","content":"Write a compact Python is_prime function."}], "temperature": 0.6, "top_p": 0.95, "top_k": 20, "max_tokens": 1024 }'`,
|
||||||
|
|
||||||
|
benchmarkCommands: {
|
||||||
|
speed:
|
||||||
|
`python3 -m sglang.bench_serving \\
|
||||||
|
--backend sglang \\
|
||||||
|
--host {{CURL_HOST}} --port {{CURL_PORT}} \\
|
||||||
|
--model {{MODEL_NAME}} \\
|
||||||
|
--dataset-name random \\
|
||||||
|
--random-input-len {{ISL}} --random-output-len {{OSL}} \\
|
||||||
|
--num-prompts {{NUM_PROMPTS}} --max-concurrency {{MAX_CONCURRENCY}}`,
|
||||||
|
numPromptsByConc: { 1: 8, 16: 32, 64: 128, 128: 256, 256: 512 },
|
||||||
|
},
|
||||||
|
|
||||||
|
dockerImages: {
|
||||||
|
h100: "lmsysorg/sglang:latest",
|
||||||
|
h200: "lmsysorg/sglang:latest",
|
||||||
|
},
|
||||||
|
|
||||||
|
github: {
|
||||||
|
cookbookModel: "deepreinforce-ai/Ornith-1.0",
|
||||||
|
},
|
||||||
|
|
||||||
|
// Parsers are added on top of the base deploy command via the Playground,
|
||||||
|
// not baked into the cells. Both ids are registered in the SGLang parser
|
||||||
|
// registries and match the model's Qwen3.5 chat template.
|
||||||
|
playgroundFeatures: {
|
||||||
|
parsers: {
|
||||||
|
items: [
|
||||||
|
{ id: "reasoning", label: "Reasoning Parser", flag: "--reasoning-parser qwen3" },
|
||||||
|
{ id: "toolCall", label: "Tool Call Parser", flag: "--tool-call-parser qwen3_coder" },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
|
||||||
|
cells: [
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "397b", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-397B",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 8",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "397b", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-397B-FP8",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 8",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "35b", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-35B",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 2",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.85",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "35b", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-35B-FP8",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 2",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.85",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h200", variant: "9b", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-9B",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 1",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.85",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h100", variant: "397b", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-397B-FP8",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 8",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.8",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h100", variant: "35b", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-35B",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 2",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.85",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h100", variant: "35b", quant: "fp8", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-35B-FP8",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 2",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.85",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
match: { hw: "h100", variant: "9b", quant: "bf16", strategy: "balanced", nodes: "single" },
|
||||||
|
env: [],
|
||||||
|
flags: [
|
||||||
|
"--model-path deepreinforce-ai/Ornith-1.0-9B",
|
||||||
|
"--served-model-name {{MODEL_NAME}}",
|
||||||
|
"--tp 1",
|
||||||
|
"--context-length 262144",
|
||||||
|
"--mem-fraction-static 0.85",
|
||||||
|
"--host {{HOST_IP}}",
|
||||||
|
"--port {{PORT}}",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
Reference in New Issue
Block a user