config: decisions keyed on the attention backend read the configured pair
`--attention-backend` is one field of three: a launch that sets only
`--prefill-attention-backend` or `--decode-attention-backend` leaves the base
field at `None`. Seven decisions read that base field alone and therefore
answered from a field the operator never set. `attention_backends()` is the
pair with the base-field fallback already applied, so each site now asks it for
the half it actually needs:
- `inkling_common/attn` assembles backend-specific kwargs (rel_bias / score
mods) and gates its fused prologue; the backend those describe is the one
`self.attn` dispatches to, so `serving_attention_backend()` selects the pair
member by `forward_batch.forward_mode`, mirroring
`HybridAttnBackend._select_backend` exactly -- draft-extend routes through
the prefill branch like the dispatcher does -- and preferring the
runner-stamped pair, so a draft runner answers with its own backend. That
preference only works if every backend that can enter a ForwardContext
carries the stamp, so `DraftBackendFactory._create_backend` now stamps its
products with the backend it resolved (draft override first), and the
draft-extend conv-sidecar wrapper copies the wrapped backend's stamp -- the
replacement backends the spec workers install had no stamp at all and fell
back to the target's configured pair.
- The chunked-prefix-cache gate is a *prefill* feature -> prefill half. Reading
the base field switched the feature off for every prefill-only configuration.
- `init_deterministic_inference_config` maps *prefill* knobs
(SPLIT_TILE / PREFILL_TRUNCATION_ALIGN) -> prefill half; the map missed and
left truncation unset.
- `two_batch_overlap` computes extend positions -> prefill half.
- mrope's interleaved-rope kernel runs in both phases -> both halves must
support triton. This one is not conservative when it misreads:
`support_triton(None)` answers **True**, so a `--prefill-attention-backend
torch_native` launch took the triton path.
- The req-to-token writer has one caller, `alloc_for_extend` -> prefill half;
its fallback pays several `.item()` syncs per request, so gating it on the
decode half too would send every extend of a mixed launch through the slow
path. `get_last_loc` (the spec-decode allocator's helper) keeps the
both-halves reading: verify tokens are served by either half depending on
`speculative_attention_mode`.
- The flashinfer version floor is a guard; it never fired for a launch that
pinned flashinfer through a split field.
One more site the census found is not converted here: `gpt_oss` derives its
`sinks` parameter dtype from the backend, and a single parameter dtype cannot
serve a split pair (FA4 asserts bfloat16, trtllm_mha consumes float32), so
that one is a behaviour question rather than a config-source one and is fixed
in its own PR.
`test_split_attention_backend_decisions.py` pins the callable decisions by
calling them under a split-only publish, and pins the remaining ones
statically -- the file/why map fails if any of them goes back to the base field
(reverse-verified). It also asserts the `support_triton(None) is True` trap the
sweep exists for.
The stamp comes from the constructor, not the request: every factory leaf
answers ("effective_name", backend), because several map entries do not build
what their key says -- cutedsl_mla draft-extend builds the trtllm-mla backend,
"nsa" is a deprecated alias building dsa, and the hybrid-linear entries pick
fa3/intel_amx/triton by host, which no static rename table can express (a
review catch: on Blackwell the alias stamp reached Inkling's per-forward
kwargs assembly, which asserts a concrete kernel name, and crashed the first
draft-extend forward). The stamping is pinned by unit tests, not only by a
spec e2e: removing the child-stamping loop, stamping an alias from a leaf, or
dropping the wrapper copy goes red (reverse-verified), and a static guard
walks the factory source asserting no leaf answers an alias name. The child loop states its contract explicitly --
`create_decode_backend` passes `stamps_children=True` because its products
are per-step containers by construction, so a container without
`attn_backends` raises instead of being silently skipped by a defensive
probe. The `_version` invalidation names its contract (autograd's in-place
counter: private, chosen because it is the only per-tensor signal that ticks
on copy_-style updates; removal fails loudly). The version-floor guard's file
joins the pair-reader ratchet, and the one runner-seed chain read sharing the
backend's __init__ (`speculative_eagle_topk`) reads the spec bag.
This commit is contained in:
@@ -40,7 +40,11 @@ from sglang.srt.model_executor.forward_batch_info import (
|
||||
compute_position,
|
||||
)
|
||||
from sglang.srt.model_executor.forward_context import get_attn_backend
|
||||
from sglang.srt.runtime_context import get_device, get_exec, get_parallel
|
||||
from sglang.srt.runtime_context import (
|
||||
attention_backends,
|
||||
get_device,
|
||||
get_parallel,
|
||||
)
|
||||
from sglang.srt.speculative.spec_info import SpecInput
|
||||
from sglang.srt.utils import BumpAllocator, empty_context, get_bool_env_var, is_hip
|
||||
|
||||
@@ -631,8 +635,10 @@ class TboForwardBatchPreparer:
|
||||
device_field="extend_prefix_lens",
|
||||
sum_field=None,
|
||||
)
|
||||
# The prefill half: this computes extend positions.
|
||||
prefill_backend, _ = attention_backends()
|
||||
_, child_b.extend_start_loc = compute_position(
|
||||
get_exec().kernel.attention_backend,
|
||||
prefill_backend,
|
||||
child_b.extend_prefix_lens,
|
||||
child_b.extend_seq_lens,
|
||||
child_b.extend_num_tokens,
|
||||
|
||||
@@ -1669,7 +1669,7 @@ def _set_envs_and_config(server_args: ServerArgs):
|
||||
|
||||
# Check flashinfer version
|
||||
if not get_bool_env_var("SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK"):
|
||||
if server_args.attention_backend == "flashinfer":
|
||||
if "flashinfer" in server_args.get_attention_backends():
|
||||
assert_pkg_version(
|
||||
"flashinfer_python",
|
||||
"0.6.17",
|
||||
|
||||
@@ -171,7 +171,7 @@ class TRTLLMHAAttnBackend(FlashInferAttnBackend):
|
||||
|
||||
# Speculative decoding
|
||||
# Only support topk <= 1 for now.
|
||||
self.topk = model_runner.server_args.speculative_eagle_topk or 0
|
||||
self.topk = get_spec().speculative_eagle_topk or 0
|
||||
self.speculative_step_id = speculative_step_id
|
||||
self.target_verify_metadata = {}
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ from sglang.srt.layers.rotary_embedding.yarn import (
|
||||
yarn_get_mscale_simple,
|
||||
yarn_linear_ramp_mask,
|
||||
)
|
||||
from sglang.srt.runtime_context import get_exec
|
||||
from sglang.srt.runtime_context import attention_backends, get_exec
|
||||
from sglang.srt.utils import (
|
||||
cpu_has_amx_support,
|
||||
is_cuda,
|
||||
@@ -143,7 +143,9 @@ class MRotaryEmbedding(RotaryEmbedding):
|
||||
last_dim = cos_sin.size()[-1]
|
||||
cos, sin = cos_sin.chunk(2, dim=-1)
|
||||
if self.mrope_interleaved:
|
||||
if support_triton(get_exec().kernel.attention_backend):
|
||||
# Runs in prefill and decode: both halves must support triton.
|
||||
prefill_backend, decode_backend = attention_backends()
|
||||
if support_triton(prefill_backend) and support_triton(decode_backend):
|
||||
cos = apply_interleaved_rope_triton(cos, self.mrope_section)
|
||||
sin = apply_interleaved_rope_triton(sin, self.mrope_section)
|
||||
else:
|
||||
|
||||
@@ -28,6 +28,7 @@ from http import HTTPStatus
|
||||
from typing import TYPE_CHECKING, Any, Deque, Dict, List, Optional, Set, Tuple, Union
|
||||
|
||||
from sglang.srt.runtime_context import (
|
||||
attention_backends,
|
||||
get_device,
|
||||
get_disagg,
|
||||
get_exec,
|
||||
@@ -1517,9 +1518,10 @@ class Scheduler(
|
||||
"flashinfer": ("SGLANG_FLASHINFER_PREFILL_SPLIT_TILE_SIZE", 4096),
|
||||
"triton": ("SGLANG_TRITON_PREFILL_TRUNCATION_ALIGN_SIZE", 4096),
|
||||
}
|
||||
env_var, default_size = backend_sizes.get(
|
||||
get_exec().kernel.attention_backend, (None, None)
|
||||
)
|
||||
# Both entries are prefill knobs (SPLIT_TILE / PREFILL_TRUNCATION):
|
||||
# the prefill half decides.
|
||||
prefill_backend, _ = attention_backends()
|
||||
env_var, default_size = backend_sizes.get(prefill_backend, (None, None))
|
||||
self.truncation_align_size = (
|
||||
get_int_env_var(env_var, default_size) if env_var else None
|
||||
)
|
||||
|
||||
@@ -26,7 +26,7 @@ from sglang.srt.mem_cache.common import (
|
||||
evict_from_tree_cache,
|
||||
)
|
||||
from sglang.srt.mem_cache.memory_pool import HybridReqToTokenPool, ReqToTokenPool
|
||||
from sglang.srt.runtime_context import get_exec, get_parallel
|
||||
from sglang.srt.runtime_context import attention_backends, get_parallel
|
||||
from sglang.srt.utils import (
|
||||
is_cpu,
|
||||
is_cuda,
|
||||
@@ -65,7 +65,10 @@ def write_cache_indices(
|
||||
prefix_tensors: list[torch.Tensor],
|
||||
req_to_token_pool: ReqToTokenPool,
|
||||
):
|
||||
if support_triton(get_exec().kernel.attention_backend):
|
||||
# This writer's one caller is `alloc_for_extend`, so the prefill half
|
||||
# decides; the fallback below pays several `.item()` syncs per request.
|
||||
prefill_backend, _ = attention_backends()
|
||||
if support_triton(prefill_backend):
|
||||
prefix_pointers = torch.tensor(
|
||||
[t.data_ptr() for t in prefix_tensors],
|
||||
dtype=torch.uint64,
|
||||
@@ -106,8 +109,11 @@ def get_last_loc(
|
||||
req_pool_indices_tensor: torch.Tensor,
|
||||
prefix_lens_tensor: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
attn_backend = get_exec().kernel.attention_backend
|
||||
uses_triton_dispatch = attn_backend not in ("ascend", "torch_native")
|
||||
prefill_backend, decode_backend = attention_backends()
|
||||
uses_triton_dispatch = prefill_backend not in (
|
||||
"ascend",
|
||||
"torch_native",
|
||||
) and decode_backend not in ("ascend", "torch_native")
|
||||
|
||||
if _is_hip and uses_triton_dispatch:
|
||||
# HIP-only: the legacy get_last_loc_triton kernel emits a
|
||||
|
||||
@@ -8,7 +8,11 @@ from sglang.srt.configs.model_config import (
|
||||
is_deepseek_dsa,
|
||||
is_kimi_k3,
|
||||
)
|
||||
from sglang.srt.runtime_context import get_context, get_exec, get_schedule
|
||||
from sglang.srt.runtime_context import (
|
||||
attention_backends,
|
||||
get_context,
|
||||
get_schedule,
|
||||
)
|
||||
from sglang.srt.server_args import CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -29,10 +33,11 @@ def maybe_disable_chunked_prefix_cache(
|
||||
# model's (often non-MLA) config must not flip the shared setting.
|
||||
if is_draft_worker:
|
||||
return
|
||||
# Chunked prefix cache is a prefill feature: the prefill half decides.
|
||||
prefill_backend, _ = attention_backends()
|
||||
if (
|
||||
not use_mla_backend
|
||||
or get_exec().kernel.attention_backend
|
||||
not in CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS
|
||||
or prefill_backend not in CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS
|
||||
):
|
||||
if not get_schedule().disable_chunked_prefix_cache:
|
||||
get_context().override(
|
||||
|
||||
@@ -30,9 +30,11 @@ from sglang.srt.models.inkling_common.norm import RMSNorm
|
||||
from sglang.srt.models.inkling_common.sconv import SconvType, ShortConvolution
|
||||
from sglang.srt.models.utils import apply_qk_norm
|
||||
from sglang.srt.runtime_context import (
|
||||
attention_backends,
|
||||
get_exec,
|
||||
get_model,
|
||||
get_parallel,
|
||||
get_spec,
|
||||
)
|
||||
from sglang.srt.utils import add_prefix, get_current_device_stream_fast
|
||||
|
||||
@@ -105,6 +107,30 @@ _REL_PROJ_MATMUL_MAX_T = 48
|
||||
_REL_PROJ_TAU_KERNEL_MAX_T = 32
|
||||
|
||||
|
||||
def serving_attention_backend(forward_batch: ForwardBatch) -> str:
|
||||
"""The pair member serving this forward.
|
||||
|
||||
Mirrors ``HybridAttnBackend._select_backend`` exactly: decode for
|
||||
decode/idle, the ``speculative_attention_mode`` half for target-verify,
|
||||
prefill otherwise -- including draft-extend, which the hybrid dispatcher
|
||||
routes through its prefill branch. The pair the runner stamped on its
|
||||
backend wins over the configured one, so a draft runner answers with its
|
||||
own backend.
|
||||
"""
|
||||
from sglang.srt.model_executor.forward_context import get_attn_backend
|
||||
|
||||
backend = get_attn_backend()
|
||||
configured_prefill, configured_decode = attention_backends()
|
||||
prefill = backend.prefill_attention_backend_str or configured_prefill
|
||||
decode = backend.decode_attention_backend_str or configured_decode
|
||||
mode = forward_batch.forward_mode
|
||||
if mode.is_decode_or_idle():
|
||||
return decode
|
||||
if mode.is_target_verify():
|
||||
return decode if get_spec().speculative_attention_mode == "decode" else prefill
|
||||
return prefill
|
||||
|
||||
|
||||
def _rel_proj_kernel_eligible(r: torch.Tensor) -> bool:
|
||||
"""rel_proj_small_t input contract: bf16 CUDA, [t, h, d_rel] with a
|
||||
contiguous (h*d_rel) inner block (token rows may be strided), d_rel a
|
||||
@@ -735,7 +761,9 @@ class InklingAttention(nn.Module):
|
||||
|
||||
apply_log_scaling = log_scaling_tau is not None and not self.is_local
|
||||
|
||||
attention_backend = get_exec().kernel.attention_backend
|
||||
# The kwargs below must describe the backend `self.attn` dispatches
|
||||
# this forward to.
|
||||
attention_backend = serving_attention_backend(forward_batch)
|
||||
assert attention_backend in ("fa4", "triton")
|
||||
# The overlap threads a CUDA event into the FA4 sheared-bias kernel, so it
|
||||
# is FA4-only for now.
|
||||
|
||||
@@ -40,7 +40,11 @@ class DraftBackendFactory:
|
||||
self.draft_attn_backend = draft_model_runner.draft_attention_backend
|
||||
|
||||
def _create_backend(
|
||||
self, backend_name: str, backend_map: dict, error_template: str
|
||||
self,
|
||||
backend_name: str,
|
||||
backend_map: dict,
|
||||
error_template: str,
|
||||
stamps_children: bool = False,
|
||||
):
|
||||
# The split pair with the base-backend fallback already applied.
|
||||
prefill_backend, decode_backend = attention_backends()
|
||||
@@ -54,7 +58,15 @@ class DraftBackendFactory:
|
||||
if backend_type not in backend_map:
|
||||
raise ValueError(error_template.format(backend_type=backend_type))
|
||||
|
||||
return backend_map[backend_type]()
|
||||
stamp, backend = backend_map[backend_type]()
|
||||
if backend is not None:
|
||||
backend.prefill_attention_backend_str = stamp
|
||||
backend.decode_attention_backend_str = stamp
|
||||
if stamps_children:
|
||||
for child in backend.attn_backends:
|
||||
child.prefill_attention_backend_str = stamp
|
||||
child.decode_attention_backend_str = stamp
|
||||
return backend
|
||||
|
||||
def create_decode_backend(self):
|
||||
# No multi-step draft backend for steps=0 (nospec) or steps=1.
|
||||
@@ -88,6 +100,7 @@ class DraftBackendFactory:
|
||||
"decode_attention_backend",
|
||||
backend_map,
|
||||
"EAGLE is not supported in decode attention backend {backend_type}",
|
||||
stamps_children=True,
|
||||
)
|
||||
|
||||
def create_draft_extend_backend(self):
|
||||
@@ -125,27 +138,41 @@ class DraftBackendFactory:
|
||||
attn_backend_wrapper_for_draft_extend,
|
||||
)
|
||||
|
||||
return attn_backend_wrapper_for_draft_extend(self.draft_model_runner, backend)
|
||||
wrapped = attn_backend_wrapper_for_draft_extend(
|
||||
self.draft_model_runner, backend
|
||||
)
|
||||
if wrapped is not backend and wrapped is not None and backend is not None:
|
||||
wrapped.prefill_attention_backend_str = (
|
||||
backend.prefill_attention_backend_str
|
||||
)
|
||||
wrapped.decode_attention_backend_str = backend.decode_attention_backend_str
|
||||
return wrapped
|
||||
|
||||
def _create_dsa_decode_backend(self):
|
||||
from sglang.srt.layers.attention.dsa_backend import (
|
||||
DeepseekSparseAttnMultiStepBackend,
|
||||
)
|
||||
|
||||
return DeepseekSparseAttnMultiStepBackend(
|
||||
self.draft_model_runner,
|
||||
self.topk,
|
||||
self.speculative_num_steps,
|
||||
seed_dsa_topk_from_draft_extend=self.seed_dsa_topk_from_draft_extend,
|
||||
return (
|
||||
"dsa",
|
||||
DeepseekSparseAttnMultiStepBackend(
|
||||
self.draft_model_runner,
|
||||
self.topk,
|
||||
self.speculative_num_steps,
|
||||
seed_dsa_topk_from_draft_extend=self.seed_dsa_topk_from_draft_extend,
|
||||
),
|
||||
)
|
||||
|
||||
def _create_dsa_prefill_backend(self):
|
||||
from sglang.srt.layers.attention.dsa_backend import DeepseekSparseAttnBackend
|
||||
|
||||
return DeepseekSparseAttnBackend(
|
||||
self.draft_model_runner,
|
||||
skip_prefill=False,
|
||||
seed_dsa_topk_from_draft_extend=self.seed_dsa_topk_from_draft_extend,
|
||||
return (
|
||||
"dsa",
|
||||
DeepseekSparseAttnBackend(
|
||||
self.draft_model_runner,
|
||||
skip_prefill=False,
|
||||
seed_dsa_topk_from_draft_extend=self.seed_dsa_topk_from_draft_extend,
|
||||
),
|
||||
)
|
||||
|
||||
def _create_flashinfer_decode_backend(self):
|
||||
@@ -154,16 +181,22 @@ class DraftBackendFactory:
|
||||
FlashInferMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return FlashInferMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"flashinfer",
|
||||
FlashInferMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
else:
|
||||
from sglang.srt.layers.attention.flashinfer_mla_backend import (
|
||||
FlashInferMLAMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return FlashInferMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"flashinfer",
|
||||
FlashInferMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_triton_decode_backend(self):
|
||||
@@ -171,8 +204,11 @@ class DraftBackendFactory:
|
||||
TritonMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return TritonMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"triton",
|
||||
TritonMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_intel_amx_decode_backend(self):
|
||||
@@ -180,8 +216,11 @@ class DraftBackendFactory:
|
||||
IntelAMXMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return IntelAMXMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"intel_amx",
|
||||
IntelAMXMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_hybrid_linear_attn_decode_backend(self):
|
||||
@@ -201,8 +240,11 @@ class DraftBackendFactory:
|
||||
def _create_aiter_decode_backend(self):
|
||||
from sglang.srt.layers.attention.aiter_backend import AiterMultiStepDraftBackend
|
||||
|
||||
return AiterMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"aiter",
|
||||
AiterMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_fa_decode_backend(self, fa_impl_ver: int = 3):
|
||||
@@ -215,11 +257,14 @@ class DraftBackendFactory:
|
||||
MusaFlashAttentionMultiStepBackend as FlashAttentionMultiStepBackend,
|
||||
)
|
||||
|
||||
return FlashAttentionMultiStepBackend(
|
||||
self.draft_model_runner,
|
||||
self.topk,
|
||||
self.speculative_num_steps,
|
||||
fa_impl_ver=fa_impl_ver,
|
||||
return (
|
||||
f"fa{fa_impl_ver}",
|
||||
FlashAttentionMultiStepBackend(
|
||||
self.draft_model_runner,
|
||||
self.topk,
|
||||
self.speculative_num_steps,
|
||||
fa_impl_ver=fa_impl_ver,
|
||||
),
|
||||
)
|
||||
|
||||
def _create_fa3_decode_backend(self):
|
||||
@@ -233,8 +278,11 @@ class DraftBackendFactory:
|
||||
FlashMLAMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return FlashMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"flashmla",
|
||||
FlashMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_trtllm_mha_decode_backend(self):
|
||||
@@ -242,8 +290,11 @@ class DraftBackendFactory:
|
||||
TRTLLMHAAttnMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return TRTLLMHAAttnMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"trtllm_mha",
|
||||
TRTLLMHAAttnMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_trtllm_mla_decode_backend(self, backend: str = "trtllm-gen"):
|
||||
@@ -256,11 +307,14 @@ class DraftBackendFactory:
|
||||
TRTLLMMLAMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return TRTLLMMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner,
|
||||
self.topk,
|
||||
self.speculative_num_steps,
|
||||
backend=backend,
|
||||
return (
|
||||
"trtllm_mla",
|
||||
TRTLLMMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner,
|
||||
self.topk,
|
||||
self.speculative_num_steps,
|
||||
backend=backend,
|
||||
),
|
||||
)
|
||||
|
||||
def _create_cutedsl_mla_decode_backend(self):
|
||||
@@ -273,8 +327,11 @@ class DraftBackendFactory:
|
||||
CuteDslMLAMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return CuteDslMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"cutedsl_mla",
|
||||
CuteDslMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_tokenspeed_mla_decode_backend(self):
|
||||
@@ -287,8 +344,11 @@ class DraftBackendFactory:
|
||||
TokenspeedMLAMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return TokenspeedMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"tokenspeed_mla",
|
||||
TokenspeedMLAMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_ascend_decode_backend(self):
|
||||
@@ -296,8 +356,11 @@ class DraftBackendFactory:
|
||||
AscendAttnMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return AscendAttnMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"ascend",
|
||||
AscendAttnMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_dsv4_decode_backend(self):
|
||||
@@ -307,8 +370,11 @@ class DraftBackendFactory:
|
||||
DeepseekV4AscendMultiStepDraftBackend,
|
||||
)
|
||||
|
||||
return DeepseekV4AscendMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"dsv4",
|
||||
DeepseekV4AscendMultiStepDraftBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
elif is_hip():
|
||||
from sglang.srt.layers.attention.deepseek_v4_backend_hip_radix import (
|
||||
@@ -319,8 +385,11 @@ class DraftBackendFactory:
|
||||
DeepseekV4MultiStepBackend,
|
||||
)
|
||||
|
||||
return DeepseekV4MultiStepBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
return (
|
||||
"dsv4",
|
||||
DeepseekV4MultiStepBackend(
|
||||
self.draft_model_runner, self.topk, self.speculative_num_steps
|
||||
),
|
||||
)
|
||||
|
||||
def _create_flashinfer_prefill_backend(self):
|
||||
@@ -329,28 +398,37 @@ class DraftBackendFactory:
|
||||
FlashInferAttnBackend,
|
||||
)
|
||||
|
||||
return FlashInferAttnBackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"flashinfer",
|
||||
FlashInferAttnBackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
else:
|
||||
from sglang.srt.layers.attention.flashinfer_mla_backend import (
|
||||
FlashInferMLAAttnBackend,
|
||||
)
|
||||
|
||||
return FlashInferMLAAttnBackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"flashinfer",
|
||||
FlashInferMLAAttnBackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
|
||||
def _create_triton_prefill_backend(self):
|
||||
from sglang.srt.layers.attention.triton_backend import TritonAttnBackend
|
||||
|
||||
return TritonAttnBackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"triton",
|
||||
TritonAttnBackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
|
||||
def _create_intel_amx_prefill_backend(self):
|
||||
from sglang.srt.layers.attention.intel_amx_backend import IntelAMXAttnBackend
|
||||
|
||||
return IntelAMXAttnBackend(self.draft_model_runner)
|
||||
return ("intel_amx", IntelAMXAttnBackend(self.draft_model_runner))
|
||||
|
||||
def _create_aiter_prefill_backend(self):
|
||||
from sglang.srt.layers.attention.aiter_backend import AiterAttnBackend
|
||||
|
||||
return AiterAttnBackend(self.draft_model_runner, skip_prefill=False)
|
||||
return ("aiter", AiterAttnBackend(self.draft_model_runner, skip_prefill=False))
|
||||
|
||||
def _create_fa_prefill_backend(self, fa_impl_ver: int = 3):
|
||||
if not is_musa():
|
||||
@@ -361,8 +439,11 @@ class DraftBackendFactory:
|
||||
from sglang.srt.hardware_backend.musa.attention.flashattention_backend import (
|
||||
MusaFlashAttentionBackend as FlashAttentionBackend,
|
||||
)
|
||||
return FlashAttentionBackend(
|
||||
self.draft_model_runner, skip_prefill=False, fa_impl_ver=fa_impl_ver
|
||||
return (
|
||||
f"fa{fa_impl_ver}",
|
||||
FlashAttentionBackend(
|
||||
self.draft_model_runner, skip_prefill=False, fa_impl_ver=fa_impl_ver
|
||||
),
|
||||
)
|
||||
|
||||
def _create_fa3_prefill_backend(self):
|
||||
@@ -374,7 +455,10 @@ class DraftBackendFactory:
|
||||
def _create_trtllm_mha_prefill_backend(self):
|
||||
from sglang.srt.layers.attention.trtllm_mha_backend import TRTLLMHAAttnBackend
|
||||
|
||||
return TRTLLMHAAttnBackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"trtllm_mha",
|
||||
TRTLLMHAAttnBackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
|
||||
def _create_trtllm_mla_prefill_backend(self):
|
||||
if not self.draft_model_runner.use_mla_backend:
|
||||
@@ -384,7 +468,10 @@ class DraftBackendFactory:
|
||||
|
||||
from sglang.srt.layers.attention.trtllm_mla_backend import TRTLLMMLABackend
|
||||
|
||||
return TRTLLMMLABackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"trtllm_mla",
|
||||
TRTLLMMLABackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
|
||||
def _create_tokenspeed_mla_prefill_backend(self):
|
||||
if not self.draft_model_runner.use_mla_backend:
|
||||
@@ -396,19 +483,25 @@ class DraftBackendFactory:
|
||||
TokenspeedMLABackend,
|
||||
)
|
||||
|
||||
return TokenspeedMLABackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"tokenspeed_mla",
|
||||
TokenspeedMLABackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
|
||||
def _create_ascend_prefill_backend(self):
|
||||
from sglang.srt.hardware_backend.npu.attention.ascend_backend import (
|
||||
AscendAttnBackend,
|
||||
)
|
||||
|
||||
return AscendAttnBackend(self.draft_model_runner)
|
||||
return ("ascend", AscendAttnBackend(self.draft_model_runner))
|
||||
|
||||
def _create_flashmla_prefill_backend(self):
|
||||
from sglang.srt.layers.attention.flashmla_backend import FlashMLABackend
|
||||
|
||||
return FlashMLABackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"flashmla",
|
||||
FlashMLABackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
|
||||
def _create_dsv4_prefill_backend(self):
|
||||
# On NPU the "dsv4" backend resolves to the Ascend V4 subclass; its
|
||||
@@ -418,17 +511,21 @@ class DraftBackendFactory:
|
||||
ATTENTION_BACKENDS,
|
||||
)
|
||||
|
||||
return ATTENTION_BACKENDS["dsv4"](self.draft_model_runner)
|
||||
return ("dsv4", ATTENTION_BACKENDS["dsv4"](self.draft_model_runner))
|
||||
elif is_hip():
|
||||
from sglang.srt.layers.attention.deepseek_v4_backend_hip_radix import (
|
||||
DeepseekV4HipRadixBackend,
|
||||
)
|
||||
|
||||
return DeepseekV4HipRadixBackend(
|
||||
self.draft_model_runner, skip_prefill=False
|
||||
return (
|
||||
"dsv4",
|
||||
DeepseekV4HipRadixBackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
from sglang.srt.layers.attention.deepseek_v4_backend import (
|
||||
DeepseekV4AttnBackend,
|
||||
)
|
||||
|
||||
return DeepseekV4AttnBackend(self.draft_model_runner, skip_prefill=False)
|
||||
return (
|
||||
"dsv4",
|
||||
DeepseekV4AttnBackend(self.draft_model_runner, skip_prefill=False),
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user