Skip to content

DFlash 2 Reference

Context

Look-up facts for DFlash 2 and its DFlash (v1) base: release timeline, engine support matrix with first-supporting releases, drafter checkpoints and their licenses, the drafter config keys every engine reads, per-engine flags, the NeMo training config, and all published benchmark tables with their test conditions. For how the drafter works, see Explanation. For deployment steps, see How-to Guides.

Release Timeline

Date Event Source
2026-01-06 z-lab/dflash Issue #1 opened (request for training code). Still open Issue #1
2026-02-05 DFlash paper v1 on arXiv (2602.06036). The Inco blog says DFlash was "released in January" arXiv
2026-05-28 DFlash paper v2 on arXiv. Accepted at ICML 2026 arXiv v2, ICML poster
2026-06-15 LMSYS blog: DFlash on the SGLang Spec V2 engine, Qwen3.5-397B-A17B drafter LMSYS
2026-06-23 NVIDIA blog: up to 15x throughput on Blackwell Ultra (TensorRT-LLM, DFlash v1) NVIDIA
2026-07-09 z-lab/dflash Issue #146 (CUDA-graph crash under load on vLLM 0.22.1) Issue #146
Aug 2026 DFlash 2 announced by Inco AI with two drafters (Qwen3.8-27B, Muse-Glimmer-30B) Inco blog
2026-08-18 dflash 0.1.0 on PyPI (MIT, Python >= 3.10) PyPI
2026-08-19 SGLang PR #35371 (DFlash2) merged PR #35371
2026-08-21 vLLM PR #52816 (DFlash2) merged PR #52816
2026-08-22 NeMo AutoModel PR #3605 (DFlash 2 draft model, trainer, recipe) merged PR #3605
2026-08-26 vLLM v0.28.0: first vLLM release with DFlash 2 vLLM v0.28.0
2026-08-27 llama.cpp PR #27342 (DFlash2) merged PR #27342
2026-09-04 SGLang v0.5.19: first SGLang release with DFlash 2 SGLang v0.5.19
2026-09-18 Inco AI releases Splash, an Apple-silicon engine that ships DFlash 2 drafts incoai/splash
2026-09-23 TensorRT-LLM 1.3.0rc28: first tagged build with the DFlash 2 selector path (pre-release) PyPI

Release dates for the engine releases and the dflash package come from PyPI upload times (UTC). The first-supporting release is confirmed from the source at each tag: the DFlash 2 draft class (DFlash2DraftModel) is in SGLang v0.5.19 but not v0.5.18, and in vLLM v0.28.0 but not v0.27.1.

Engine Support Matrix

Status as of 2026-09-25.

Engine DFlash (v1) DFlash 2 First release with DFlash 2 Latest release (date) Status
SGLang --speculative-algorithm DFLASH (PR #22077, then Spec V2 in PR #23000) Same DFLASH algorithm. DFlash2DraftModel is picked from the checkpoint config (PR #35371) v0.5.19 (2026-09-04) 0.5.20 (2026-09-18) Mainline
vLLM method: dflash in --speculative-config Same method. DFlash2DraftModel maps to qwen3_dflash2 (PR #52816) v0.28.0 (2026-08-26) 0.30.0 (2026-09-22) Mainline
TensorRT-LLM DFlashDecodingConfig / decoding_type: DFlash (in 1.3.0rc15, 2026-05-21, and later) Same config, no extra arguments 1.3.0rc28 (2026-09-23, pre-release) Stable 1.2.1 has no DFlash. Pre-release 1.3.0rc28 Main branch and pre-releases only
llama.cpp --spec-type draft-dflash Auto-detected from the GGUF (selector_top_k > 0) under the same type (PR #27342) master after 2026-08-27 Rolling bNNNN builds Mainline
ollama — PR #17865 "mlx: add DFlash2 support" (MLX engine only) None — Open PR, branch build
oMLX Fork z-lab/omlx-fork release 0.6.2-dflash2 (signed arm64 dmg) Fork only — Fork build
Splash (Inco AI) — Ships a DFlash 2 draft inside each model package 2026-09-18 launch — Vendor engine, Apache-2.0
vLLM TPU (tpu-inference) JAX port (Google blog) None: the latest release, tpu-inference 0.29.0 (PyPI, 2026-09-10), ships only the v1 DFlashDraftModel / qwen3_dflash path, with no DFlash2DraftModel (wheel inspected 2026-09-27) — 0.29.0 (2026-09-10) v1 only

Other ports (checked 2026-09-28): exllamav3 merged DFlash 2 draft support in PR #334 (2026-09-20). vLLM-Ascend merged DFlash 2 docs in PR #17383 (2026-09-24), while its feature PRs (#14524, #15886, #17557 for GLM-5.3-Flash on Model Runner V2) are still open.

Drafter Checkpoints

DFlash 2 Drafters

Checkpoint Target Format License Notes
incoai/Qwen3.8-27B-DFlash2 (mirror z-lab/Qwen3.8-27B-DFlash2) Qwen/Qwen3.8-27B Safetensors, BF16, ~2B params Apache-2.0 (HF metadata) Launch drafter. Block size 8 (7 draft tokens)
incoai/Qwen3.8-27B-DFlash2-GGUF (mirror z-lab/Qwen3.8-27B-DFlash2-GGUF) Qwen3.8-27B GGUF GGUF Check the card llama.cpp path
incoai/Muse-Glimmer-30B-DFlash2 (mirror z-lab/Muse-Glimmer-30B-DFlash2) meta-models/Muse-Glimmer-30B Safetensors Check the card Launch drafter. Launch evals used block size 16
incoai/GLM-5.3-DFlash2 zai-org/GLM-5.3 Safetensors Check the card Post-launch drafter
incoai/GLM-5.3-Flash-DFlash2 GLM-5.3-Flash Safetensors CC BY-NC-ND 4.0 (research and evaluation). Commercial licensing from Inco Post-launch drafter. No derivatives: do not ship requantized copies
incoai/Qwen3.8-27B-Splash, incoai/Qwen3.6-35B-A3B-Splash Qwen3.8-27B, Qwen3.6-35B-A3B (4-bit) Splash package (17.4 GB, 20.9 GB) Check the card Bundles target + DFlash 2 draft. Loads only in Splash

Licenses differ per checkpoint

The code (z-lab/dflash, the dflash package) is MIT. Drafter weights are not covered by that license. Read each model card before commercial use. "Check the card" in the table means the license is not recorded here.

DFlash (v1) Targets

The DFlash collection lists these targets (z-lab README, 2026-09-25), about 25 checkpoints:

Family Targets
Qwen Qwen3.6 (27B, 35B-A3B). Qwen3.5 (4B, 9B, 27B, 35B-A3B, 122B-A10B, 397B-A17B). Qwen3 (4B and 8B non-thinking, Coder-Next, Coder-30B-A3B)
Gemma Gemma 4 (12B, 31B, 26B-A4B)
MiniMax M2.5, M2.7
Kimi K2.5, K2.6, K2.7-Code
Others gpt-oss (20B, 120B), Llama-3.1-8B, GLM 5.1, Alpamayo 1.5/R1 10B

Vendor-published v1 drafters also exist, for example nvidia/Kimi-K2.7-Code-DFlash and modal-labs/Qwen3.5-397B-A17B-DFlash (mirrored as lmsys/ and z-lab/).

Drafter Config Keys

Engines rebuild the drafter from the dflash_config block in the drafter's config.json. The DFlash 2 keys are all optional. Their presence switches an engine to the DFlash 2 model class.

Key Applies to Meaning Example value
block_size DFlash, DFlash 2 Verify window the drafter was trained for (draft tokens = block size - 1) 8 (Qwen3.8-27B), 16 (NeMo example)
mask_token_id DFlash, DFlash 2 Token that fills block positions 1..K-1. Must match training 151669 (NeMo Qwen3 example)
target_layer_ids DFlash, DFlash 2 Target layers whose hidden states are injected For example [1, 9, 17, 25, 33]
conv_kernel_size DFlash 2 Convolution taps (2 = position + predecessor). Needs conv_group_size 2
conv_group_size DFlash 2 Channels that share one dynamic tap correction 16
selector_rank DFlash 2 Width of the selector's A/B token codebooks and context gate. Needs selector_top_k 256
selector_top_k DFlash 2 Candidates kept and scored per position 16

SGLang rejects a config where only one of a pair is set (conv_kernel_size/conv_group_size, selector_rank/selector_top_k). TensorRT-LLM reads mask_token_id and target_layer_ids from the checkpoint when unset and warns if max_draft_len + 1 differs from block_size.

Engine Flags

SGLang

Flag Meaning
--speculative-algorithm DFLASH Enables DFlash and DFlash 2 (built-ins: EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM, DFLASH, DSPARK)
--speculative-draft-model-path <repo> Drafter checkpoint
--speculative-dflash-block-size <n> DFLASH only. Alias of --speculative-num-draft-tokens
--speculative-num-draft-tokens <n> Verify window length for DFLASH
--speculative-draft-attention-backend <b> Drafter attention backend: flashinfer, fa3, fa4, triton, ascend, trtllm_mha (DFLASH rejects trtllm_mha when its per-layer needs are not met)
--speculative-dflash-draft-window-size <n> Sliding window for the drafter's local KV cache
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 Env var used in the LMSYS Qwen3.5-397B launch command

vLLM

--speculative-config key Meaning
method "dflash" for DFlash and DFlash 2 (vLLM can also infer it when the drafter name contains "dflash")
model Drafter checkpoint
num_speculative_tokens K draft tokens. DFlash uses one bonus query plus K mask queries, so K = block size - 1

llama.cpp

Flag Meaning
--spec-type draft-dflash DFlash drafting. DFlash 2 GGUFs are detected automatically
-hfd / --spec-draft-hf <user>/<model>[:quant] Drafter from a Hugging Face repo
-md <path> Local drafter GGUF
--spec-draft-n-max <n> Tokens to draft per step (env LLAMA_ARG_SPEC_DRAFT_N_MAX)

TensorRT-LLM (DFlashDecodingConfig)

Field Meaning
max_draft_len Draft tokens per step. Set to block_size - 1 of the checkpoint
speculative_model Path or HF id of the drafter
mask_token_id, target_layer_ids Read from the drafter config when None
attention_backend Cross-attention backend: VANILLA (default, runs anywhere), TRTLLM (SM100/SM103 only), FA4 (SM90 only)
skip_ctx_buffer_budget_check Skip the conservative config-time memory check for the context K/V buffers
decoding_type: DFlash YAML form for trtllm-serve / trtllm-bench

dflash CLI (0.1.0)

Command Backends Key options
dflash generate <backend> transformers, mlx, openai --model, --draft, --reasoning, --temperature, --top-p, --top-k, --block-size, --draft-bits, --base-url
dflash benchmark <backend> transformers, mlx, openai --dataset (gsm8k, math500, humaneval, mbpp, mt-bench), --num-prompts, --concurrency, --max-samples

Install extras: pip install dflash (client for an OpenAI-compatible server) or pip install "dflash[local]" (MLX on Apple Silicon, Transformers on Linux). Backend coverage: Transformers runs DFlash 2 for Muse-Glimmer-30B and DFlash for Qwen3 and Llama-3.1-8B. MLX runs DFlash 2 for Qwen3.8-27B and DFlash for Qwen3, Qwen3.5, Qwen3.6, and Gemma 4.

Reasoning controls: Qwen3.8 uses reasoning_effort (low, medium, xhigh, default xhigh). Muse Glimmer uses reasoning_strength (low, medium, high, xhigh, default high).

NeMo AutoModel Training Config

Values from examples/speculative/dflash/qwen3_dflash2.yaml (recipe TrainDFlash2Recipe, entry point nemo_automodel.recipes.llm.train_dflash2).

Field Example Meaning
target_model_name_or_path Qwen/Qwen3-8B Frozen target
draft_num_hidden_layers 5 Drafter depth
block_size 16 Tokens drafted per block
num_anchors 256 Blocks sampled per sequence per step
loss_decay_gamma 7.0 Position decay of the block loss
mask_token_id 151669 Required. A reserved, unused token, never pad
attention_backend flex_attention Or sdpa as the portable fallback
conv_kernel_size / conv_group_size 2 / 16 DFlash 2 convolution
selector_rank / selector_top_k / selector_loss_weight 256 / 16 / 1.0 DFlash 2 selector. Loss = base loss + weight x selector loss
seq_length, num_epochs, lr 3072, 6, 6.0e-4 Training schedule in the example

Supported target architectures for DFlash 2 training: Qwen3ForCausalLM, Qwen3MoeForCausalLM, and the Qwen3.5 family (Qwen3_5*ForCausalLM, Qwen3_5*ForConditionalGeneration, which is what Qwen3.8-27B ships as). Kimi K3 is registered for DFlash (v1) training only. DFlash 2 logs train/base_accept_len, train/candidate_recall, and train/selector_loss in addition to the DFlash metrics.

Benchmarks

H200 (SGLang, Qwen3.8-27B, block size 8 = 7 draft tokens)

Source: the incoai/Qwen3.8-27B-DFlash2 model card. The GSM8K concurrency-1 figure (236.1 tok/s, 3.43x) is also in the SGLang PR #35371 description and the v0.5.19 release notes. FlashAttention 3 for target and draft. Model-default sampling (temperature 1.0, top-p 0.95, top-k 20), xhigh reasoning effort, 4096 max new tokens. Throughput = output tokens / wall time, speedup vs autoregressive:

Concurrency GSM8K MATH-500 HumanEval MBPP MT-Bench
1 236.1 tok/s (3.43x) 230.7 (3.34x) 214.6 (3.11x) 226.9 (3.29x) 184.0 (2.67x)
8 1,328.7 (2.84x) 1,368.3 (2.85x) 1,291.5 (2.67x) 1,328.0 (2.78x) 1,090.2 (2.27x)
32 1,922.5 (1.45x) 1,951.8 (1.30x) 1,799.0 (1.16x) 1,886.8 (1.25x) 1,525.3 (1.01x)

Native MTP on the same rig: 2.59x on GSM8K at concurrency 1, and below 1x on several tasks at concurrency 32 (0.77-0.94x). All speculative methods proposed 7 draft tokens per step in this comparison. SGLang v0.5.19 notes add one higher-load point: DFlash 2 is about 24% faster than DFlash at concurrency 64.

Acceptance Length (Inco AI blog)

Per-request mean acceptance length. Lossless rejection sampling, default sampling per model, block size 8 for Qwen3.8-27B and 16 for Muse Glimmer:

Target MTP DFlash DSpark DFlash 2
Qwen3.5-4B (mean of 5 tasks) 4.54 4.92 5.49 5.97
Qwen3.8-27B (vs native MTP) 4.28 — 3.62 4.80
Muse-Glimmer-30B (vs official DFlash) — 4.44 4.48 5.70

Module costs on a 5-layer Qwen3-4B-class drafter: path selector +2.0M params and +0.6% cycle latency. Convolutions +16.5M params (+3%) and +0.7% cycle latency. Both together +1.3%. DSpark-style correction heads: +77.8M params and +9.6%.

Engine PR Measurements

Source Rig Result
vLLM PR #52816 Qwen3.8-27B, GSM8K Acceptance 5.34 (DFlash 2) vs 4.27 (DSpark), +25.2%. 224.6 tok/s (3.51x) at concurrency 1. 2,759.4 tok/s (2.20x) at concurrency 32. Selector about 0.84% of step time at batch 1
SGLang PR #35371 Qwen3.8-27B 236.1 tok/s (3.43x) on GSM8K at concurrency 1. Convolution + selector cost 1.5-2.7% of step time, falling as batch size grows
llama.cpp PR #27342 Qwen3.8-27B Q4_K_M, Apple M5 Pro 18.89 tok/s decode, 1.81x, acceptance 5.03
ollama PR #17865 Qwen3.8-27B, Apple M5 Pro (MLX) About 2.45x vs autoregressive, 16.2% ahead of built-in MTP (author's sweep)
Splash README (vendor claim) Qwen3.8-27B 4-bit, 48 GB M5 Pro 2x the decode speed of the next-fastest engine. 282 ms first token with 32K context cached. Inco also claims 144 tok/s on an M5 Max

PR figures are the authors' own measurements, summarized from the PR descriptions.

NVIDIA Blackwell (TensorRT-LLM, DFlash v1 drafter)

gpt-oss-120b on 8x DGX B300 (Blackwell Ultra), SPEED-Bench coding: >15x higher throughput at 500-600 tok/s/user interactivity vs autoregressive, 1.5x higher than EAGLE-3 at the same point. Batch size 1 more than doubles interactivity. Interactivity speedups at matched concurrency: gpt-oss-120b, DFlash 2.3x avg vs EAGLE-3 1.7x. Llama-3.1-8B, DFlash 2.8x vs 2.2x. Single-GPU: Gemma-4-31B on vLLM/B300 up to 5.8x (MATH-500). Qwen3-8B on SGLang/B200 5.1x (MATH-500).

Google TPU v5p (JAX, vLLM tpu-inference, DFlash v1)

Average 3.13x tokens/s across datasets (peak about 6x on math. MATH-500 8.02 → 1.40 ms/token at K=16, greedy. MBPP 9.81 → 3.48 ms/token). Head-to-head with out-of-the-box Llama-3.1-8B checkpoints: DFlash 2.29x vs EAGLE-3 1.30x end-to-end serving speedup. Scaling: K=16 already captures >90% of the theoretical maximum. K from 16 to 128 adds less than one accepted token per step. Improving per-position acceptance is 2-3x more valuable than growing K.

SGLang Spec V2 and Qwen3.5-397B (LMSYS, DFlash v1)

  • Spec V2 overlap scheduler with DFlash: 11.4 → 15.3 ktok/s (>33%), Qwen3-8B on one B200 at concurrency 32.
  • Qwen3.5-397B-A17B (BF16) on 8x B200, HumanEval, greedy, thinking on: >4.3x the baseline throughput and 1.5x the MTP throughput at concurrency 1 (DFlash block size 16, MTP 7 steps). Higher throughput than native MTP at every tested concurrency from 1 to 32.

Ablations (LMSYS, Qwen3-4B, 5-layer drafters, acceptance / speedup)

Configuration GSM8K HumanEval MT-Bench
EAGLE-3 (5-layer) 4.2 / 2.1x 4.3 / 2.2x 3.1 / 1.4x
DFlash (both techniques) 4.2 / 3.3x 4.0 / 3.2x 3.0 / 2.2x
DFlash diffusion only 3.5 / 2.9x 3.5 / 2.9x 2.6 / 2.0x
DFlash injection only 4.8 / 2.4x 4.6 / 2.3x 3.4 / 1.5x

Production Data Points

  • CoreWeave's Kimi K2.7 Code endpoint pairs a custom NVFP4 quantization with a DFlash (v1) drafter on vLLM by default. It was the fastest provider for that model on Artificial Analysis at publication (281.9 output tok/s).
  • Xiaomi's MiMo v2.5-Pro-UltraSpeed uses DFlash to reach over 1,000 output tok/s (cited by LMSYS).
  • The Inco blog reported more than 3.5M Hugging Face downloads across DFlash models (Aug 2026).

Benchmark Caveats

  • The "15x" from NVIDIA is throughput at matched interactivity on an 8-GPU Pareto curve, not a batch-1 latency speedup. The 2.7-3.4x from the H200 model card describes a different operating point.
  • The two Qwen3.8-27B engine measurements disagree at concurrency 32: 1.16-1.45x on SGLang (model card) vs 2.20x on vLLM (PR #52816, GSM8K). Rigs and sampling settings differ. Benchmark your own stack.
  • Acceptance on Qwen3.8-27B differs by source: 4.80 (blog, default sampling) vs 5.34 (vLLM PR, GSM8K). The DSpark baseline also moves (3.62 vs 4.27).
  • The blog's +1.3% overhead is per draft-verify cycle on a 5-layer Qwen3-4B-class drafter. SGLang PR #35371 measures 1.5-2.7% of serving step time on Qwen3.8-27B.
  • Secondary summaries of the Google TPU post attribute the 3.13x average to Llama-3.1-8B-Instruct. The MATH-500 ms/token figures were recorded here as Qwen3-4B. Check the post before quoting either model.
  • Several numbers above come from vendor or author measurements. No independent third-party DFlash 2 benchmark was found as of 2026-09-25.

Sources