Speculative Decoding Deep Dive III: EAGLE, DFlash

EAGLE

EAGLE stands for Extrapolation Algorithm for Greater Language-model Efficiency. It drafts the next hidden state, then reuses the frozen target LM head to turn that state into a token.

Core idea

Notation:

• $f_j$ = target last hidden state, the direct input to the LM head;
• $p_{j+1}$ = next-token distribution from the frozen LM head:

$$p_{j+1}=\mathrm{LMHead}(f_j)$$

• $T_{1:j}\to f_j\to p_{j+1}\to t_{j+1}$ = normal decoding process.

EAGLE's core ideas:

1. Features are smoother than tokens. Consecutive hidden states $f_1,f_2,\dots$ change a little at a time, so a single extra layer can extrapolate the next one. However, token IDs do not: they jump from one discrete word to another.
2. The next feature depends on the sampled token. From the same $f_j$, the LM head might sample am or always, and those two choices produce different $f_{j+1}$. Treat the drafter as a dynamics model: current state $f_j$ plus action $t_{j+1}$ gives the next state. That mapping is well-defined. Predicting $f_{j+1}$ from $f_{1:j}$ alone would mix those branches.

$$f_{j+1}\approx g(f_{1:j},t_{j+1})$$

Problem with Medusa:

Extra heads predict later tokens from the current context only, so they never see the token just sampled: $P(t_{j+2}\mid T_{1:j})$.

EAGLE-1 — autoregressive feature drafting

Feeds the sampled token into the next step: $P(t_{j+2}\mid T_{1:j},t_{j+1})$.

Train

• Train 1 FC (last feature + shifted token embedding) + 1 decoder layer. Embedding and LM head stay frozen.
• Feature regression + soft-target CE vs $p_{i+2}$ (not one-hot). Noise $\mathcal{U}(-0.1,0.1)$ on input features so the head tolerates its own drift.

$$L=SmoothL1(f_{i+1},{\hat{f}}_{i+1})+w_{\mathrm{cls}}\mathrm{CE}(p_{i+2},{\hat{p}}_{i+2}),w_{\mathrm{cls}}=0.1$$

Inference

• Drafter predicts $\hat{f}$ from $F_{1:i}$ and shifted tokens $T_{2:i+1}$; frozen LM head → token; feed $(\hat{f},\hat{t})$ back.

$${\hat{f}}_{i+1}=D(F_{1:i},T_{2:i+1}),{\hat{p}}_{i+2}=\mathrm{LMHead}({\hat{f}}_{i+1}),{\hat{t}}_{i+2}\sim {\hat{p}}_{i+2}$$

• Tree draft; one target pass (tree attention + multi-round speculative sampling) — lossless.

EAGLE-2 — context-aware dynamic tree

• Same drafter / training as EAGLE-1; only the tree is dynamic.
• Calibrated $c_j\approx p_j$ scores paths as $V_i={\prod }_j{c}_j$ without the target.
• Expand top-$k$, keep top-$m$ by $V$, then verify. $+20$–$40\%$ ($\tau \approx 4$–$5.5$), still AR.

EAGLE-3 — free representation + training-time test + feature fusion

Key innovation vs EAGLE / EAGLE-2: free representation.

• EAGLE-1. Simulate the target hidden state:

$${\hat{f}}_{i+1}\approx f_{i+1},L_{\mathrm{feature}}=SmoothL1({\hat{f}}_{i+1},f_{i+1})$$

• EAGLE-3. Guess the next token, not copy $f$. Drop $L_{\mathrm{feature}}$. The $d$-dim output is a free representation $a$ — still an intermediate vector, just not regressed onto $f$. Only the frozen LM head must match the target:

$$a\stackrel{\mathrm{LMHead}}{\to }{q}_{\mathrm{draft}},q_{\mathrm{draft}}(t)\approx p_{\mathrm{target}}(t)$$

Three changes:

• Drop feature regression. Train only so $\mathrm{LMHead}(a)$ matches the target tokens. First-token accept ($0$-$\alpha$) jumps — capacity is no longer spent copying $f$.
• Training-time test. $a\not\approx f$, so later steps collapse if trained on teacher $f$. Fix: unroll on own outputs at train time ($f_1\to a_2\to a_3\to \dots$), same as inference. Accept stays flat through $5$-$\alpha$.
• Multi-layer feature fusion. The top feature is only good for the next token. Fuse low/mid/high; verified positions use $g$, later steps use own $a$, plus the shifted token embedding:

$$g=\mathrm{FC}([l;m;h])$$

EAGLE-1. Stay in last-layer feature space. After the verified prefix, seed with target $f$; each later step predicts $\hat{f}$ and feeds that $\hat{f}$ back:

$$(f_{\mathrm{can}},e_I)\to {\hat{f}}_I\to \mathrm{do},({\hat{f}}_I,e_{\mathrm{do}})\to {\hat{f}}_{\mathrm{do}}\to \mathrm{it}$$

EAGLE-3. Two input regimes. Verified positions use fused multi-layer $g$; later steps recycle a free latent $a$ that is not trained to match $f$:

$$(g_{\mathrm{How}},g_{\mathrm{can}},e_I)\to a_I\to \mathrm{do},(a_I+e_{\mathrm{do}})\to a_{\mathrm{do}}\to \mathrm{it}$$

	After verified prefix	Later draft steps
EAGLE-1	target feature $f$	predicted $\hat{f}$
EAGLE-3	fused feature $g$	free latent $a$

DFlash

DFlash-1 drafts the whole speculative block in one parallel pass. DFlash-2 keeps that architecture and fixes two quality problems: weak inter-position dependence and naive Top-1 selection.

DFlash-1: parallel block + KV injection

Target on The quick brown produces two things: the next token (anchor) and multi-layer hidden features.

The quick brown ──► Target ──► fox                 # confirmed anchor
                         └──► {H^(l)} ──► Fuse ──► H_t

• Anchor block. Seed with the verified token; the rest are masks. Training copies this: sample a clean start, mask the next $\gamma$ positions, predict them together.

$$[\mathrm{fox},m_1,m_2,m_3,m_4]$$

• Shared embedding / LM head. $E(\mathrm{fox}),E(\mathrm{MASK}),\dots$ and the draft logits reuse the frozen target tables. This LM head is draft logits, not verification.

$$q_t=\mathrm{softmax}(W_{\mathrm{LM}}{h}_t)$$

• KV injection (every draft layer). Ordinary attention is $Q,K,V$ from $H_d$ only. DFlash prepends $H_t$ as extra KV memory. EAGLE-3 fuses $g$ only at the input; injecting $H_t$ into every layer lets acceptance keep rising with depth (5–8 layers vs 1).

$$H_t=\mathrm{RMSNorm}(W_c[H^{(l_1)};\dots ;H^{(l_5)}])$$ $$Q_i=W_i^Q{H}_d,K_i=[W_i^K{H}_t;W_i^K{H}_d],V_i=[W_i^V{H}_t;W_i^V{H}_d]$$

H_t ──► Layer 1 KV, …, Layer N KV     # not input-only
MASK… ──► Q ──► draft layers ──► LM head ──► block

• Block-parallel vs AR. AR draft cost grows with $\gamma$. All masks go in one pass. $T_{\mathrm{draft}}\downarrow$, $\tau \uparrow$. A 5-layer DFlash drafting 16 beats a 1-layer EAGLE-3 drafting 8. ~6×, up to 2.5× over EAGLE-3.

$$T_{\mathrm{draft}}=\gamma t_{\mathrm{step}}$$ $$T_{\mathrm{draft}}=t_{\mathrm{parallel}}\ll \gamma t_{\mathrm{step}},L=\frac{T_{\mathrm{draft}}+T_{\mathrm{verify}}}{\tau }$$

• Training. Freeze target, embedding, LM head. Weight early positions (an early reject voids the suffix). Bidirectional attention inside a block, none across blocks.

$$w_k=\exp (-(k-1)/\gamma )$$

• Verify. Target checks the draft in one pass; speculative sampling — lossless. Shared LM head $\ne$ verification.

DFlash-2: path selector + local conv

DFlash-1 drafts every mask in one pass, so neighbors are not finalized. DFlash-2 adds two inference steps.

Path selector. After the parallel draft, keep Top-$k$ candidates per position. Score every adjacent pair $(a,b)$ in one shot, then walk $a\to b\to \cdots$ to pick one path. The target still verifies that path — lossless.

$$S_t(a,b)=U_t(b)+⟨A(a)\odot H(h_t),B(b)⟩$$

$U_t(b)$ is the DFlash logit; $A,B$ embed the two tokens; $H(h_t)$ gates by context.

Dynamic conv. Inside each draft layer, a two-tap conv lets position $t$ read $x_t$ and $x_{t-1}$ (first position reads the last verified token). The $y_t$ outputs do not depend on each other, so the block stays parallel. Attention still sees global / target context; conv only sees the immediate predecessor.

$$\mathrm{Conv}(x{)}_t=k_{t,0}\odot x_t+k_{t,1}\odot x_{t-1}$$

Summary

Method	Architecture	$T_{\mathrm{draft}}$ vs $\gamma$	Speedup
Classic SD	Small AR drafter predicts future tokens sequentially; target verifies the block.	$\gamma \cdot t_{\mathrm{step}}$	≈2–3×
Medusa-1	Parallel future-token heads on the target hidden state; candidates form a tree for verify.	one parallel head pass; no $\gamma$ sequential draft steps	≈2–3×
EAGLE-1	AR drafts future target features, conditioned on the target feature and shifted sampled token, then maps them to tokens.	$\gamma$ sequential steps, lightweight drafter	≈2.7–3.5×
EAGLE-2	EAGLE-1 feature-level AR draft + context-aware dynamic tree (more branches on high-confidence paths).	$\gamma$ sequential steps; tree changes allocation, not sequentiality	≈3.05–4.26×
EAGLE-3	AR predicts draft tokens directly; conditions on fused low/mid/high target features; training matches inference.	$\gamma$ sequential steps	up to ≈6.5×
DFlash	Fused multi-layer target features KV-injected into every layer of a lightweight block-diffusion drafter; whole masked block in one pass.	$t_{\mathrm{parallel}}$; ≈independent of $\gamma$ for moderate blocks	up to ≈6.1×
DFlash-2	DFlash + short dynamic convs for neighbor dependence + Top-$k$ path select before verify.	$t_{\mathrm{parallel}}$ + small parallel select / short walk	≈2.7–3.4× AR throughput

References

• Li et al., 2024. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv 2401.15077
• Li et al., 2024. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. arXiv 2406.16858
• Li et al., 2025. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv 2503.01840
• Chen et al., 2026. DFlash: Block Diffusion for Flash Speculative Decoding. arXiv 2602.06036
• Inco AI, 2026. DFlash 2: Keep Drafting Parallel. inco.ai/blog/dflash2
• Ankner et al., 2024. Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding. arXiv 2402.05109