Summary
According to a post on X by @rohanpaul_ai, a Cohere paper tested speculative decoding on 10 mixture-of-experts (MoE) models ranging from 30B to 1T parameters on H100 GPUs. The post says neighboring tokens shared 20% to 45% of their experts, well above the 4% to 12% expected. For Qwen3-235B in FP8, checking drafted tokens ran at 89% of a normal decoding step's speed, compared with 52% predicted. The post presents this as evidence that the technique works well on MoE models even at batch size 1.
AI-assisted summary based on the listed source.
What happened
Testing 10 MoE models from 30B to 1T parameters on H100s, Cohere found neighboring tokens share 20% to 45% of their experts, far more than the 4% to 12% expected. For Qwen3-235B in FP8, checking drafted tokens ran at 89% of a normal decoding step's speed versus 52% predicted. Why it matters: Agent workloads often...
Why it matters
The post notes that agent workloads often run at low batch sizes, which makes this a potentially low-cost way to speed up MoE model responses. Teams running MoE models for agents may want to watch for the full paper to check these results and their conditions. Until then, the figures come from the post's description of the research rather than from the paper itself.
Signal Intelligence
Signal Strength 88%
Technical label WATCH
Public Interest 24
Category RESEARCH
Reader Depth TECHNICAL
Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.
Public Interest components
Recognizable Entity Score 0
Practical Impact Score 8
Novelty Interest Score 70
Consequence Score 18
Curiosity Score 16
Shareability Score 42
Why this is here
VQV surfaced this signal because it is recent, relevant to Open Source LLMs, connected to Scobleizer News Pulse (X).
