Summary
According to a post on X by @caoshuxun, too many LLM inference results now train a speculator on benchmark prompts to inflate their numbers, and report P90 figures that do not reflect real speed. The post quotes @zakariaornot, who says a 1,740 tokens-per-second decode on 4×B300 holds only for a showcase prompt with a 5.6-token acceptance rate, and drops to about 1,200 on average coding workloads. @rogerw0108 added a point about measuring only batch size 1, though the snippet cuts off before the details.
AI-assisted summary based on the listed source.
What happened
@caoshuxun said too many inference results now train a speculator on the benchmark prompts to game the numbers and report P90 figures that don't reflect real speed, quoting @zakariaornot, who said a 1,740 tokens-per-second decode on 4×B300 holds only for a showcase prompt with a 5.6-token acceptance rate and drops...
Why it matters
Engineers comparing speculative-decoding speed claims should check the prompts, acceptance rates and batch sizes behind any headline figure. The gap between the showcase and average coding numbers in the post shows how much a result can depend on the workload. Watch whether inference providers publish results across varied workloads rather than single prompts.
Signal Intelligence
Signal Strength 93%
Technical label WATCH
Public Interest 28
Category RESEARCH
Reader Depth TECHNICAL
Signal Strength reflects source quality, relevance, freshness and evidence. Public Interest helps organize discovery; it is not proof of truth.
Public Interest components
Recognizable Entity Score 0
Practical Impact Score 0
Novelty Interest Score 94
Consequence Score 34
Curiosity Score 0
Shareability Score 45
Why this is here
VQV surfaced this signal because it is recent, relevant to LLM Inference, connected to Scobleizer News Pulse (X).