New paper on speculative decoding cuts inference latency by 40 percent

Researchers from UC Berkeley released a draft on async speculative decoding today. The method allows smaller draft models to verify tokens without blocking the main generation loop. This could significantly reduce cloud compute costs for high-throughput AI applications.