Modern LLMs are autoregressive models that generate one token at a time, which is inefficient on parallel hardware. These works accelerate generation by processing multiple tokens per forward pass.

Today, training most powerful models often takes significant resources. Our research aims to make large-scale training more efficient and accessible to the entire machine learning community.

Large-scale machine learning

Language is one of the key forms of communication. We study methods of language representation and understanding to simplify human-computer interactions.

Natural language processing 

Speculative and parallel decoding


We introduce SpecExec, a speculative decoding method that delivers up to 20 tokens per iteration and up to 15x speedups in offloading settings. This enables usable LLM inference speeds for users with limited GPU VRAM sizes, who have had to use offloading or settle for lower quality models until now. 

Speculative and parallel decoding

Posts

SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices