Topic
Serving, quantization, latency, GPUs, inference engines, and deployment economics.
Latest Signals
Reduced Matrix Multiplication (RMM) is a training-free, input-adaptive method that lowers Transformer inference costs by selecting informative slices in matrix multiplications without changing model weights. This approach reduces the high-dimensional matrix products common in large language model i...
InFactPlanner addresses sustainability challenges in large-scale LLM inference by enabling operators to compare deployment options before infrastructure build-out. It focuses on reducing energy use, carbon emissions, and water consumption while maintaining service quality.
A lecture series on large language model (LLM) inference is available on YouTube, discussed briefly on Hacker News. The series covers key aspects of LLM inference techniques.
A Hacker News discussion highlights improved large language model inference speeds using llama.cpp on Apple Silicon and macOS virtual machines. The conversation includes community insights on performance optimizations.
This work introduces a sequential-dispatch measurement method to characterize and reduce the overhead of WebGPU per-operation dispatch in browser-based large language model (LLM) inference. It highlights that naive single-operation dispatch leads to significant inefficiencies.
No summary available yet.
Serving, quantization, latency, GPUs, inference engines, and deployment economics.