Hugging Face Revamps Its Kernels: More Speed and Efficiency in AI

What happened?

Hugging Face, the leading platform for artificial intelligence models, has announced a significant overhaul of its kernels (computational cores) on its official blog. These kernels are low-level components that execute critical mathematical operations on GPUs and other accelerators. The update, detailed in the post '🤗 Kernels: Major Updates', introduces optimizations in operations such as scaled dot-product attention, layer normalization, and activation functions, among others. According to the blog, fused kernels have been implemented that reduce memory overhead and improve utilization of streaming multiprocessors on modern GPUs. For example, the new implementation of scaled dot-product attention uses tiling and warp-level matrix multiply-accumulate (MMA) techniques to achieve up to 2x performance on certain batch and sequence sizes, based on internal tests with models like Llama 2 and BLOOM. Additionally, support for 4-bit quantization has been added to the attention kernels, allowing memory usage to be reduced without sacrificing precision.

Why is it important?

Kernels are the heart of deep learning performance. Small improvements in their efficiency can translate into significant reductions in training time and computational costs. This update is particularly relevant because it addresses bottlenecks in modern models like transformers, which heavily rely on attention operations. Moreover, Hugging Face has optimized the kernels for recent GPU architectures, such as NVIDIA Hopper (H100) and AMD MI300, maximizing hardware utilization. Specifically, the kernels now leverage Hopper-specific instructions like Tensor Memory Accelerator (TMA) and Distributed Shared Memory, which can improve performance by up to 30% on attention tasks according to NVIDIA benchmarks. For AMD users, compatibility with ROCm has been improved, allowing models like Stable Diffusion XL and Falcon to run more efficiently on AMD hardware. This is especially relevant in a context where the shortage of NVIDIA GPUs has led many companies to consider AMD alternatives. The update also includes optimized kernels for CPUs via Intel's oneDNN library, extending the reach to environments without GPUs.

Consequences and outlook

For developers, this update means they can train larger models in less time or reduce cloud costs. Companies using Hugging Face for inference will see improvements in latency and throughput, which is critical for real-time applications like chatbots or virtual assistants. For example, in tests with the GPT-2 model, inference latency was reduced by 15% and throughput increased by 20% on an A100 GPU. In the long term, these optimizations reinforce Hugging Face's position as the preferred ecosystem for AI research and production, directly competing with proprietary solutions like NVIDIA TensorRT and Triton Inference Server. However, unlike these, Hugging Face maintains an open-source approach, allowing the community to contribute and audit the kernels. This contrasts with NVIDIA's closed approach, which often requires specific licenses. Additionally, the update facilitates portability across hardware, reducing dependence on vendor-specific optimizations, a key factor in a market where accelerator heterogeneity is increasing. According to the blog, plans are in place to integrate these kernels into the transformers library by default in future versions, further democratizing access to these improvements.

Hugging Face's updated kernels not only improve performance but also facilitate portability across different hardware, reducing dependence on vendor-specific optimizations. This is especially relevant in a context where companies like AWS, Google Cloud, and Microsoft Azure offer instances with GPUs from different manufacturers.

What readers should know

The changes are transparent to the end user: upon updating the transformers or accelerate library, the new implementations activate automatically. Hugging Face has confirmed that the kernels are available in their GitHub repository and integrate with PyTorch and JAX. Developers are recommended to test their models with the latest version to take advantage of the improvements. To do so, they can install the latest version of transformers from the main repository (pip install git+https://github.com/huggingface/transformers.git) or wait for the next stable release. The blog also warns that some experimental kernels may require manual activation via configuration flags, such as use_flash_attention=True for optimized attention. Additionally, Hugging Face has published detailed benchmarks in their GitHub repository, comparing the new kernels with previous implementations and with other libraries like FlashAttention-2 and xFormers. Results show that Hugging Face kernels are competitive and even outperform in certain scenarios, especially on models with long sequences. Finally, the company has announced that it will continue developing kernels for new architectures, such as NVIDIA's upcoming Blackwell GPUs and AMD's RDNA 4, ensuring the platform remains at the forefront of AI performance.