Colfax Research

LLM Inference is divided into a prefill phase and a decode phase. During prefill, the model processes a large number of input tokens and populates a key-value (KV) cache. During decode, it autoregressively generates one or a few new tokens at a time using the cached keys and values. In this blog post, we discuss […] The post Optimization diaries: S/P ping-pong for FlashAttention-4 decode appeared first on Colfax Research.

LLM Inference is divided into a prefill phase and a decode phase. During prefill, the model processes a large number of input tokens and populates a key-value (KV) cache. During decode, it autoregressively generates one or a few new tokens at a time using the cached keys and values. In this blog post, we discuss […] The post Optimization diaries: S/P ping-pong for FlashAttention-4 decode appeared first on Colfax Research.

In this blog post we discuss the backward pass of FlashAttention-4 (FA4) on NVIDIA Blackwell GPUs. For head dimension 128, FA4 backward is highly performant, achieving 1237 TFLOP/s, or about 55% of peak compute throughput on a B200 GPU. However, at head dimension 64, the same kernel achieves only 30–37% of peak compute throughput for long sequence lengths. This suggests that there is plenty of room for improvement in this case. We will describe a optimization that leverages spare tensor memory (TMEM) to improve FA4 backward performance for head dimension 64. The post Optimization diaries: Improving FlashAttention-4 backward pass kernel design for head dimension 64 appeared first on Colfax Research.

In this blog post we discuss the backward pass of FlashAttention-4 (FA4) on NVIDIA Blackwell GPUs. For head dimension 128, FA4 backward is highly performant, achieving 1237 TFLOP/s, or about 55% of peak compute throughput on a B200 GPU. However, at head dimension 64, the same kernel achieves only 30–37% of peak compute throughput for long sequence lengths. This suggests that there is plenty of room for improvement in this case. We will describe a optimization that leverages spare tensor memory (TMEM) to improve FA4 backward performance for head dimension 64. The post Optimization diaries: Improving FlashAttention-4 backward pass kernel design for head dimension 64 appeared first on Colfax Research.

This article is a continuation of our series on NVFP4 blockscaling on SM12x GPUs. In Part 1, we covered relevant PTX instructions, scale-factor layout details, and implementation details in CuTe DSL, including how to convert a CUTLASS dense GEMM example into an NVFP4 blockscaled GEMM. In this article, we optimize the NVFP4 GEMM from Part […]