Diffusion Transformer

Diffusion Transformer (DiT) is a diffusion model that uses a transformer, rather than a convolutional U-Net, as its denoising backbone. William Peebles and Saining Xie introduced the architecture in 2022 (arXiv:2212.09748, ICCV 2023). It is a latent diffusion model whose image is broken into patches, embedded as tokens, and denoised by a stack of transformer blocks conditioned on the diffusion timestep and any class or text label. Their largest model, DiT-XL/2, set a new state of the art on class-conditional ImageNet generation.

The paper's central finding was a scaling law: adding transformer depth or width, or processing more, smaller patches, buys higher generation quality (lower FID) for more compute (Gflops). This is the same relationship that scaling laws describe for language models. It gave diffusion modeling access to the same recipe, and much of the same hardware and training infrastructure, that had already been optimized for transformers in language and vision.

DiT displaced the U-Net as the default architecture for large-scale generative image and video systems, underpinning models such as Stable Diffusion 3 and Sora. Its self-attention cost still scales quadratically with the number of tokens. This is punishing for video, where thousands of spatiotemporal patches must attend to each other. A large body of follow-up work applies sparse, linear, and low-rank attention variants to make DiTs practical at video scale and higher resolution.