Alternating Attention

Alternating attention is a transformer architecture pattern that interleaves two or more distinct attention mechanisms across the layer stack, with each mechanism capturing a different structural dependency in the input. The pattern was introduced in PAANet (Progressive Alternating Attention, 2021) for medical image segmentation, which alternated between spatial and channel attention, and has since become widespread for inputs with multi-axis structure.

The most common instantiation in 2026 is two-axis attention on a 2D-token grid: one axis attends along one dimension (e.g., time within a series) with causal masking, while the other axis attends along the other dimension (e.g., across multiple time series or variables) without causal masking. TimesFM-3 uses alternating causal-temporal-attention and full-variate-attention across the layer stack to capture both within-series temporal patterns and cross-series correlations. TurboVGGT and PaceVGGT (2026) use alternating attention across views for 3D visual geometry reconstruction. MViewRouter (2026) uses multi-view alternating attention for geometric equivariance.

The architectural advantage over single-axis attention is that each mechanism can specialize — temporal attention learns within-series dynamics, variate attention learns cross-series correlations, spatial attention learns 2D layout, etc. — without forcing a single attention pattern to handle all dependencies. The pattern is closely related to axial-attention (used in axial transformers for images), but generalizes the idea to multiple arbitrary axes and to alternating rather than parallel patterns.

Alternating attention is distinct from cross-attention (which mixes two token streams via attention), self-attention (which attends within one stream), and sparse-attention (which attends selectively within one stream). It is an architectural pattern at the layer-stacking level rather than at the within-layer attention mechanism level.