NVLS

NVLS (NVIDIA Low-latency Semantic) refers to a set of low-latency communication primitives for all-reduce and reduce-scatter operations implemented by NVIDIA on the Blackwell GPU architecture. In distributed transformer training and inference, the attention mechanism and mixture-of-experts routing require aggregating (all-reduce) or distributing (reduce-scatter) tensors across multiple GPUs or across multiple streaming multiprocessors (SMs) within a single GPU. These collective communication operations are a significant source of latency and bandwidth overhead, particularly in tensor parallelism and sequence parallelism strategies.

NVLS implements these communication operations with very low latency by leveraging Blackwell's hardware support for collective operations and by minimizing the synchronization points required. The implementation achieves deterministic results across different parallelism strategies, which is essential for the bitwise trainer-sampler alignment described above. A standard all-reduce implementation might use different reduction trees or synchronization patterns depending on the tensor parallelism configuration, leading to different numerical results; NVLS guarantees identical outputs regardless of the underlying parallelism topology.

The practical impact of NVLS is most visible in MoE architectures, where expert routing requires frequent all-to-all communication across GPUs. A model with 12B active parameters out of 276B total (the TML-Interaction-Small architecture) routes tokens to different experts on every forward pass, meaning that the inter-GPU communication bandwidth and latency directly impact the achievable throughput. NVLS reduces this overhead to the point where real-time interaction at 200ms micro-turn granularity becomes feasible with the available compute.

NVLS is specific to NVIDIA Blackwell architecture and represents one of the hardware-software co-design choices that makes high-performance interaction model inference possible. The primitives are exposed through NVIDIA's communication libraries and are used in the custom kernels implemented for the interaction model system, particularly for the attention and MoE layers.