Pipeline Parallelism

Pipeline parallelism trains or runs a large model by splitting its layers across several devices. Each device owns a stage of the model, and data moves through the stages in order. This makes models possible when the full set of layers or activations cannot fit comfortably on one accelerator.

The usual mechanism is to divide a batch into microbatches and send them through the pipeline like items on an assembly line. While one stage processes a later microbatch, another stage can process an earlier one, increasing hardware utilization. Training schedules also coordinate backward passes so gradients flow through the same partitioned stages.

The benefit is memory capacity and throughput for very large networks, especially when combined with data parallelism or tensor parallelism. The cost is coordination: pipeline bubbles waste time, imbalanced stages slow everyone down, and communication overhead can erase the gains. It also constrains model partitioning, because layers that depend on each other must be placed where the schedule can support them.

Open questions include how to partition models automatically as architectures become more irregular. Mixture-of-experts layers, dynamic routing, activation checkpointing, and heterogeneous clusters all complicate the schedule. For inference, the unresolved balance is between lower latency for a single request and higher throughput across many requests.