GAIL (Generative Adversarial Imitation Learning)

Generative Adversarial Imitation Learning (GAIL) is a reinforcement learning technique that enables an agent to acquire complex behaviors by observing expert demonstrations, bypassing the need for a hand-crafted reward function. Introduced by Jonathan Ho and Stefano Ermon in 2016, GAIL draws on the adversarial training framework of Generative Adversarial Networks (GANs) and applies it to imitation learning.

GAIL trains two competing models simultaneously. A generator, which is the learning agent's policy, produces actions in response to observed states, attempting to replicate the behavior seen in expert demonstrations. A discriminator network is trained in parallel to distinguish between state-action pairs drawn from the expert data and those generated by the current policy. The discriminator's output becomes an implicit reward signal, guiding the generator toward expert-like behavior. The adversarial loop continues until the discriminator can no longer reliably tell the agent's actions apart from the expert's.

GAIL differs from classical imitation learning approaches like behavioral cloning in its handling of distributional shift. Behavioral cloning trains a policy in a supervised fashion on expert trajectories, but the agent can quickly encounter states not covered by the training data and compound errors over time. GAIL addresses this by using on-policy rollouts during training, so the agent learns to recover from its own mistakes rather than memorizing expert sequences. This makes GAIL well-suited for long-horizon tasks where compounding errors are a concern.

GAIL has been applied in robotics, autonomous driving, game-playing agents, and simulated locomotion tasks, where defining a precise reward function is costly or brittle. Its main limitations include sample inefficiency, which requires many environment interactions, and sensitivity to the quality and diversity of expert demonstrations. GAIL remains a foundational method in inverse reinforcement learning and imitation learning research, with extensions that improve its scalability and applicability to real-world settings.