Random Network Distillation (RND)

Random Network Distillation (RND) is an exploration method for deep reinforcement learning introduced by Burda, Edwards, Storkey, and Klimov (2018). It estimates state novelty through the prediction error of a learned network against a fixed, randomly initialized target network. The method maintains two networks with identical architectures. The target network has its weights drawn once at initialization and never updated. The predictor network has its weights trained to predict the target network's output given the current observation as input. For states the predictor has seen many times, its predictions converge to the target's outputs and the prediction error is low. For novel states, the predictor has not yet learned the mapping and the error is high. The prediction error is then used as an intrinsic reward bonus, encouraging the policy to visit under-explored regions of the observation space.

The target network, unlike a learned forward model, never adapts to the data. The predictor's error can only decrease through repeated visitation, not through the target becoming easier to predict. This makes the novelty estimate monotonic in visitation and avoids the "noisy-TV" problem that affects prediction-error-based curiosity. When the environment contains truly random observations, a learned forward model quickly memorizes them and stops providing a useful exploration signal. The random target's output remains incompressible for the predictor, so the bonus persists. RND is computationally cheap relative to learned-world-model approaches and has become a standard baseline in deep RL exploration benchmarks, particularly for hard-exploration Atari games like Montezuma's Revenge where dense extrinsic rewards are absent.

Variants have addressed RND's limitations. Distributional RND generalizes the target from a single fixed network to an ensemble of randomly drawn targets. Anti-exploration studies have identified conditions under which RND's bonus can mislead agents into dead-end states that produce high novelty but no downstream progress. Active imitation learning extensions have used RND bonuses to identify trajectories worth asking an expert to demonstrate. RND sits within the broader family of intrinsic-motivation methods alongside the Intrinsic Curiosity Module (ICM), which uses a learned forward model instead of a fixed target, and count-based exploration bonuses, which estimate visitation directly through hash-table counts or pseudo-counts in learned embeddings.

RND is one of the first exploration methods to scale reliably to high-dimensional pixel observations without a hand-crafted state representation. It established the template that subsequent count-based and embedding-based methods generalize: a fixed reference, a learned predictor, and a prediction error that converts visitation frequency into an exploration bonus. The architecture has been re-used in offline reinforcement learning, model-based RL, and large-language-model fine-tuning for tasks where dense reward signals are unavailable.