Cross-Entropy Loss

Cross-entropy loss is a standard objective function used in machine learning, particularly for classification tasks. It comes from information theory and measures how closely the model's predicted probability distribution matches the true distribution of labels. For a single example, the loss equals the negative log-probability assigned to the correct class, so the model receives a large penalty when it assigns low probability to the right answer. When summed or averaged across a training dataset, this quantity gives the optimizer a differentiable signal to minimize during gradient descent.

Cross-entropy loss works well for training neural networks with softmax output layers. When a model confidently predicts the wrong class, the logarithmic penalty becomes very large, producing strong gradient signals that push weights toward correction. When the model assigns high probability to the correct class, the loss approaches zero and gradients shrink. The result is fast early learning and stable convergence, behavior that mean squared error does not provide as cleanly for classification.

Cross-entropy loss extends to multi-class problems as categorical cross-entropy and to binary classification as binary cross-entropy, also called log loss. In both cases, the formula is equivalent to maximizing the log-likelihood of the correct labels under the model's predicted distribution, which connects the loss function to maximum likelihood estimation.

Cross-entropy loss is the default loss for classification problems in deep learning, including image recognition and natural language processing. It works with backpropagation, stays numerically stable when paired with log-softmax implementations, and has a clear statistical basis. Practitioners use it to diagnose model behavior, read training curves, and design learning systems.