MLE (Maximum Likelihood Estimation)

Maximum Likelihood Estimation (MLE) is a fundamental statistical technique for estimating the parameters of a probability model given observed data. The central idea is straightforward: among all possible parameter settings, choose the ones that make the observed data most probable. Formally, given a dataset and a parameterized probability distribution, MLE constructs a likelihood function representing the joint probability of the data as a function of the parameters. Maximizing this function — typically by taking its derivative, setting it to zero, and solving — yields the MLE estimates. In practice, it is almost always more convenient to maximize the log-likelihood, which converts products into sums and is mathematically equivalent since the logarithm is a monotonic transformation.

MLE became central to machine learning because many canonical learning algorithms are implicitly or explicitly performing maximum likelihood estimation. Training a logistic regression model with cross-entropy loss, fitting a Gaussian mixture model with the EM algorithm, or optimizing a neural network with negative log-likelihood objectives are all instances of MLE. The framework provides a principled, probabilistic justification for these procedures and connects model training to the broader language of statistical inference.

MLE has several attractive theoretical properties that explain its widespread adoption. Under mild regularity conditions, MLE estimates are consistent — they converge to the true parameter values as sample size grows — and asymptotically efficient, meaning they achieve the Cramér-Rao lower bound on variance in large samples. These guarantees make MLE a reliable default for parameter estimation across a wide range of models.

Despite its strengths, MLE has notable limitations. It can overfit when data is scarce, since it optimizes purely for the observed sample without any regularization. It also produces point estimates rather than distributions over parameters, offering no built-in measure of uncertainty. Bayesian inference addresses both concerns by incorporating prior beliefs and returning a full posterior distribution, but at the cost of greater computational complexity. In practice, MLE remains the dominant estimation strategy in machine learning due to its simplicity, scalability, and strong theoretical foundations.