Hyperparameter

Hyperparameters are configuration values set before training begins that control the structure and behavior of a machine learning model. They differ from parameters, which are learned directly from data. Common examples include the learning rate in gradient descent, the number of layers and neurons in a neural network, the depth of a decision tree, regularization coefficients, batch size, and dropout rates. Since hyperparameters are not updated during training, they must be chosen deliberately. Their values can substantially affect whether a model converges, overfits, underfits, or generalizes well to new data.

The process of finding good hyperparameter values is called hyperparameter optimization or tuning. The simplest approach, grid search, exhaustively evaluates all combinations of candidate values across a predefined grid. Random search samples combinations stochastically and often finds competitive results more efficiently. More sophisticated methods include Bayesian optimization, which builds a probabilistic surrogate model of the objective function to select the next configuration to evaluate, and gradient-based methods that treat hyperparameter selection as a differentiable problem. Automated machine learning (AutoML) frameworks increasingly handle this process end-to-end.

Hyperparameters matter because no single configuration works universally across datasets and tasks. A learning rate that trains one network efficiently may cause another to diverge or stall. Regularization strength must be calibrated to the complexity of the data and model to avoid overfitting or underfitting. As models have grown in scale and complexity, particularly with deep neural networks, the hyperparameter search space has expanded, making principled tuning strategies essential.

The concept has been present in machine learning since early work on neural networks and statistical learning in the 1980s. Systematic hyperparameter optimization became a recognized research area in its own right as deep learning scaled up in the 2000s and 2010s. Today, hyperparameter tuning is a standard step in model development pipelines, and tools like Optuna, Ray Tune, and Keras Tuner have made automated optimization accessible to practitioners across the field.