What does Hyperparameter Tuning mean?

Hyperparameter tuning is the search for the settings a model is trained with. Hyperparameters are values fixed before training, such as the learning rate, the number of layers, or the batch size. Unlike model parameters, training does not change them; instead, they steer the training.

Grid search tests every combination in a predefined grid and quickly becomes too expensive once more than three settings are involved. Random search instead draws samples from the search space and, given the same compute budget, usually finds better values. Bayesian optimisation uses previous results to choose the next trial deliberately. Every trial is evaluated through cross-validation, meaning several different splits of the data into training and test portions.

Hyperparameter tuning pays off wherever the amount of data is fixed and quality can only still be improved through the settings. Examples include forecasting models for demand and utilisation, classifiers for text categories, and image recognition with small training sets. For very large models, compute cost limits how many trials are feasible.

Its advantage over a library’s default settings is that the values fit the organisation’s own data. This gain comes without collecting additional data. The series of trials also shows which setting a model is sensitive to and which has barely any effect.

Settings must not be chosen using the same data that later measures quality, or the result will look better than it is. A separate test set and documentation of the chosen values together with the model version are therefore essential.