Grid Search is a hyperparameter optimization technique used in machine learning to find suitable values for a model's hyperparameters. Instead of choosing these values manually, Grid Search tests multiple predefined combinations and evaluates how well the model performs with each combination.
Hyperparameters are settings that are selected before the model is trained. Examples include learning rate, tree depth, number of estimators, and regularization strength.
From a practical perspective, Grid Search provides a systematic way to compare different settings and identify a combination that performs well on validation data.
What Are Hyperparameters?
Hyperparameters are configuration values that control how a machine learning algorithm learns.
For example, a decision tree may have hyperparameters such as:
- Maximum tree depth
- Minimum samples required for a split
- Minimum samples per leaf
Similarly, a machine learning algorithm may have settings such as:
- Learning rate
- Number of iterations
- Regularization strength
- Number of estimators
These values are different from model parameters, which are learned from the training data.
How Grid Search Works
Grid Search begins with a list of possible values for each hyperparameter.
For example, suppose a model has two hyperparameters:
Learning Rate: 0.01, 0.1, 0.2
Maximum Depth: 3, 5, 7