Variance in machine learning refers to how sensitive a model is to changes in the training data. It measures how much the modelβs predictions would change if we trained it on a different dataset.
In simple terms:
π High variance = model changes a lot with small changes in data
π Low variance = model stays more stable and consistent
1. What Does Variance Mean in Model Performance?
Variance shows how well a model generalizes across different datasets.
- High variance model: learns training data too closely (including noise)
- Low variance model: ignores important patterns and becomes too simple
π Good models aim for a balance between bias and variance
2. Variance and Overfitting
Variance is directly related to overfitting.
High Variance β Overfitting
When a model has high variance:
- It memorizes training data instead of learning general patterns
- It performs extremely well on training data
- It performs poorly on new/unseen data
π This is called overfitting
3. Why High Variance Models Overfit
High variance models:
- Learn noise in training data
- Become too complex
- Capture small fluctuations instead of real patterns
So:
- Training accuracy β very high
- Test accuracy β low
π The model fails to generalize
4. Visual Understanding (Statistical Meaning)
Variance is mathematically based on how much predictions spread around the mean.
ξgenuiξ{"probability_statistics_learning_block":{"type_id":"VARIANCE"}}ξ
π This shows that variance measures how much values deviate from the average.
5. Simple Example
Imagine a student learning for exams:
- High variance student β memorizes exact questions from practice paper
- If questions change β performance drops badly
- Low variance student β understands concepts β performs well in any exam
π Machine learning works the same way
6. High vs Low Variance in ML
- High variance model = complex model (like deep decision trees, overfitted neural networks)
- Low variance model = simple model (like linear regression)
π High variance = unstable predictions
π Low variance = stable but sometimes too simple
7. How to Reduce High Variance
To reduce variance (and avoid overfitting), we use:
- More training data
- Regularization (L1, L2)
- Simplifying the model
- Cross-validation
- Pruning in decision trees
- Dropout in neural networks
Conclusion
Variance in machine learning measures how much a modelβs predictions change when trained on different datasets. High variance means the model is too sensitive to training data and captures noise instead of real patterns, leading to overfitting. Such models perform well on training data but poorly on unseen data. Therefore, controlling variance is essential to ensure good generalization, and techniques like regularization, cross-validation, and model simplification are used to reduce it and improve model reliability.