Skip to content

Deep Learning Underfitting and Overfitting: Technical Guide

Table of Contents

1. Concepts and Definitions

Underfitting
- Occurs when a model is too simple to capture the underlying patterns in the data. - High bias, low variance: Poor performance on both training and test data. - Example: Linear model on a highly non-linear dataset[1][3][4][6].

Overfitting
- Occurs when a model is too complex and learns not only the underlying patterns but also the noise in the training data. - Low bias, high variance: Excellent performance on training data but poor generalization to unseen data[1][2][3][4][6]. - Example: Deep neural network with too many parameters trained on a small dataset.

Generalization
- The ability of a model to perform well on new, unseen data. - The goal is to find the right balance between underfitting and overfitting[4][5].

2. Applications & Use Cases

  • Computer Vision: Overfitting is common with small labeled datasets; underfitting can occur with shallow architectures.
  • NLP: Large transformer models can overfit on small corpora; simple models may underfit complex language tasks.
  • Time Series: Overfitting can arise with highly seasonal or noisy data; underfitting with overly smoothed models.
  • Healthcare: Overfitting is a risk when data is limited or imbalanced, e.g., rare disease detection.

3. Challenges

  • Data Limitations: Not enough data increases overfitting risk; noisy or irrelevant features exacerbate both issues[1][3].
  • Model Complexity: Too many parameters lead to overfitting; too few cause underfitting[2][3][5].
  • Feature Engineering: Poor feature selection/engineering can render even good models ineffective[3].
  • Bias-Variance Tradeoff: Balancing model bias (simplicity) and variance (complexity) is non-trivial[3][6].

4. Detection

  • Loss/Generalization Curves:
  • Overfitting: Training loss decreases, validation loss increases after a point[4].
  • Underfitting: Both training and validation losses remain high[4][5].
  • Performance Metrics:
  • Overfit: High train accuracy, low test accuracy.
  • Underfit: Low train and test accuracy.

5. Solutions & Best Practices

To Mitigate Overfitting:

  • Get More Data: Increases generalization[1][5].
  • Reduce Model Complexity: Fewer layers/parameters[5].
  • Regularization: L1/L2 penalties, Dropout layers[1][3][5].
  • Early Stopping: Stop training when validation loss starts increasing[4].
  • Cross-Validation: Ensures model generalizes across different data splits[1][3].
  • Data Augmentation: Especially in vision, to synthetically enlarge dataset[5].
  • Ensemble Methods: Bagging, boosting, stacking to average out errors[1][3].

To Mitigate Underfitting:

  • Increase Model Complexity: Add layers/units or use more expressive architectures[3][5].
  • Reduce Regularization: Lower penalty terms or dropout rates[3][5].
  • Improve Feature Engineering: Add relevant features or use embeddings[3].
  • Train Longer: Allow more epochs if not yet converged[2][5].
  • Decrease Bias: Use more sophisticated models or algorithms[6].

6. Code Example: Overfitting and Underfitting in Keras

import tensorflow as tf
from tensorflow.keras import layers, models

# Generate synthetic data
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Underfitting: Too simple model
underfit_model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),
    layers.Dense(16, activation='relu'),
    layers.Dense(10, activation='softmax')
])
underfit_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
underfit_model.fit(x_train, y_train, epochs=3, validation_split=0.2)

# Overfitting: Too complex model, no regularization
overfit_model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),
    layers.Dense(1024, activation='relu'),
    layers.Dense(1024, activation='relu'),
    layers.Dense(10, activation='softmax')
])
overfit_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
overfit_model.fit(x_train, y_train, epochs=20, validation_split=0.2)

# Good fit: Proper capacity + regularization
good_model = models.Sequential([
    layers.Flatten(input_shape=(28, 28)),
    layers.Dense(256, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])
good_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
good_model.fit(x_train, y_train, epochs=20, validation_split=0.2, callbacks=[
    tf.keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)
])

7. Example: Real-World Overfitting and Underfitting

  • Overfitting:
    In medical imaging, a deep CNN trained on a small set of X-rays may achieve 99% train accuracy but only 70% test accuracy. Adding dropout, augmenting data, and using early stopping improves generalization[1][5].
  • Underfitting:
    A linear regression model for predicting housing prices on a complex dataset yields low accuracy on both train and test sets. Switching to a deep MLP or adding polynomial features can help[3][5].

8. Summary Table: Techniques and Their Effects

Problem Cause Solution/Technique Example
Overfitting Too complex, not enough data Regularization, dropout, more data Deep CNN on small dataset
Underfitting Too simple, too much regularization Add layers, reduce regularization Linear model on complex data
Both Poor feature engineering Feature selection/engineering Irrelevant features present

9. Best Practices

  • Always start simple, then increase complexity as needed[5].
  • Use validation curves to monitor for over/underfitting[4][5].
  • Combine multiple mitigation strategies for best results[5].
  • Regularly retrain and monitor models in production for drift[1][3][5].

In summary:
Underfitting and overfitting are two sides of the generalization challenge in deep learning. The key is balancing model complexity and data quality, using regularization, data augmentation, and careful monitoring to achieve robust, generalizable models[1][3][4][5][6].

Citations:

  • [1] https://www.simplilearn.com/tutorials/machine-learning-tutorial/overfitting-and-underfitting
  • [2] https://aiixx.ai/blog/epochs-overfitting-and-underfitting-a-beginners-guide
  • [3] https://www.pecan.ai/blog/machine-learning-model-underfitting-and-overfitting/
  • [4] https://developers.google.com/machine-learning/crash-course/overfitting/overfitting
  • [5] https://www.tensorflow.org/tutorials/keras/overfit_and_underfit
  • [6] https://curiousily.com/posts/hackers-guide-to-fixing-underfitting-and-overfitting-models/

Source (Perplexity)