Grid Search Hyperparameter Tuning in Python

Hyperparameter tuning is essential for improving machine learning model performance. Grid Search is one of the most effective techniques for systematically finding the best hyperparameters. I’ve spent 20+ years empowering businesses, especially startups, to achieve extraordinary results through strategic technology adoption and transformative leadership. This guide explains Grid Search with an example using GridSearchCV in Python to optimize a Random Forest Classifier.

For more detail hyperparamter information: Hyperparameter Tuning for Optimal Model Performance: Finding the Perfect Balance for Machine Learning Models >>

What is Grid Search?

Grid Search exhaustively evaluates all possible combinations of hyperparameters within a predefined range. It guarantees the best combination within the specified grid but can be computationally expensive. It is best suited for small-scale models or when computational resources are not a constraint.

Why Use Grid Search?

Guaranteed best combination: Finds the optimal hyperparameters from the given search space.
Systematic approach: Evaluates each combination without random selection.
Best for small models: Works well when computational power is available.

However, for large datasets or deep learning models, Grid Search can be slow and expensive. In such cases, Random Search or Bayesian Optimization are better alternatives.

Example: Grid Search for Hyperparameter Tuning in Random Forest

Step 1: Import Libraries & Load Dataset

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_iris

# Load Iris dataset
data = load_iris()
X, y = data.data, data.target

# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Step 2: Define the Parameter Grid

# Define hyperparameter grid
param_grid = {
    'n_estimators': [50, 100, 200],  # Number of trees
    'max_depth': [5, 10, 15],        # Maximum depth of trees
    'criterion': ['gini', 'entropy'] # Splitting criterion
}

Step 3: Perform Grid Search

# Initialize the model
rf = RandomForestClassifier(random_state=42)

# Perform Grid Search with cross-validation
grid_search = GridSearchCV(estimator=rf, param_grid=param_grid,
                           cv=5, scoring='accuracy', n_jobs=-1, verbose=1)

# Fit the model
grid_search.fit(X_train, y_train)

Step 4: Get the Best Parameters & Evaluate Performance

# Get the best combination of hyperparameters
best_params = grid_search.best_params_
print("Best Hyperparameters:", best_params)

# Train the model with best hyperparameters
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)

# Evaluate model performance
accuracy = accuracy_score(y_test, y_pred)
print(f"Best Model Accuracy: {accuracy:.4f}")

Limitations of Grid Search

Computationally expensive: Testing all combinations can take a long time.
Not scalable: Inefficient for deep learning or large datasets.
Fixed grid: The predefined search space may miss better values outside the grid.

Better Alternatives to Grid Search

If Grid Search is too slow, consider:

Random Search: Randomly selects hyperparameters, reducing search time.
Bayesian Optimization: Uses probability models to efficiently find the best values.
Hyperband & Optuna: Modern, adaptive techniques for hyperparameter tuning.

My Tech Advice: Grid Search is a powerful but resource-intensive hyperparameter tuning method. It systematically finds the best model configuration within a defined space. The program slows down as it tests all possible combinations within the defined boundaries. If computational cost is a concern, Random Search or Bayesian Optimisation can be more efficient alternatives.
#AskDushyant

Note: The example and pseudo code is for illustration only. You must modify and experiment with the concept to meet your specific needs.

#TechConcept #TechAdvice #AI #ML #ModelTuning  #GridSearch