Statistical Machine Learning, Week 12: Hyperparameter Optimization, KerasTuner, Hyperband Example

#neuralnetworks #deeplearning #tensorflow

What are hyperparameters?

Variables that control the training process and structure of a machine learning model.
Set before training begins and remain constant during training.
Have a significant impact on the model's performance.

Types of hyperparameters:

Model hyperparameters: Affect the architecture of the model.
Examples: Number of hidden layers in a neural network, number of neurons per layer.
Algorithm hyperparameters: Influence the learning process itself.
Examples: Learning rate in gradient descent, regularization strength.


Common techniques for hyperparameter optimization:

Grid Search: You define a grid of possible values for each hyperparameter, and the algorithm evaluates every single combination.
Simple to implement and guarantees finding the best combination within the specified grid. However, it is computationally expensive.
Random Search: Instead of exhaustively trying every combination, random search samples a predefined number of configurations from the hyperparameter space.
Often more efficient than Grid Search, especially when some hyperparameters are more important than others. However, it does not guarantee finding the absolute best combination.
Bayesian Optimization: This technique uses a probabilistic model (often a Gaussian Process) to learn the relationship between hyperparameters and model performance.
Can be more complex to implement. The performance depends on the choice of the probabilistic model and acquisition function.
Hyperband: This is an early stopping technique designed for hyperparameter optimization. It focuses on quickly identifying and stopping poorly performing configurations, allowing more resources to be allocated to promising ones.


Although hyperparameter optimization is rapidly advancing, comprehensive coverage of this topic in textbooks remains limited. However, a wealth of research articles provides valuable insights into this crucial aspect of machine learning.
For a thorough overview of the techniques mentioned earlier, the following paper offers a particularly comprehensive and up-to-date analysis.
Bischl, Bernd, et al. "Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges." Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 13.2 (2023): e1484.
Link: https://wires.onlinelibrary.wiley.com... (Open Access)

Keras, and specifically the Keras Tuner library, provides tools for implementing hyperparameter optimization techniques. We will explore some of these techniques in this lecture.

Install and import the Keras Tuner:
```pip install -q -U keras-tuner```

```import keras_tuner as kt```