Скачать или смотреть Different Variants of Gradient Descent (Visualization)

Gradient-based optimization is the core technique used to train most machine learning models, particularly neural networks, as it iteratively updates model parameters in the direction that minimizes a defined loss function.

💡 The idea is simple: at each iteration of an epoch, we update the model parameters by taking a step toward the minimum. The learning rate (η) controls the size of this step, while the negative gradient (-∇f) determines its direction.

🎨What is being visualized here?
⚖️The visualization compares three types of gradient descent, each differing in the "batch size" (the amount of training data used per iteration). The position of the ball 🟠 shows the current value of loss visually (the function’s minimum).

🧪To simulate the training process of a model, 50 Rastrigin-based surfaces with random noise are used as the loss surfaces of the training data. This corresponds to a total of 50 training samples. The surface is represented by evaluating the Rastrigin function over a range of parameter values. This allows us to observe the global minima (lowest point).

🤔Why does batch size matter?
Using more training data per update makes the loss surface smoother. Local minima get averaged out, which makes it harder for the optimizer to escape them. 🚀Furthermore, from the performance perspective, using mini-batches helps reduce memory requirements and takes advantage of the parallelization capabilities of GPUs

ℹ️For example, if we use all 50 training samples in a single batch, the gradient is averaged over all of them. This creates a stable surface where the optimizer is more likely to get stuck in local minima. By contrast, with a batch size of 1, the loss surface shifts slightly at each iteration. These fluctuations cause the gradient to vary more, helping the optimizer “wiggle” its way out of local minima.

🤔 How does batch size affect iterations?
An epoch is one complete pass of the entire training dataset through the model. The iteration is the number of times a batch of data is passed through the model during training.

The relationship between batch size and iterations is inversely proportional; therefore, in each epoch, the ball’s 🟠 position is updated fewer times with a larger batch size than with a smaller batch size.

🤔Why the Rastrigin function?
The Rastrigin function is often used to simulate the highly non-convex surface of a neural network’s loss landscape. While a neural network has far more than two parameters (making direct visualization impossible), the Rastrigin function provides a useful 3D analogy.

🤔Why does the graphs wiggle more with smaller batch sizes?
As discussed earlier, smaller batches prevent averaging. During each epoch, the optimizer effectively switches between individual samples, which produces sharp jumps between loss surfaces. This leads to the characteristic “wiggling” trajectories observed in the visualization.

#ai #science #artificialintelligence #machinelearning #education #python #programming