Machine learning involves using an algorithm to learn and generalize from historical data in order to make predictions on new data. We can describe this problem as approximating a function that maps input examples to output examples. To solve this, we frame the problem as function optimization. This is what we actually mean when we refer to machine learning optimization.
Definition
Any problem that deals with maximizing or minimizing any physical quantity. Which can be mathematically computed/expressed in the form of some variables are defined as an Optimization Problem. Some of the common examples of optimisation techniques are: Feature Scaling, Batch Normalization, Mini-batch Gradient Descent, Gradient Descent with Momentum.
Batch Normalization
Batch Normalization is used to accelerate the training of deep neural networks by normalizing the input of each layer. It involves normalizing the activations of each layer across mini batches. Typically by subtracting the batch mean and dividing by the batch standard deviation.
Gradient Descent with Momentum
Gradient Descent with Momentum is an extension of the standard Gradient Descent algorithm. It accelerates convergence by incorporating past gradients into the update rule. It introduces a momentum term that dampens oscillations and speeds up convergence. Especially in the presence of high curvature or noisy gradients.
Learning Rate Decay
Learning Rate Decay is a technique used to gradually reduce the learning rate during training to facilitate convergence towards the end of the optimization process. It prevents overshooting the minimum and enables fine-tuning of model parameters in the later stages of training.
Mini-batch Gradient Descent
Mini-batch Gradient Descent is a variant of the Gradient Descent algorithm where the gradient is computed using a subset (mini-batch) of the training data in each iteration. It strikes a balance between the efficiency of batch gradient descent and the noisy updates of stochastic gradient descent.
Feature Scaling
Feature scaling is a preprocessing technique used to standardize the range of independent variables or features in the dataset. It ensures that all features have the same scale, preventing certain features from dominating the learning process due to their larger magnitudes. The two common methods for feature scaling are Min-Max Scaling and Standardization.
RMSProp Optimization
RMSProp (Root Mean Square Propagation) is an adaptive learning rate optimization algorithm. It adjusts the learning rate for each parameter based on the magnitude of its gradients. It divides the learning rate by an exponentially decaying average of squared gradients. Hence, effectively reducing the learning rate for frequently occurring parameters and increasing it for infrequent ones.
Adam Optimization
Adam (Adaptive Moment Estimation) is an adaptive learning rate optimization algorithm that combines the advantages of RMSProp and Momentum methods. It maintains two moving averages of gradients (first moment and second moment) and adapts the learning rate for each parameter accordingly. In training deep neural networks for image classification tasks, Adam optimization offers efficient convergence and robustness to variations in the dataset by adapting learning rates based on both first and second moments of gradients.
By leveraging machine learning optimization judiciously and adapting them to the specific characteristics of the dataset and model architecture, practitioners can achieve faster convergence.
