What is a Batch Normalisation Layer ?

Batch Normalization (BN) is a technique commonly used in deep neural networks to improve the training stability and convergence speed. It was introduced by Sergey Ioffe and Christian Szegedy in their paper titled "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift."

In deep neural networks, during training, the distribution of the input to each layer may change, a phenomenon known as "covariate shift." This can slow down the training process and make it more challenging for the network to converge. Batch Normalization addresses this issue by normalizing the inputs to a layer across a mini-batch of data.

The mathematical formulas for Batch Normalization are as follows, where x is the input, μ is the mean, σ is the standard deviation, γ is the scaling parameter, and β is the shifting parameter:

Here's a breakdown of how Batch Normalization works:

1. Normalization: For each mini-batch, Batch Normalization calculates the mean and variance of the input features. It then normalizes the input by subtracting the mean and dividing by the square root of the variance. This process ensures that the inputs to the network have a mean of 0 and a variance of 1, which can help in faster convergence and better performance.

2. Scaling and Shifting: After normalization, Batch Normalization applies a scale (gamma) and a shift (beta) to the normalized inputs. These parameters are learned during the training process, allowing the network to adjust the scale and shift based on the data it is learning from. This step is crucial for maintaining the network's flexibility and performance.

3. Regularization Effect: Batch Normalization also acts as a form of regularization. By normalizing the inputs, it can help in reducing overfitting by making the network less sensitive to the scale of the input features. This can lead to better generalization performance on unseen data.

4. Acceleration: By normalizing the inputs, Batch Normalization can help in accelerating the training process. It can reduce the number of training epochs required to train a deep neural network, as it helps in faster convergence.

5. Improved Performance: Batch Normalization can lead to improved performance of the network. It can help in stabilizing the learning process, making the network more robust to the initialization of the weights, and reducing the sensitivity to the initial learning rate.

Batch Normalization is widely used in deep learning frameworks like TensorFlow and PyTorch, and it's a key component in many state-of-the-art models. It's particularly effective in training deep neural networks, where it can significantly improve the speed and performance of the training process.

Batch Normalization is typically applied to the activations of intermediate layers within a neural network. It has several benefits, including:

• Improved training stability: Batch Normalization reduces internal covariate shift, making it easier for the network to learn and converge.

• Reduced sensitivity to weight initialization: BN allows for the use of higher learning rates and is less sensitive to the initial values of the network's weights.

• Regularization effect: Batch Normalization introduces a slight regularization effect, reducing the need for other regularization techniques like dropout in some cases.

Batch Normalization layers are often inserted after the linear transformation and before the activation function in a neural network layer. They have become a standard component in the architecture of many deep learning models.