why regularisation improve overfitting

Regularisation is a technique used to prevent overfitting in machine learning models. Overfitting occurs when a model learns the training data too well, including its noise and outliers, which leads to poor performance on unseen data.

Here's how regularisation helps to improve overfitting, step by step:

1. Penalising Complexity: Regularisation works by adding a penalty term to the loss function, which increases as the complexity of the model increases. This discourages the learning algorithm from assigning too much importance to any individual feature, thus reducing the risk of overfitting.

2. Shrinking Coefficients: In linear models like linear regression or logistic regression, regularisation techniques (like L1 and L2 regularisation) shrink the coefficients of less important features towards zero. This effectively reduces the number of features, which simplifies the model and helps to prevent overfitting.

3. Improving Generalisation: By discouraging complexity, regularisation helps to ensure that the model generalises well from the training data to unseen data. This is because a simpler model is less likely to be influenced by noise and outliers in the training data, and is therefore more likely to make accurate predictions on new data.

4. Preventing Multicollinearity: Regularisation can also help to prevent multicollinearity (a situation where two or more features are highly correlated) by shrinking the coefficients of correlated features. This can help to improve the stability and performance of the model.

In summary, regularisation improves overfitting by adding a penalty term to the loss function that discourages complexity, shrinks the coefficients of less important features, improves the model's generalisation ability, and prevents multicollinearity.