Overfitting, underfitting, and regularisation

Welcome back. This is Lesson 8 in our machine learning series. Last time we talked about classification and how to choose a threshold. Today we are going to look at something more fundamental, something that affects every model you will ever build. Why do models sometimes look brilliant on paper and then fall apart in the real world? Or why do they sometimes seem too dumb to be useful at all? This is the story of overfitting and underfitting, and what to do about it.

Let me start with a picture you can hold in your mind while you walk. Imagine you are trying to draw a line through a scatter of points on a graph. The points represent something simple, like house prices against square footage. Most of the points follow a gentle upward trend, but there is noise. One small house sold for a surprisingly high price because it was next to a famous school. One large house sold cheaply because it needed a full renovation. The general pattern is clear, but the individual points wiggle around it.

Now, you ask a model to learn from these points. You give it two options. Option one is a straight line. It slopes upward, roughly capturing the trend, but it misses a lot of the points. It is too stiff, too rigid. It cannot bend to follow the curve of the data. This is underfitting. The model is too simple to capture the real pattern. It sees the big picture but gets the details wrong, and even the big picture is a bit off because real house prices do not rise in a perfectly straight line.

Option two is a very flexible curve. It can twist and turn to pass through every single point. It hits the expensive small house. It hits the cheap large house. It threads through every dot perfectly. On the training data, it looks amazing. Zero error. But here is the problem. That curve has learned the noise, not just the signal. It thinks the famous school bump and the renovation discount are part of the general pattern. When you show it a new house, one it has never seen, it makes wild predictions because it has memorised quirks that do not generalise. This is overfitting. The model is too complex. It has learned the training data by heart, including all the accidents and outliers, and it fails on new data.

So underfitting is when your model is too simple. It cannot capture the pattern. Overfitting is when your model is too complex. It captures the pattern plus all the noise. Both are bad, but they fail in different ways. An underfit model is consistently wrong. An overfit model is often very wrong in surprising ways because it has learned things that are not true.

Think of it like a student preparing for an exam. The underfit student read the textbook once, skimmed the summary, and thinks they know the shape of the subject. But when the exam asks a specific question, they cannot answer it properly because they never engaged with the detail. They have a vague sense but no real grasp.

The overfit student memorised every past paper, every worked example, every footnote. They can recite the answers to questions that have been asked before. But when the exam presents a new scenario, a slightly different framing, they are lost. They memorised the specific questions instead of understanding the underlying principles. They cannot generalise.

The student you want is somewhere in the middle. They understand the principles well enough to handle new questions they have never seen before. That is what a good model looks like. It learns the underlying pattern and ignores the noise.

Now, here is a rule of thumb that will serve you well. The more complex your model, the more data it needs and the more discipline you must apply. A simple straight line can learn from a small dataset because it only has to figure out a slope and an intercept. Two numbers. But a highly flexible model, the kind that can twist and turn, has hundreds or thousands or millions of parameters to learn. If you give it a small dataset, it will just memorise. It has the capacity to fit every point exactly, so why would it not? It does not know which details matter and which are noise. It needs enough data to see the pattern repeated many times before it starts to ignore the wiggles.

This is why people say you need more data for deep learning. It is not because deep learning is magical and hungry. It is because deep learning models are extremely flexible. They have enormous capacity. Without enough examples to average out the noise, they will memorise. A simpler model with less data can sometimes outperform a complex model because the simple model cannot overfit even if it wanted to. It physically cannot bend that far.

So how do you fight overfitting? There are several practical brakes you can apply, and I want to walk you through the most important ones.

The first is hold-out validation. This is simple and powerful. You do not let the model see all your data during training. You hold back a chunk, maybe twenty percent, and you hide it. The model trains on the other eighty percent. Then you test it on the hidden twenty percent. If the model has overfit, it will do well on the training data and poorly on the held-out data. The gap between training performance and validation performance is your signal. A small gap means the model is learning something real. A large gap means it is memorising. If you see the training error going down nicely but the validation error starts climbing back up, you know the model has crossed the line. It has stopped learning the pattern and started learning the noise.

This leads directly to the second brake, early stopping. Many models train iteratively. They make small adjustments over many rounds. At first, they pick up the big patterns. Later, they start fine-tuning to fit the quirks. If you watch the validation error during training, you will see it decrease for a while and then start to rise. The moment it starts rising, you stop. You do not let the model keep going. You catch it at the point where it has learned the most useful patterns but has not yet started memorising the noise. It is like pulling a cake out of the oven at the right moment. Leave it in too long and it burns.

The third brake is regularisation. This is the one that sounds technical but the idea is beautifully simple. Regularisation is a penalty for complexity. You tell the model, learn the pattern, but I am going to charge you for every twist and turn you add. Every time you make the curve more complicated, there is a cost. The model has to decide, is this extra wiggle worth the penalty? If the wiggle dramatically reduces error, fine, pay the cost. If the wiggle only helps a little, the penalty makes it not worth it. The model is forced to prefer simpler explanations unless the data strongly justifies something more complex.

Think of it like writing. You could describe a scene with twenty adjectives and five subordinate clauses, or you could describe it with three well-chosen words. Regularisation is the editor who says, cut it down. Keep only what earns its place. The result is clearer, more general, more robust. It communicates the idea without getting lost in unnecessary detail.

There are different flavours of regularisation. Some shrink all the model's parameters a little, pulling everything toward zero, toward simplicity. Others are more aggressive and will drop parameters entirely, removing features that are not pulling their weight. You do not need to remember the names right now. The concept is what matters. Regularisation is the principle of preferring simpler explanations, applied as a built-in cost during training.

Now let me put this all together with a concrete example. Suppose you are building a model to predict customer churn, whether a customer will cancel their subscription. You have data on ten thousand customers. You use a very flexible model with many parameters. You train it and check the training accuracy. It is ninety-eight percent. Amazing, you think. Then you test on your hold-out set and the accuracy is seventy-one percent. That twenty-seven point gap is screaming at you. The model has overfit. It learned which specific customers in your training set churned, including all the coincidental details, but it cannot generalise to new customers.

What do you do? You apply regularisation to penalise complexity. You try early stopping to catch the model before it memorises. You might also simplify the model itself, choosing something with less capacity. And you check whether you have enough data for the complexity you are attempting. Maybe ten thousand is not enough for this model. You retrain, and now training accuracy is eighty-two percent and validation accuracy is seventy-nine percent. The gap is small. The model is learning something real. It will perform more consistently in production.

On the other side, suppose your churn model gets sixty percent on training data and fifty-nine percent on validation. The gap is tiny, which is good, but both numbers are poor. This model is underfit. It is too simple. It cannot capture the patterns in your data. You need more capacity, more features, or a more flexible model. The small gap tells you the model is not overfitting, but the low scores tell you it is not learning enough.

So the gap between training and validation performance is your diagnostic tool. Large gap means overfitting. Small gap with poor performance means underfitting. Small gap with good performance means you are in the sweet spot.

Let me end with the symptoms you would notice in a failing model project, because these are the things you will actually see in practice, and recognising them early saves months of wasted effort.

Symptom one. The model looks fantastic in development but disappoints in production. Training metrics are high, maybe even suspiciously high, but real-world results are mediocre. This is the classic overfitting signature. The model learned the training data too well and cannot handle the messiness of live data.

Symptom two. You keep adding features and the model keeps getting better on training data but not on validation data. Each new feature helps the model memorise more but does not help it generalise. You are feeding the overfitting, not fixing it.

Symptom three. The model's predictions are unstable. Small changes in the input lead to large swings in the output. This often means the model has learned fragile patterns that depend on specific combinations of features that do not hold up in new data.

Symptom four. The model performs very differently on different slices of data. Great on one segment, terrible on another. This can mean it has overfit to the dominant pattern in the training data and ignored minority cases.

Symptom five, on the underfitting side. The model performs about the same no matter how much data you give it. Adding more data does not help because the model is too simple to use the extra information. It has hit its capacity ceiling.

When you see these symptoms, do not just throw more data at the problem or tune parameters randomly. Step back and ask, is this model too simple or too complex for the data I have? Am I holding out a proper validation set? Am I using regularisation? Should I be stopping training earlier? These are the questions that separate someone who understands machine learning from someone who just runs scripts.

As you finish your walk, try this. Think of a model or a prediction system you use at work. Ask yourself whether it might be overfitting or underfitting. Does it seem to know the training data perfectly but struggle with new situations? That is overfitting. Does it seem too crude, missing obvious patterns? That is underfitting. Just naming the problem correctly puts you halfway to fixing it. Next time, we will look at how to measure a model across all thresholds at once, so you can compare classifiers before picking a cutoff. For now, enjoy the rest of your walk.

Quick recap of today's lesson. Underfitting is when your model is too simple to capture the real pattern in your data. It gets the general direction but misses the detail, and even the general direction can be off. Overfitting is when your model is too complex. It memorises the training data, including all the noise and quirks, and then fails badly on new data it has never seen. The key diagnostic is the gap between training performance and validation performance. A large gap means overfitting. A small gap with poor performance means underfitting. A small gap with good performance means you are in the sweet spot. More complex models need more data and more discipline, because they have the capacity to memorise rather than generalise. Regularisation is a penalty for complexity. It forces the model to prefer simpler explanations unless the data strongly justifies something more complex. Early stopping means you watch the validation error during training and stop the moment it starts rising, catching the model before it begins memorising noise. Hold-out validation means you hide a chunk of data from the model during training and test on it afterwards to see if the model has learned something real or just memorised. The symptoms of a failing project include great training metrics but poor production results, unstable predictions, and performance that varies wildly across different segments of data. When you see these signs, ask whether your model is too simple or too complex for the data you have, and apply the right brake.