Loss, error, and what 'learning' actually optimises

Welcome back. This is lesson five in the machine learning series. Last time we talked about honest evaluation, about splitting your data into training, validation, and test piles so you can trust your numbers. Today we are going one layer deeper. We are going to look at what actually happens when a model learns. Because learning, in machine learning, is not magic. It is not the model suddenly understanding the world. It is something very specific and very mechanical. The model is trying to reduce a number. That number is called the loss. And the whole point of this walk is for you to understand what that number is, where it comes from, and why getting it wrong will give you a model that behaves badly even if it looks like it is performing well.

Let us start with the simplest idea. A prediction and the truth. Imagine you are predicting house prices. You have a house that actually sold for three hundred thousand pounds. Your model predicts two hundred and seventy thousand. The gap between the prediction and the truth is thirty thousand. That gap is an error. One prediction, one error. Simple enough.

But your dataset is not one house. It might be ten thousand houses. For each one, the model makes a prediction, and for each one there is a gap between that prediction and the actual sale price. So you have ten thousand errors. Some are big, some are small, some are positive, some are negative. You cannot look at ten thousand individual errors and decide whether the model is good. You need one number. One score that summarises all of those errors into something you can compare, something you can try to improve.

That score is the loss. The loss function takes all of your errors and combines them into a single number. A lower loss means the model is, on the whole, less wrong. A higher loss means it is more wrong. When the model is learning, it is not trying to be smart. It is not trying to understand the housing market. It is trying to make that one number go down.

Now, how exactly do you turn many errors into one number? There are different ways, and the choice matters enormously. Let us start with the most common approach for predicting a number, like a price or a temperature. You take each error, you square it, and then you average all those squared values. Squaring does two useful things. First, it makes every error positive, because a negative number squared becomes positive. That matters because if you just added up the raw errors, a prediction that is thirty thousand too high and a prediction that is thirty thousand too low would cancel each other out. You would think the model is perfect when it is actually consistently wrong in both directions. Second, squaring penalises large errors more than small ones. An error of ten becomes one hundred. An error of twenty becomes four hundred. So being very wrong on one house hurts the score much more than being slightly wrong on many houses. This approach is called mean squared error, and it is the workhorse of numerical prediction. You will see it everywhere.

But here is the thing. Mean squared error is great when you are predicting a number. It is the wrong tool when you are predicting a category. Imagine you are building a model that looks at an email and predicts whether it is spam or not spam. The truth is a label, spam or not spam. The prediction is also a label. If the model says spam and the truth is spam, that is correct, no error. If the model says spam and the truth is not spam, that is an error. But what is the size of that error? It is not thirty thousand pounds. It is just wrong. A binary outcome does not have a natural distance from the truth the way a price does.

So for classification, we need a different kind of loss. The most common one works with probabilities. Instead of the model saying a hard yes or no, it says something like, there is a ninety percent chance this email is spam. The loss function then asks, how confident was the model in the correct answer? If the truth is spam and the model said ninety percent chance of spam, the loss is small. If the truth is spam and the model said ten percent chance of spam, the loss is large. The key intuition is this. The loss is not just about being right or wrong. It is about how confident the model was in the right direction. A model that is barely right on everything is different from a model that is confidently right on most things and confidently wrong on a few. The loss function captures that distinction.

So now you have a loss number. The model has many parameters, many internal knobs that it can turn. In a simple model, these might be weights that multiply each input feature. In a more complex model, there could be millions of them. Learning means adjusting all of those knobs slightly, in a direction that makes the loss go down. Think of it like this. You are standing on a hillside in the dark. You cannot see the landscape. But you can feel the slope under your feet. You take a step downhill. Then you feel the slope again. You take another step downhill. You keep doing this, and gradually you descend. That is what training is. The model computes the loss, figures out which way to nudge its parameters to reduce that loss, takes a step, and repeats. Each step is called a training step. Over hundreds or thousands of steps, the loss creeps downward and the model gets less wrong.

Now, that hillside analogy has a catch. The landscape is not a single smooth bowl. It is bumpy. There are dips and valleys and flat patches. Sometimes the model steps downhill and reaches a small valley that is not the lowest point on the whole landscape. It cannot go further down without first going up, and the training process only goes down. So it gets stuck. That is called a local minimum. It is locally the best, but globally there might be a much deeper valley somewhere else that the model never finds. There are techniques to deal with this, like taking bigger steps early on or adding a bit of randomness to the direction, and we will touch on those in a later lesson. For now, the key intuition is that the training process is not guaranteed to find the best possible model. It finds a good enough model given where it started and how it walked.

There is also the opposite problem. Sometimes the landscape is not a valley at all. It is a vast flat plain. The slope is nearly zero in every direction. The model takes steps but the loss barely changes. This is called a plateau. The model is not stuck in a bad spot exactly, but it is making no progress. Training can grind to a halt not because the model is bad but because the landscape is unhelpful. Again, there are practical tricks for this, but the point for you right now is to know that these things happen, and that a model that stops improving is not necessarily broken. It might just be sitting on a plateau.

Now I want to talk about the most important idea in this lesson. Possibly the most important idea in all of practical machine learning. You get what you optimise. The loss function is not just a measurement. It is the definition of what the model is trying to achieve. If you choose the wrong loss, the model will dutifully optimise it, and you will get behaviour that is technically correct but practically useless or even harmful.

Here is a concrete example. Suppose you are building a model to predict whether a patient has a serious disease. You have a hundred patients. Ninety five are healthy. Five are sick. If you use a simple loss that just counts how many predictions are right, the model can achieve ninety five percent accuracy by predicting that everyone is healthy. It never predicts a single sick patient. It is ninety five percent right. That sounds impressive until you realise it is completely useless for its actual purpose, which is to catch the sick patients. The loss function rewarded the model for being right about the easy majority and ignored the fact that it failed catastrophically on the cases that mattered most. The fix is to use a loss that penalises missing a sick patient much more than it penalises a false alarm. You weight the errors differently. You tell the model, through the loss function, what matters to you. The model does not know what matters. Only the loss function tells it.

Another example. You are predicting delivery times for a logistics company. If you use mean squared error, the model will be penalised heavily for large errors. It will try hard to avoid being very wrong on any single delivery. But what if your business cares more about being consistently a little bit early than occasionally very late? Mean squared error does not know that. It treats being early and being late symmetrically, and it punishes big errors regardless of direction. If your business cost of being late is different from your business cost of being early, your loss function should reflect that. If it does not, you will optimise the wrong thing and the model will give you answers that are mathematically good but commercially wrong.

This is why choosing the loss function is one of the most consequential decisions you will make. It is more important than choosing the model architecture. A simple model with the right loss will outperform a sophisticated model with the wrong loss, because the sophisticated model will just be very good at optimising the wrong target. When you start a project, before you write any code, ask yourself this question. What does it mean for a prediction to be wrong, and how much does each kind of wrongness cost? The answer to that question should drive your choice of loss function.

So let me bring this together. When someone says a machine learning model is learning, what is actually happening is this. The model makes predictions on the training data. Each prediction is compared to the truth, producing an error. The loss function turns all those errors into a single number. An optimiser adjusts the model's parameters slightly to make that number go down. Then the process repeats. Step by step, the model gets less wrong according to the loss function you chose. Learning is not understanding. Learning is optimisation. The model is not getting smarter. It is getting less wrong according to a specific definition of wrong that you gave it.

And that definition might not match what you actually care about. That gap, between the thing you optimise and the thing you want, is where most real world machine learning projects quietly fail. Not in the code, not in the model, but in the mismatch between the loss and the goal.

As you keep walking, let me give you a mental picture to carry with you. Think of machine learning as a machine with four moving parts. The first part is the data, the examples the model learns from. The second part is the model, the system with adjustable parameters that turns inputs into predictions. The third part is the loss, the score of how wrong the model is, the thing being optimised. The fourth part is the optimiser, the mechanism that nudges the parameters to reduce the loss. Data, model, loss, optimiser. Those four parts are always there, in every machine learning system, no matter how simple or complex. Change any one of them and you change the behaviour of the whole system. Understanding each part, and how they interact, is the foundation of everything else in this field.

Try this as you walk. Think of a prediction problem from your own work. What would the truth look like? What would the errors look like? And crucially, what kind of being wrong would cost you the most? If you can answer that last question clearly, you already know more about choosing a loss function than most people who have just finished a machine learning course. The loss function is where your domain knowledge enters the system. The model does not know your business. The loss function is how you teach it.

Quick recap of today's lesson. A prediction compared to the truth gives you an error, and the loss function turns many errors into one number that the model tries to reduce. For predicting numbers, mean squared error is common. It squares each error so they are all positive and large errors are penalised more heavily. For classification, you need a different approach, one that works with how confident the model was in the correct label.

Training works by nudging the model's parameters slightly, step by step, in whatever direction reduces the loss. Think of walking downhill in the dark. But the landscape is bumpy. The model can get stuck in a local minimum, a small valley that is not the deepest point, or it can stall on a plateau where the slope is nearly flat.

The most important idea is this. You get what you optimise. If you choose the wrong loss function, the model will dutifully optimise it and give you behaviour that is technically correct but practically wrong. A model that predicts everyone is healthy can be ninety five percent accurate and completely useless if your goal is to catch the rare sick patients. The loss function is where your domain knowledge enters the system.

Remember the four moving parts. Data, model, loss, optimiser. Every machine learning system has all four. Change any one and you change the whole system. The loss function is not just a measurement. It is the definition of what learning means.