Regression — predicting numbers

Welcome back. This is Lesson 6 in the machine learning series, and today we are talking about regression, which is the business of predicting a number. Last time you learned about loss functions and optimisation. Now we get to see those ideas in action on the simplest and most useful kind of prediction problem there is.

Let me start with a question. What kind of thing are you predicting? If the answer is a category, like spam or not spam, or churn or stay, that is classification, and we will get to that in a later walk. But if the answer is a number on a continuous scale, like a price, a quantity, a duration, a temperature, a score, that is regression. The word regression sounds technical, but it just means predicting a number. How much will this house sell for? How many units will we sell next week? How many minutes late will this flight arrive? All regression problems.

Now let me build your intuition for the most classic regression model, linear regression. Imagine you are predicting the price of a house. You have a few facts about each house. The floor area in square metres. The number of bedrooms. The distance to the nearest train station in kilometres. Linear regression says, take each of those facts, multiply it by a weight, add them all up, and then add one more number on top called the intercept or the bias. That final sum is your predicted price.

Think of the weights as how much each fact matters and in which direction. A bigger area might have a positive weight, meaning more area pushes the price up. A longer distance to the station might have a negative weight, meaning being further away pushes the price down. The intercept is the starting point, the baseline price before any of the features contribute. If every feature were zero, the prediction would just be the intercept. Of course no house has zero area and zero bedrooms, but the intercept is still part of the maths that makes the line fit the data.

That is really all linear regression is. A weighted sum of your inputs plus a constant. The model learns by adjusting those weights and that intercept to minimise the loss, exactly the process we talked about last time. The loss function for classic linear regression is usually squared error, which means you take each prediction, subtract the true value, square the difference, and average across all your examples. The optimiser nudges the weights to make that average squared error as small as possible.

Now, here is something that sounds obvious but that people constantly forget. Before you judge your regression model, you need a baseline. The simplest baseline is this. Predict the average. If you are predicting house prices and the average price in your training data is four hundred thousand pounds, then a model that just predicts four hundred thousand pounds for every single house, no matter what, that is your baseline. If your fancy linear regression model cannot beat that, it is not useful. It might even be worse than just guessing the average every time.

This sounds like a trivial point, but it is one of the most important habits in machine learning. Always compare against a dumb baseline. The average is the dumbest regression baseline. If your model does not beat it, something is wrong. Maybe your features do not contain the information you need. Maybe your model is too simple or too complex. Maybe your data is noisy. But you will never know unless you measure against that baseline first.

So how do you measure whether your model is good? You need a metric, a single number that summarises how wrong your predictions are on average. There are two you will see everywhere, and you should understand the difference between them in your gut, not just on paper.

The first is mean absolute error. Take each prediction, find the difference from the truth, take the absolute value so negatives become positives, and average them all. The result is in the same units as your target. If you are predicting pounds, the mean absolute error is in pounds. If it comes out as twenty thousand pounds, that means on average your predictions are off by twenty thousand pounds. It is beautifully easy to explain to a business stakeholder. On average we are off by this much.

The second is root mean squared error. This one is a bit more involved. You take each error, square it, average the squared errors, and then take the square root of that average. The squaring has an important effect. It punishes large errors much more than small ones. Being off by ten pounds produces an error contribution of one hundred. Being off by one hundred pounds produces an error contribution of ten thousand. The big miss hurts a hundred times more, not just ten times more, because of the squaring. Then you take the square root to bring the number back into the original units so it is interpretable again.

So when do you use which? Here is the way I think about it. If large errors are roughly the same kind of problem as small errors, just bigger, then mean absolute error feels right. If being off by a lot is disproportionately painful, if a huge miss could cause a crisis while a small miss is just a minor inconvenience, then root mean squared error captures that better. For example, if you are predicting demand for a perishable product and being off by a small amount just means a slightly suboptimal order, but being off by a huge amount means a stockout that loses a major customer, then the squared metric aligns better with the real cost. On the other hand, if you are predicting how many minutes a bus will be late and a five minute error is about five times as annoying as a one minute error, not twenty five times as annoying, then mean absolute error is the more honest reflection of the cost.

There is a trap here that you need to understand. Because squared error punishes large errors so heavily, a single outlier can dominate the whole metric. Imagine you are predicting delivery times. Ninety nine of your deliveries are predicted well, each off by about two minutes. But one delivery was delayed by three hours because of a freak accident, and your model predicted a normal time. That one giant error, when squared, can be larger than all the other errors combined. Your root mean squared error will look terrible, driven almost entirely by that one data point. Your mean absolute error will look much more reasonable because it treats that three hour miss as just a large but proportionate contribution.

This is not a bug in the metric. It is a feature. It forces you to ask a question. Do I care about that outlier? If the answer is yes, if that three hour delay is exactly the kind of thing I need to get right, then the squared error metric is telling me the truth. My model is not good enough. But if the answer is no, if that was a one off event that I cannot predict from my features and that I do not need to optimise for, then I might want to look at mean absolute error instead, or I might want to remove or cap that outlier before training. The choice of metric is a judgement call about what kind of wrongness matters to you. This connects directly to what we discussed last time. The loss function is where your domain knowledge enters the system.

Let me bring this together with a small story. Imagine you work for a logistics company and you build a regression model to predict how many minutes late each delivery will arrive. You have features like distance, time of day, weather, driver experience, and traffic conditions. You train a linear regression model. Your baseline, predicting the average delay of twelve minutes, gives a mean absolute error of eight minutes. Your model brings that down to four minutes. The root mean squared error goes from fifteen minutes to seven minutes. You check for outliers and find a handful of extreme delays that pull the squared error up, but they represent real costly failures, so you decide to keep them in the evaluation.

Now your manager asks you to explain the result. Here is how you would do it. You say, on average our predictions are off by about four minutes, which is half the error of just guessing the average every time. The model is most useful when the delay is likely to be large, because those are the cases where being prepared matters most. The squared error metric tells us we still struggle with the worst cases, the extreme delays, and that is where we should focus next, either with better features or a different modelling approach.

Notice what you did there. You did not recite a formula. You compared to a baseline. You used the metric that matched the business cost. You acknowledged where the model falls short. That is what good regression practice looks like. Not the most complex model, but the right model, measured the right way, explained in language the business can act on.

As you walk, try this. Think of a number you might want to predict in your own work. What would the average be as a baseline? What would a small error cost you, and what would a huge error cost you? Would you rather be judged by mean absolute error or root mean squared error? If you can answer that, you already understand more about regression than most people who have just run their first model.

Quick recap. Regression means predicting a continuous number, like a price, a quantity, or a delay in minutes. Linear regression is the simplest version. It works by taking each input feature, multiplying it by a learned weight, adding all those up, and adding an intercept on top. The weights tell you how much each feature matters and in which direction. The model learns those weights by minimising a loss function, usually squared error.

Before you judge any regression model, compare it against a baseline. The simplest baseline is predicting the average every time. If your model cannot beat that, it is not adding value.

Two metrics matter most. Mean absolute error tells you the average size of your mistakes in the same units as your target. It is easy to explain. Root mean squared error also punishes large errors disproportionately because of the squaring step. Use mean absolute error when big errors are just proportionally worse. Use root mean squared error when a huge miss is disproportionately costly.

Watch out for outliers. A single extreme error can dominate your squared error metric and make a good model look bad. Ask yourself whether that outlier is something you need to predict or something you can safely set aside. That choice is a domain judgement, not a mathematical one.

When you explain a regression result to someone in business, compare to the baseline, state the metric in plain units, and be honest about where the model still struggles. That is what makes a numeric predictor genuinely useful.