Train, validation, and test — honest evaluation

Welcome back. This is lesson four in our machine learning series. Last time, we talked about examples, features, and labels, the raw material of learning. Now we need to talk about something even more fundamental than the data itself. We need to talk about honesty. Specifically, how do you know if your model is actually any good? And how do you avoid lying to yourself about it?

Here is the problem. You build a model. You feed it data. It makes predictions. Some are right, some are wrong. You measure how many it got right, and you get a number. Ninety percent accuracy. Brilliant, you think. But here is the question that should keep you up at night. Ninety percent on what? On the data it already saw? Or on data it has never seen before? Those are very different claims, and confusing them has sunk more machine learning projects than every bug and bad algorithm combined.

Let me give you a concrete example. Imagine you are a teacher. You give your students a practice exam with twenty questions. They study it, memorise every answer, and then you give them the same twenty questions on the real exam. They score perfectly. Are they brilliant? You have no idea. You never tested their understanding. You tested their memory. A model is the same. If you evaluate it on the same data it learned from, you are not measuring how well it generalises. You are measuring how well it memorised. And models are very, very good at memorising, especially modern ones with lots of parameters.

This is why we split our data. We take our dataset and we divide it into separate piles. The first pile is the training set. This is what the model learns from. It sees every example, every feature, every label. It adjusts its internal parameters to fit this data. Think of it as the practice exam. The model is allowed to look at the answers and learn from them.

The second pile is the validation set. The model never learns from this data directly. Instead, after each round of learning, you test the model on the validation set and see how it performs. You use this to make decisions. Should I train for more rounds? Should I change the learning rate? Should I try a different model architecture? Should I add or remove features? Every tuning decision, every choice about how to configure your model, should be guided by performance on the validation set. Think of it as a dress rehearsal. The model has not seen these examples during learning, but you are using the results to adjust your approach.

The third pile is the test set. This is the one you touch exactly once, at the very end, after you have made every decision and frozen your model. You run the model on the test set and you report the result. That number is your honest estimate of how the model will perform in the real world. Think of it as the final exam. No peeking, no revising, no going back to change things. One shot. What you get is what you report.

Now, why three piles and not two? Why not just train and test? Because of a subtle form of cheating that creeps in when you are not looking. If you only have a training set and a test set, you train your model, you test it, and you get a mediocre result. So you go back, change some settings, train again, test again. Better. You change more settings, train again, test again. Worse. You try another configuration, train, test. Slightly better. You keep doing this until you get a result you like. But here is the problem. You have now tuned your model to perform well on the test set. You have indirectly memorised the test set through your own choices. The test set is no longer an honest measure of real world performance. It has become a second training set, just one that you used more carefully.

The validation set prevents this. You do all your tuning against the validation set. You try different settings, you compare results, you pick the best configuration. The test set stays locked away the entire time. Only when you are completely done, when you have chosen your final model and you are ready to report results, do you unlock the test set and run it once. Because you never tuned against the test set, it remains an honest judge.

Let me make this more concrete with an example. Suppose you are building a model to predict whether a customer will cancel their subscription. You have data from the past two years. You split it randomly into three piles. Eighty percent goes to training, ten percent to validation, ten percent to test. The model learns from the training pile. You check it against the validation pile and notice it is overfitting, meaning it does great on training but poorly on validation. So you simplify the model. You check again on validation. Better. You try a different feature set. Check on validation. Slightly worse. You go back to the previous version. Finally, you are happy with the validation performance. You run the model on the test set. You get your final number. That is the number you report to your stakeholders. That is your honest estimate.

Now, there is an important wrinkle. I said you split the data randomly, and often that is right. But sometimes random splitting is dangerous. When does random splitting fail? It fails when the order of your data matters. Think about time. If you are predicting stock prices, or customer churn, or website traffic, or anything where the world changes over time, then a random split mixes the past and the future together. Your training set might contain data from December, and your test set might contain data from October. That means you are training on the future and testing on the past. In the real world, you will always train on the past and predict the future. A random split lets your model cheat by learning from the future.

When order matters, you should split by time. Put the earliest data in training, the middle chunk in validation, and the latest data in test. Now your model is in the same position it will be in production. It learns from the past, tunes on the more recent past, and is evaluated on the most recent data, which is the closest thing to the future you have. This is called a temporal split or time-based split, and for any problem where the world evolves, it is the honest way to evaluate.

There is another technique you should know about, called cross-validation. The idea is simple. Instead of splitting your data into one training set and one validation set, you divide it into several chunks, say five. Then you train on four chunks and validate on the fifth. You repeat this five times, each time leaving out a different chunk for validation. You average the five validation scores to get a more stable estimate of how your model performs. Cross-validation is especially useful when you do not have much data, because it lets you use every example for both training and validation, just not at the same time. It is like taking five different dress rehearsals and averaging the results. You still hold out a separate test set for the final evaluation. Cross-validation improves your validation, not your test.

Now, even with perfect splitting, there is one more threat to honesty that you need to understand. It is called distribution shift. Here is what happens. You train your model on data from January through June. You test it on data from July. It performs well. You deploy it. Then October comes, and the model starts making bad predictions. What happened? The world moved. Your customers changed their behaviour. A competitor launched a new product. The economy shifted. A new regulation took effect. The data your model sees in production no longer looks like the data it was trained on. The distribution has shifted.

Distribution shift is the most common reason models degrade in production. It is not a bug in your code. It is not a flaw in your algorithm. It is the simple fact that the world is not stationary. The relationship between features and labels that held in June may not hold in October. This is why monitoring is essential. You need to watch your model's performance over time and be ready to retrain when the world has moved enough that your model is no longer trustworthy.

A good way to catch distribution shift early is to compare the features coming in at prediction time against the features in your training data. If the average values have drifted significantly, or if new categories have appeared that your model never saw, that is a warning sign. You do not need to wait for predictions to go wrong. You can often detect the shift in the inputs before the outputs suffer.

Let me bring all of this together with a spoken diagram. Imagine three piles of data sitting in front of you. On the left is the training pile. In the middle is the validation pile. On the right is the test pile. The training pile is the biggest. The model is allowed to look at it as much as it wants. It learns from it. It fits its parameters to it. The validation pile is smaller. The model is not allowed to learn from it, but you, the developer, are allowed to look at how the model performs on it. You use it to choose between models, to tune settings, to decide which features to include. The validation pile influences your decisions, but it does not directly influence the model's parameters. The test pile is the smallest, and it is locked. Nobody looks at it. Nobody tunes against it. The model does not learn from it. You do not use it to make decisions. Only at the very end, when everything is settled, do you unlock it, run the model across it once, and report the result. That result is your promise of honesty. It is the closest thing you have to a prediction of real world performance.

If you take only one thing from this lesson, let it be this. The goal of evaluation is not to get a good number. The goal is to get an honest number. A good number that is dishonest will get you into trouble in production. A mediocre number that is honest lets you make informed decisions. Intellectual honesty is not just a virtue in machine learning. It is a practical necessity. The model that looked great on training data and failed in production was not a technical failure. It was an honesty failure. The splits are your defence against that failure.

As you keep walking, try this. Think about a model you might build at work. What would your training, validation, and test sets look like? Would a random split be safe, or do you need a time-based split? And if you deployed the model today, what could change in the world six months from now that might make it less accurate? Holding that question in your mind is the habit that separates people who build models from people who build models that actually work.

Quick recap of what we covered today. You split your data into three piles. Training is what the model learns from. Validation is what you use to tune and choose between models. Test is what you touch once at the end to get an honest final number. Testing on your training data is like giving students the same exam they practised on. It measures memorisation, not understanding. If you tune against your test set, it stops being honest and becomes a second training set. That is why the validation set exists, to absorb all your tuning decisions so the test set stays clean. When order matters, like in time series data, split by time rather than randomly. Train on the earliest data, validate on the middle, test on the most recent. Cross-validation means splitting your data into several chunks and rotating which one is held out for validation, giving you a more stable estimate. But you still keep a separate test set for the final honest check. Distribution shift is when the world changes after you deployed your model, so the data in production no longer matches your training data. It is the most common reason models degrade over time. Monitor your inputs and outputs, and retrain when the world has moved. The three piles are training, validation, and test. Training influences the model's parameters. Validation influences your design decisions. Test influences nothing. It only reports the truth.