Trees, forests, and ensemble intuition

Welcome back. This is lesson nine in the machine learning series, and today we are going to talk about one of the most practical and widely used families of models in the field. Decision trees, random forests, and the broader idea of ensembles. If you work with tabular data, meaning data that looks like a spreadsheet with rows and columns, these models are likely the first thing an experienced practitioner will reach for. They are not always the flashiest, but they win an enormous number of real business problems. By the end of this walk, you will understand why.

Let us start with the decision tree itself. The idea is beautifully simple. A decision tree is just a sequence of yes or no questions about your features. Imagine you are trying to predict whether a customer will cancel their subscription next month. The tree might start by asking, has this customer logged in during the past two weeks? If the answer is no, it asks a follow-up. How many months have they been a customer? If the answer is less than three, maybe the tree predicts a high chance of cancellation. If the answer is more than three, it asks another question. Did they contact support recently? And so on, branching down until it reaches a prediction.

Each question splits the data into two groups. Each answer leads to another question or to a final prediction at the bottom, which people call a leaf. The tree learns these questions automatically from your data, picking the splits that best separate the outcomes. It looks at each feature, finds the threshold that creates the cleanest division between, say, customers who cancel and customers who stay, and uses that as its first question. Then it repeats the process on each subgroup.

The great advantage of a decision tree is that you can picture it. You can draw it on a whiteboard. You can explain it to a non-technical stakeholder. If someone asks why the model predicted a particular customer would cancel, you can trace the exact path. Well, they had not logged in for two weeks, they had been a customer for only two months, and they did contact support. That is the path the tree took, and that path is fully transparent. This is a rare and valuable property in machine learning.

But trees have a serious weakness, and it connects directly to what we covered last time. A tree can overfit badly if you let it grow too deep. Think about it. If the tree keeps asking questions and splitting the data, eventually each leaf contains just a handful of examples, maybe even one. At that point, the tree is not learning a general pattern. It is memorising individual cases. You saw the symptoms of overfitting in the last lesson. A tree grown without any limit is the poster child for this problem. It will achieve near-perfect accuracy on training data and then fall apart on new data because its questions became too specific, too tailored to the exact examples it saw.

So in practice, you control tree depth. You might say, only allow five levels of questions. Or you might say, each leaf must contain at least twenty examples. These limits keep the tree honest. They force it to learn broader patterns rather than memorising individual rows. A shallow tree might underfit, missing real structure in the data. A very deep tree overfits. Somewhere in between is your sweet spot, just as we discussed last time.

Now, here is where the story gets interesting. A single decision tree, even well-tuned, is often not the most accurate model you can build. But something remarkable happens when you combine many trees together. This is the core idea behind ensembles, and it is one of the most powerful intuitions in all of machine learning.

Think about it this way. If you ask one person for their opinion on a difficult question, they might be wrong. If you ask a hundred people and take a vote, the collective answer is often more reliable than any individual answer. This is not a machine learning idea. It is a statistical idea that goes back centuries. The wisdom of the crowd. In machine learning, we apply this to models.

A random forest takes this idea literally. It builds many decision trees, often hundreds, and lets them vote on the final prediction. But there is a crucial detail. If every tree is identical, voting accomplishes nothing. You need the trees to be different from each other, to make different mistakes. The random forest creates this diversity in two ways. First, each tree is trained on a different random sample of the data, drawn with replacement. This means some rows appear multiple times in one tree's training set and do not appear at all in another's. Second, at each split, the tree is only allowed to consider a random subset of the features. It cannot always pick the most obvious feature. It has to work with what it is given.

These two sources of randomness mean each tree sees a slightly different view of the problem. Each tree makes different errors. And when you average their predictions, the errors cancel out. The collective prediction is more stable and more accurate than any single tree. This is why random forests are so popular for tabular data. They are remarkably strong out of the box, they handle messy data well, they do not require careful feature scaling, and they are hard to badly overfit because the averaging provides a natural buffer.

There is another way to combine trees, and it is called boosting. The intuition is different from a random forest. In a random forest, each tree is built independently and they vote. In boosting, trees are built one after another, and each new tree focuses on correcting the mistakes of the ones that came before it.

Here is a concrete way to picture it. Imagine you have a team of people trying to predict house prices. The first person makes their best guesses. Some are too high, some are too low. The second person does not try to predict all house prices from scratch. They focus only on the cases where the first person was wrong. They try to correct those specific errors. The third person then focuses on the remaining errors that the first two could not fix. And so on. Each person in the sequence is specialised for the mistakes left behind by the team so far.

This is what boosting does. It builds a sequence of simple trees, each one trained to fix the residual errors of the combined model so far. The result is a model that can be extremely accurate, often more accurate than a random forest, because it is systematically hunting down its own weaknesses. Boosting models dominate many business competitions and real-world tabular problems for this reason. They are powerful, they are fast, and they handle the kind of structured data that most businesses actually have.

So why do ensembles win so many tabular business problems? Let me put the pieces together. Most business data sets are wide, with many columns of mixed types. Some numeric, some categorical, some with missing values. Decision trees handle this naturally. They do not care about feature scales. They do not need normalisation. They can split on a numeric threshold or on a category directly. And when you combine many trees through random forests or boosting, you get a model that captures complex interactions between features without requiring you to engineer those interactions by hand. The trees discover them automatically through their sequence of questions.

For a lot of business problems, predicting customer churn, estimating insurance claim costs, forecasting inventory demand, detecting fraudulent transactions, this combination of flexibility and raw predictive power is exactly what you need. Ensembles are not always the best choice for unstructured data like images or text, where neural networks tend to dominate. But for the spreadsheet-shaped data that fills most corporate databases, ensembles are often the strongest practical choice.

Now, there is a trade-off, and you need to understand it clearly. As you move from a single decision tree to a random forest to a boosted ensemble, accuracy generally goes up. But explainability goes down. A single tree, you can draw and trace and explain. A random forest with three hundred trees, you cannot easily point to one path. You can still extract useful information, like which features are most important across all the trees, but you cannot tell a stakeholder the exact reasoning for one specific prediction in the same transparent way. A boosted ensemble is even more opaque. The sequence of error-correcting trees is harder to interpret than a single tree or even a forest.

This is the accuracy versus explainability trade-off, and it is one of the most important judgements you will make in practice. In some contexts, explainability is not just nice to have. It is legally required. If you are denying someone a loan, in many jurisdictions you must be able to explain why. If you are making a clinical decision that affects patient care, doctors may need to understand the reasoning. In these cases, a single transparent tree, even if it is less accurate, might be the right choice. In other contexts, the prediction itself matters more than the explanation. If you are ranking products for a search result or predicting whether a machine will fail in the next hour, the accuracy might matter more than the interpretability, and a boosted ensemble is the better tool.

So when might you prefer a simple tree over a black box? Let me give you three situations. First, when you are in a regulated industry and you must explain individual decisions. Second, when you need to communicate the model's logic to non-technical stakeholders as part of getting buy-in. Sometimes showing someone the actual tree, the actual questions it asks, builds more trust than any accuracy number. Third, when you are in the early stages of a project and you want a quick, interpretable baseline. A simple tree trained in minutes can tell you whether your features contain useful signal at all. If a shallow tree cannot beat a naive baseline, your problem is probably in the data, not the model. You do not need a complex ensemble to discover that.

Here is how I would summarise the practical guidance. Start with a single tree to understand your data and get a baseline. Move to a random forest when you want strong accuracy with minimal tuning. Consider boosting when you want to squeeze out the best possible performance on tabular data and you are willing to accept a less interpretable model. And always, always keep the explainability question in mind from the start of the project, not as an afterthought at the end.

As you finish your walk, try this. Think of a prediction problem at work, one that involves structured data in a spreadsheet or a database. Imagine building a single decision tree for it. What would the first question be? What feature would it split on first? Now imagine growing a forest of three hundred such trees, each seeing slightly different data. Would you trust the forest more than the single tree? And would your stakeholders accept the forest's answer if you could not explain the exact path it took? Holding both sides of that trade-off in your mind is what makes you a thoughtful practitioner. Next time, we will look at how to measure a classifier across all thresholds at once, so you can compare models before committing to a single cutoff. For now, enjoy the rest of your walk.

Quick recap of today's lesson. A decision tree is a sequence of yes or no questions about your features. Each answer leads to another question or a final prediction. Trees are easy to picture and easy to explain, which makes them valuable when transparency matters. But a tree grown too deep will overfit, memorising individual examples instead of learning general patterns, just as we discussed in the previous lesson.

A random forest builds many trees, each trained on a different random sample of the data and with a random subset of features at each split. The trees vote on the final prediction. Because each tree makes different errors, averaging them produces a more stable and accurate result than any single tree.

Boosting builds trees in sequence, where each new tree focuses on correcting the mistakes of the combined model so far. This makes boosting extremely accurate, often the strongest approach for tabular business data.

Ensembles win many business problems because they handle mixed feature types, missing values, and complex interactions without manual engineering. But there is a trade-off. As you move from a single tree to a forest to a boosted ensemble, accuracy goes up and explainability goes down.

Prefer a simple tree when you need to explain individual decisions, when you are building trust with non-technical stakeholders, or when you want a quick interpretable baseline early in a project. Start simple, reach for ensembles when the extra accuracy is worth the loss of transparency, and always keep the explainability question in mind from the start.