Training Data vs Test Data

The mistake that ruins more ML projects than any bad algorithm

Imagine you're calibrating a sensor. You take 50 measurements against a known reference, fit a calibration curve, and then report the error of that curve — measured against the same 50 points you used to fit it.

Any engineer would immediately spot the problem: you're grading your own homework. The calibration curve was specifically designed to fit those 50 points. Of course it looks good against them. The real question is how well it performs on new measurements it hasn't seen before.

This is the single most important concept in machine learning, and it's the source of more failed projects than any wrong algorithm choice or bad hyperparameter setting:

The reason is exactly the same as the sensor calibration example. A model that was trained on a dataset has been specifically optimised to fit that dataset. Measuring its performance on the same data tells you how well it memorised, not how well it learned. And memorising is useless — you already know the answers for those data points. What you need is a model that gives good answers for data it's never seen before.

The train/test split: your first line of defence

The solution is simple. Before you do anything else, you split your dataset into two parts:

Training set (~70–80% of your data). This is the data the model learns from. It sees the features and the labels, and uses them to find patterns.

Test set (~20–30% of your data). This is data the model has never seen during training. You only use it at the very end, to measure how well the model performs on genuinely new data.

The test set acts as a stand-in for the real world. If your model performs well on the test set, you have reasonable confidence it will perform well on future data too — because the test set was just as unknown to the model as tomorrow's data will be.

If you've ever done experimental validation in engineering — designing a test to independently verify a simulation, a calculation, or a calibration — the logic is identical. The training set is where you build the model. The test set is where you validate it. You'd never skip the validation step for a simulation, and you should never skip it for an ML model.

What happens if you skip the split

Let's make this concrete with an engineering example.

Suppose you have 200 data points from an injection moulding process. Each row contains five process parameters (features) and a shrinkage measurement (label). You train a decision tree model on all 200 points and measure its prediction error — on those same 200 points.

The model reports an error of nearly zero. Excellent, right?

Then you take it to the production floor and use it to predict shrinkage for new parameter combinations. The predictions are terrible. Nowhere close to reality.

What happened? The model overfitted. It didn't learn the general relationship between process parameters and shrinkage — it memorised the specific 200 data points it was given. It found patterns that exist in your particular dataset but don't hold up in reality. Random noise, measurement quirks, specific batch effects — it treated all of these as real patterns.

Overfitting is one of the most important concepts in ML, and it gets its own article later in this knowledge base. But the train/test split is your first and simplest defence against it. If you always evaluate on data the model hasn't seen, you'll catch overfitting immediately — because an overfit model performs well on training data and poorly on test data.

But wait — what about the validation set?

In practice, you often need three sets, not two. Here's why.

When building a model, you don't just train it once. You experiment. You try different algorithms, adjust settings (called hyperparameters — more on those in a later article), add or remove features, and compare results. Each time you make a change, you need to check whether the model got better or worse.

If you use the test set for those comparisons, you have a problem. You're making decisions based on the test set, which means you're indirectly fitting your model to it. By the time you're done experimenting, the test set is no longer truly "unseen" — you've leaked information from it into your model selection process.

The solution is to carve out a third piece:

Training set (~60–70%). The model learns from this.

Validation set (~15–20%). You use this to compare different models and tune settings during development. It guides your decisions, but the model doesn't directly train on it.

Test set (~15–20%). You touch this once, at the very end, to get a final, unbiased performance estimate. Think of it as a sealed envelope.

How to actually split your data

In practice, you don't manually separate rows into folders. Scikit-learn does it for you in one line:

     
from sklearn.model_selection import train_test_split

# X = your features table, y = your labels column
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
                    

A few things worth noting:

The split is random. Each sample has an equal chance of ending up in either set. This is important because if your data has some natural ordering — say, chronological, or sorted by batch — a non-random split could put all the "easy" samples in one set and all the "hard" ones in the other, giving you a misleading result.

random_state=42 makes it reproducible. The number 42 isn't special — it's just a seed that ensures you get the same split every time you run the code. This matters when you're comparing models: you want them all evaluated on exactly the same test set. (And yes, the number is a reference to The Hitchhiker's Guide to the Galaxy. Data scientists have a sense of humour.)

test_size=0.2 means 20% for testing. This is a common starting point. With 500 samples, that gives you 400 for training and 100 for testing. With very small datasets (under 100 samples), you might need a different approach — we'll cover cross-validation in a later article, which is designed for exactly that situation.

Common pitfalls engineers should watch for

Time-dependent data needs special treatment. If your data has a time component — sensor readings, production logs over weeks, seasonal effects — you can't just randomly split it. Future data points might contain information about the past that wouldn't be available in practice. Instead, you split chronologically: train on earlier data, test on later data. This mimics how the model would actually be used.

Don't split, then preprocess. Preprocess, then split? Actually: split first, preprocess second. This is a subtle but important point. If you normalise your features (e.g., scale them to zero mean and unit variance) using the entire dataset before splitting, the test set statistics have leaked into your training set. The correct approach is to split first, then fit your preprocessing only on the training set, and apply it to the test set. Scikit-learn's Pipeline object handles this automatically — we'll cover it in a later article.

Stratification for imbalanced data. If you're doing classification and 95% of your parts are "pass" and 5% are "fail," a random split might accidentally put all the "fail" samples in the training set and none in the test set (or vice versa). Stratified splitting ensures both sets maintain the same ratio of classes. In scikit-learn, this is as simple as adding stratify=y to the split:

     
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
                    

The engineering mindset advantage

If you come from a testing or validation background, you already think this way. You wouldn't sign off on a simulation result without independent verification. You wouldn't trust a calibration that was only checked against its own reference points. You wouldn't submit a test report where the "test" was just re-running the same conditions used to develop the method.

The train/test split is the ML version of this instinct. It's not a statistical curiosity — it's quality assurance for your model. The only difference is that in engineering, independent verification is standard practice. In ML, beginners often skip it because nobody told them it matters. Now you know.

When you see ML results reported without a proper train/test split — or worse, when someone reports "99% accuracy" without mentioning how they evaluated — you'll know to be sceptical. That's a model that graded its own homework. And you'd never accept that from a colleague.

Key takeaways

Never evaluate on training data. A model tested on the data it was trained on tells you how well it memorised, not how well it learned. This is the most common beginner mistake in ML.

Split your data before anything else. Typically 80% for training, 20% for testing. Use scikit-learn's train_test_split — it takes one line of code.

For serious work, use three sets. Training (learn), validation (compare and tune), test (final sealed-envelope evaluation). This mirrors the engineering pattern of development testing followed by formal qualification.

Watch for leakage. Split before preprocessing. Handle time-series data chronologically. Stratify when your classes are imbalanced. These details matter more than algorithm choice in many real-world projects.

You already have the right instincts. Independent verification isn't a new idea — it's something every engineer practises. The train/test split is just the ML implementation of a principle you already trust.