Train, Validation and Test Sets Explained

Every machine learning model is trained on examples, then marked on how well it does.

The catch is which examples it is marked on. Standard practice splits the data into a training set, a validation set and a test set the model never learns from, so the score reflects new data, not memory.

Updated 16 October 2026 · 6 sources

Hold data back
Tune on validation
Test once, at the end
Three habits behind an honest score

Check a test set-up

THE SHORT ANSWER

A score only counts
on data it has not seen.

Three ideas from standard machine learning guidance,
in plain English.

THE RULE

Hold some data back

scikit-learn's guide calls training and testing on the same data a methodological mistake: a model that just repeats labels it has seen would score perfectly and predict nothing useful.

Check a set-up

THE TEST SET

Save it for the end

Google's course advises tuning on a validation set and using the test set to double-check. Both sets wear out with repeated use, so fresh data helps.

The workflow

LEAKS

Keep the test out

Data leakage is when information not available at prediction time is used to build the model. It gives overly optimistic scores and poorer results on genuinely new data.

See the numbers

A small puddle of water spreading across a wooden table from the base of a clear glass jug, as if from an unseen leak — AI-generated image, not a photograph of a real person or place.

A FAIR WORKFLOW IN FOUR STEPS

Split first,
look at the test last.

These steps follow the scikit-learn user guide and Google's Machine Learning Crash Course. They apply whether the model is a small classifier or something much larger.

Remove duplicates, then split
Divide the data into training, validation and test sets, and delete test or validation examples that duplicate training ones.
Learn preprocessing from training data only
Fit scaling, imputation or feature selection on the training set, then apply the same transformation to the other sets.
Tune on validation or k-fold
Compare settings on the validation set, or average scores across k folds of the training data.
Test once, and refresh
Use the test set for the final check, and collect new data to refresh it when it has been used too often.

TWO NUMBERS WORTH KNOWING

New data is harder,
and leaks are common.

One finding from rebuilding a famous test set,
one from a survey of leakage in science.

11–14%

Accuracy drop on a freshly built ImageNet test set

Researchers rebuilt the ImageNet and CIFAR-10 test sets following the original methods. A broad range of models lost 11% to 14% accuracy on the new ImageNet set and 3% to 15% on CIFAR-10.

NEW TEST SETS · RECHT ET AL. 2019

329 papers

Papers in 17 fields where leakage errors were found

A survey of research fields that adopted machine learning found errors across 17 fields affecting 329 papers, in some cases leading to wildly over-optimistic conclusions.

LEAKAGE · KAPOOR & NARAYANAN

Recht and colleagues found that models which gained accuracy on the original test sets still gained more on the new ones, and put the drops down to slightly harder images rather than to reuse of the old sets.

A QUICK SET-UP CHECK

How is the model
being marked?

Pick a way of testing a model to see what the guidance says about it. The bar shows how likely the score is to be over-optimistic.

Based on the scikit-learn user guide and Google's Machine Learning Crash Course. General information about evaluation practice, not a review of any product.

Evaluation check

Choose a set-upHeld-out test set, used once at the endSame test set used to tweak, round after roundTest set full of training duplicatesPreprocessing fitted on all the dataMarked on the training data itself

Held-out test set, used once at the endFair test

Train on the training set, tune on a validation set or with k-fold cross-validation, then do a final check on the test set. This is the workflow both guides describe.

Same test set used to tweak, round after roundWears out

Google's course warns that the model can implicitly fit the peculiarities of a test set used to guide changes, like a teacher teaching to the test.

Test set full of training duplicatesInflated

In Google's spam example, duplicate emails in the test set gave 99% precision on both sets. The advice is to delete test examples that duplicate training ones.

Preprocessing fitted on all the dataLeaked

In scikit-learn's demo, selecting features before splitting gave 0.76 accuracy on completely random labels, where about 0.5 is expected. Split first, then preprocess.

Marked on the training data itselfMeaningless

A model that memorised the labels would score perfectly. scikit-learn's guide calls this overfitting and says to hold out part of the data as a test set.

Sources: scikit-learn user guide, Cross-validation and Common pitfalls (data leakage); Google for Developers, Machine Learning Crash Course: Dividing the original dataset.

A sealed plain envelope resting on a desk beside a pile of blank practice sheets and a pencil — AI-generated image, not a photograph of a real person or place.

WHY BIG MODELS NEED THIS

Big networks can
memorise anything.

A 2017 study found that image networks trained in the usual way easily fit training data whose labels had been randomly shuffled, and even pure random noise. A perfect training score proves very little.

The same study notes that the number of parameters usually exceeds the number of data points. That is why the score that matters is the one on data the model was never trained on.

In conversation

Questions people ask

What is the difference between a validation set and a test set?

The validation set is used during development to compare settings and catch problems. The test set is kept back for a final check, so it gives a cleaner estimate of performance on new data.

What is overfitting?

It is when a model fits the examples it was trained on so closely that it does poorly on new ones. scikit-learn's guide gives the extreme case of a model that just repeats labels it has seen.

What is k-fold cross-validation?

The training data is split into k parts. The model is trained on k minus one parts and checked on the remaining one, k times over, and the scores are averaged. A separate test set is still held back.

What is data leakage?

Using information that would not be available at prediction time when building the model, for example fitting preprocessing on all the data before splitting it. It makes scores look better than they really are.

Can a test set really wear out?

Yes. Google's course says the more you use the same data to make decisions about a model, the less confidence you can have that it will do well on new data, and suggests collecting more data to refresh it.

Sources

Each figure on this page comes from the sources below.

Practice on one set,
sit the exam on another.