Label Errors in Machine Learning Data: How to Find Mislabeled Examples, Review Them and Re-Check Your Model

A machine learning model learns from labels. A label is the answer attached to an example, such as "spam" on an email or "cat" on a photo. If some labels are wrong, the model learns from wrong answers, and your test score may be wrong too.

This article explains label errors in plain English. You will see how they affect training and evaluation, how open-source tools flag them, and how to review flagged examples without trusting the tool blindly. We follow one illustrative example: a support-ticket classifier.

Article index

• What label errors are
• How wrong labels affect training
• How wrong labels affect evaluation
• How tools flag likely label errors
• A worked example
• Review, privacy and limits
• Common mistakes
• Recap
• Research Sources

What label errors are

A label error is an example whose given label does not match the truth. A refund request might be tagged "bug report". A photo of a fox might be tagged "dog". People make these slips when they are tired, when instructions are vague, or when two classes overlap.

Some errors come from unclear categories rather than careless work. The confident learning paper, for example, measures overlap between related classes in ImageNet, such as images labeled with a child class that actually belong to its parent class (Northcutt, Jiang and Chuang). That is a documented finding about one dataset. Your own data will have its own pattern.

Labels can also be wrong on purpose. An attacker who can edit labels can plant bad examples. That is a data-poisoning risk, so label checks help with quality and with basic data hygiene.

How wrong labels affect training

A large neural network can memorize its training set. Researchers showed that state-of-the-art convolutional networks trained with stochastic gradient methods can "easily fit a random labeling of the training data" (Zhang and colleagues, ICLR 2017). That experiment used fully random labels, which is an extreme case.

My explanation of what follows: a model with enough capacity can also fit a few wrong labels. It then treats a mistake as a pattern. The model may look fine on training data and still fail on new data.

The confident learning authors report that cleaning data with their method before training moderately improved accuracy for a ResNet on ImageNet, and improved sentiment classification on Amazon Reviews text (paper abstract). Gains depend on your data, so measure them yourself.

How wrong labels affect evaluation

Wrong labels in a test set are arguably worse. The test set is the ruler you use to compare models. A bent ruler gives bad comparisons.

A study of 10 widely used vision, language and audio datasets found label errors in their test sets, averaging at least 3.3%, and at least 6% of the ImageNet validation set (Northcutt, Athalye and Mueller). The authors flagged candidates with algorithms and had people check them. About 51% of flagged candidates were confirmed as mislabeled, on average.

The same paper reports that rankings can change. On ImageNet with corrected labels, ResNet-18 beat ResNet-50 if the share of mislabeled test examples rose by only 6%. On CIFAR-10, VGG-11 beat VGG-19 if the share rose by only 5%. These are the authors' results on those benchmarks, not a rule for every project.

The lesson is practical. If two models score close together, label noise in the test set can decide which one looks better.

How tools flag likely label errors

The idea is simple. Train a model, then look for examples where the model confidently disagrees with the given label. Those examples deserve a human look.

The catch is that a model must not predict an example it has already trained on, because it may have memorized that label. The fix is cross-validation. Scikit-learn splits the data into folds, trains on all but one fold, and predicts the held-out fold. Its cross_val_predict function returns, for each sample, the prediction made when that sample was in the test set (scikit-learn 1.9.1 documentation). The same page warns that this output is not an appropriate measure of generalization error, so use it for finding problems, not for reporting your score.

Confident learning is the method behind the open-source cleanlab library, which is released under the Apache-2.0 license. The paper describes it as estimating the joint distribution between given labels and unknown true labels, assuming a class-conditional noise process (paper). In plain terms, it assumes a label's chance of being wrong depends on its class, not on the individual example.

The cleanlab function find_label_issues takes your labels and an array of predicted class probabilities. Its documentation says results are most accurate with out-of-sample probabilities, and recommends cross-validation to get them. It returns a true/false mask by default, or ranked indices if you set return_indices_ranked_by (cleanlab filter documentation).

A worked example

Illustrative scenario, not a test run. A small company has 5,000 support tickets labeled billing, bug, account or other. They want a classifier that routes new tickets. The numbers below are invented to show the process.

Prerequisites: Python 3.10 or newer (the version cleanlab's page states), scikit-learn (the cited documentation is for 1.9.1), and cleanlab. Labels must be integers from 0 to K-1, where K is the number of classes. Work on a copy of the labels file. Nothing here is destructive if you never overwrite the original.

1. Freeze a copy of the original labels. Input: your labels file. What happens: you save it read-only. Result: a baseline you can return to. Why it matters: you can undo any wrong fix. Check: compare file hashes before and after.
2. Get out-of-sample probabilities. Input: a feature matrix X (for example, text turned into numbers) and integer labels labels. What happens: cross-validation trains five models and predicts each ticket with a model that did not see it. Result: one probability row per ticket. Why it matters: the documentation says flags are most accurate with these.

   from sklearn.linear_model import LogisticRegression
   from sklearn.model_selection import cross_val_predict
   
   pred_probs = cross_val_predict(
       LogisticRegression(max_iter=1000),
       X, labels, cv=5, method="predict_proba",
   )

   Check: pred_probs.shape should be (5000, 4) and each row should sum to about 1. If it fails, check that every class has at least five examples.
3. Rank suspicious tickets. Input: labels and pred_probs. What happens: cleanlab flags tickets whose label looks inconsistent with the probabilities and sorts them, worst first. Result: a list of ticket positions. Why it matters: you review a short list instead of 5,000 tickets.

   from cleanlab.filter import find_label_issues
   
   issues = find_label_issues(
       labels=labels,
       pred_probs=pred_probs,
       return_indices_ranked_by="self_confidence",
   )

   Check: issues should be a list of positions, not a mask. If you get a mask, you left out return_indices_ranked_by.
4. Have a person review the top of the list. Input: the ticket text, the given label and the model's top guess. What happens: a reviewer decides who is right, using written class definitions. Result: confirmed errors, plus cases where the class definitions are unclear. Why it matters: in the published study only about 51% of flagged candidates were confirmed on average, so flagged does not mean wrong.
5. Fix labels in a new file and record why. Input: confirmed errors. What happens: you write corrected labels to a new version with a short note per change. Result: a changelog. Why it matters: you can trace any score change back to a data change.
6. Re-train and re-evaluate on a corrected test set. Input: the corrected data, split as before. What happens: you train again and compare to the old score. Result: a new score you can trust more. Why it matters: both training and test noise are reduced. If the score barely moves, your labels were probably already good enough.

Keep the train and test split fixed while you compare. If you fix test labels, fix them for every model you compare, so the comparison stays fair.

Review, privacy and limits

Privacy. Label review shows real examples to people. Support tickets can hold names, emails and account details. Remove or mask personal data first, and check your data-handling rules before you send examples to outside reviewers or hosted services. Running an open-source library on your own machine is one option. Confirm what any tool you use sends over the network.

Security. Treat label changes like code changes. Keep a history, limit who can edit labels, and keep the original. A silent label edit is a way to damage a model.

Limits. Detection depends on the model. A weak model flags many correct labels, and a model can agree with a wrong label if the same mistake appears often. The confident learning paper proves consistency under stated conditions (paper), and real data may not meet them. The cleanlab page does not list formal limitations, so do not assume it catches everything.

Some "errors" are really unclear definitions. If reviewers disagree, rewrite the class definitions before you relabel.

Common mistakes

• Using predictions from a model trained on the same data. It may have memorized the labels. Use cross-validated, out-of-sample probabilities.
• Auto-deleting every flagged example. About half of flagged candidates were not errors in one study. Review a ranked sample first.
• Cleaning only the training set. Test-set errors can change model rankings. Review test labels too, and keep a record.
• Reporting cross_val_predict output as your final score. The scikit-learn docs say it is not an appropriate measure of generalization error. Use it to find problems only.
• Fixing labels without definitions. Write down what each class means so reviewers agree.
• Overwriting the original labels. Keep the original and version every change.

Recap

• Label errors damage both training and evaluation, and they can flip which model looks best.
• Published checks of 10 benchmark test sets found an average error rate of at least 3.3%, so even famous datasets are imperfect.
• Use cross-validation to get out-of-sample probabilities, then rank examples where the model and label disagree.
• Treat flags as leads. Only about 51% were confirmed in one study, so a person must decide.
• Protect personal data during review, version every label change and re-evaluate afterward.

Research Sources

• Confident Learning: Estimating Uncertainty in Dataset Labels (arXiv 1911.00068): first submitted 31 October 2019; JAIR 2021.
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (arXiv 2103.14749): first submitted 26 March 2021.
• Understanding deep learning requires rethinking generalization (arXiv 1611.03530): first submitted 10 November 2016; ICLR 2017.
• cleanlab repository on GitHub: open-source library, Apache-2.0 license.
• cleanlab documentation: filter module (find_label_issues)
• scikit-learn documentation: cross-validation (version 1.9.1)