Download App
Label noise handling isn't optional once your training data comes from weak supervision: noisy labels don't automatically doom a model, but training as if they were ground truth does. Treat labels as a signal with an estimable error rate: detect where they're probably wrong, combine weak sources into one probabilistic estimate instead of trusting any single one, and pick a loss function and evaluation set that don't just memorize the noise.
What "Noisy" and "Weakly Supervised" Actually Mean
Label noise is any case where the label attached to an example doesn't match reality. It shows up from tired annotators, genuinely ambiguous class boundaries, transcription errors, or automated heuristics that mislabel a predictable slice of the data.
Weak supervision is a strategy for generating labels at scale without paying for hand annotation on every example. Instead of one ground-truth labeler, you combine several imperfect sources: keyword rules, regex patterns, existing (possibly unrelated) models, distant supervision from a database, or crowdworkers doing quick low-effort passes. Each source is individually noisy, and weak supervision frameworks like Snorkel (Ratner et al., "Snorkel: Rapid Training Data Creation with Weak Supervision," VLDB 2017) are built specifically to combine several such sources statistically rather than by simple majority vote.
The two problems compound. Weakly supervised labels are noisy by definition, since they're synthesized from sources that were never meant to be authoritative on their own.
How Do You Detect Label Noise Before You Train?
Don't wait for a confusing validation curve to tell you something's wrong with the labels. Run a detection pass on the training set first.
Train a quick baseline model with k-fold cross-validation, then compare each example's out-of-fold predicted probability against its given label. Examples where the model confidently and consistently predicts a different class than the one it was assigned are your prime suspects. This is the core idea behind confident learning (Northcutt, Jiang, and Chuang, "Confident Learning: Estimating Uncertainty in Dataset Labels," Journal of Artificial Intelligence Research, 2021), which estimates a class-conditional noise matrix from exactly this kind of prediction-versus-label disagreement, without needing any ground truth to calibrate against.
This won't find every mislabeled example, and it will flag some genuinely hard or ambiguous cases as "noise" when they're just difficult. Treat the output as a prioritized review list, not a final verdict.
Combining Weak Supervision Signals Without Amplifying Their Errors
Once you have several labeling sources, the naive move is a majority vote across them. That fails as soon as two sources share a blind spot, because correlated errors don't cancel out, they reinforce each other and produce a majority that's confidently wrong.
The better approach, and the one Snorkel-style frameworks are built around, is a generative label model that learns each source's approximate accuracy and pairwise correlation from the agreement and disagreement patterns between sources, then outputs a probabilistic label per example. Sources that tend to agree with everyone get less independent weight than sources whose errors look genuinely uncorrelated with the rest.
Prioritize adding labeling sources with different failure modes over adding more sources that make the same mistakes in the same way. Five heuristics that all key off the same surface feature give you one opinion five times, not five opinions.
Training Techniques That Don't Just Memorize the Noise
Standard cross-entropy loss has a specific failure mode on noisy labels: deep networks are expressive enough to eventually memorize even randomly assigned labels (Zhang et al., "Understanding Deep Learning Requires Rethinking Generalization," ICLR 2017), and vanilla training doesn't distinguish between fitting real signal and fitting noise. Given enough epochs, it will do both.
A few training-time mitigations that specifically target this:
Swap the loss function. Generalized Cross Entropy (Zhang and Sabuncu, NeurIPS 2018) and symmetric cross-entropy variants are built to be less sensitive to a subset of mislabeled examples than standard cross-entropy, at a small cost to convergence speed on clean data.
Use small-loss selection. Co-teaching (Han et al., NeurIPS 2018) trains two networks simultaneously, where each network selects the lowest-loss (likely-clean) examples in a batch to teach the other. It works because mislabeled examples tend to produce a higher loss than correctly labeled ones early in training, before the network has had a chance to memorize them.
Feed the model soft labels, not hard ones. If your weak supervision pipeline already outputs a probabilistic label (a 0.7/0.3 split rather than a forced single class), let the downstream model train on that directly instead of collapsing it to a hard label first. You throw away real uncertainty information the moment you round it off.
Apply label smoothing as a cheap baseline. It's not a substitute for the above, but replacing hard 0/1 targets with a slightly softened distribution costs nothing to implement and reduces how hard the model commits to any single noisy label.
How Do You Evaluate a Model When You Don't Fully Trust the Test Labels Either?
Carve out a small, expensively hand-verified gold set and keep it completely separate from the large noisy or weakly supervised training set. This is the standard pattern in weak supervision pipelines: cheap labels at scale for training, an expensive clean set reserved only for measuring how well it worked.
Don't reuse labels from the same noisy sources for evaluation, even after combining them probabilistically. Doing so will overstate performance specifically on the failure modes your noise sources share, which is exactly the blind spot you need visibility into.
Track accuracy per labeling source too, not just the model's aggregate score. When performance degrades later, knowing which upstream source's noise pattern shifted is a much faster diagnosis than re-auditing the entire training set from scratch.
A Practical Workflow for Training on Noisy, Weakly Supervised Labels
Inventory every label source you have, and write down, even roughly, how each one tends to fail.
Build a small, hand-audited gold set (hundreds of examples, not millions) reserved exclusively for evaluation.
Combine weak sources with a probabilistic label model instead of a hard majority vote.
Run a confident-learning-style noise detection pass on the combined training set before you train anything on it.
Train with a noise-robust loss or small-loss selection method, not vanilla cross-entropy, especially if you suspect noise is concentrated rather than uniform.
Evaluate exclusively against the gold set, and re-audit that gold set periodically as your definition of "correct" shifts with the product.
The Takeaway
Label quality isn't a preprocessing checkbox you tick once before the real modeling work starts. Treat it as an ongoing part of the modeling problem itself: estimate the noise, combine sources deliberately, train with methods that account for it, and evaluate against labels you actually trust. Models built on that assumption degrade gracefully; models built on the assumption that labels are ground truth just fail quietly until someone notices in production.
FAQ
What's the difference between label noise and weak supervision?
Label noise describes a property of a dataset: some labels are wrong, no matter how they were produced. Weak supervision describes how the labels were produced in the first place, generated from multiple imperfect sources like heuristics or existing models instead of hand annotation. Weakly supervised labels are noisy by construction, but hand-annotated labels can be just as noisy.
How much label noise can a deep learning model actually tolerate?
There's no fixed threshold, because the pattern of noise matters more than the rate. Uniform random noise spread evenly across classes is far easier for a model to train through than noise concentrated on a specific pair of easily-confused classes, which a network will happily learn as if it were real signal.
Do I need clean labels for my validation and test sets?
Yes. Train on the large noisy or weakly supervised set, but reserve a small, hand-checked gold set purely for evaluation. Scoring a model against the same noisy labels it trained on hides exactly the errors you most need to catch.
Is weak supervision the same as semi-supervised learning?
No, and the two get conflated often. Semi-supervised learning uses a small pool of clean labels plus a large pool of unlabeled data. Weak supervision generates labels for the entire dataset from imperfect, programmatic, or heuristic sources, so there's no unlabeled data left over.
What's the fastest way to start auditing labels I already have?
Train a quick baseline model with cross-validation, then flag examples where the model's out-of-fold prediction confidently disagrees with the given label. It won't catch every error, but it turns an open-ended manual review problem into a short, ranked list.