How Do You Compress a Model Without Losing Accuracy?

Shrinking a model for production means applying model compression techniques in a specific order: prune the architecture, fine-tune to recover accuracy, then quantize the result to whatever precision your serving hardware supports. Skip the fine-tuning step, or quantize before pruning, and you get a model that's smaller on disk but no faster, or faster but noticeably worse.

What model compression techniques actually do

Three distinct techniques get lumped under "model compression," and mixing them up leads to picking the wrong one for the problem you have.

Quantization reduces the numerical precision used to store and compute weights and activations, typically from 32-bit floating point (FP32) down to 16-bit float (FP16) or 8-bit integer (INT8). It shrinks memory footprint and can speed up inference, but only on hardware with native low-precision math support.

Pruning removes parameters from the network entirely: individual weights (unstructured pruning) or whole structural units like channels, filters, or attention heads (structured pruning). It shrinks the parameter count and, if done at the structural level, the compute graph itself.

Knowledge distillation trains a smaller "student" model to reproduce the output distribution of a larger "teacher" model, rather than training the student from scratch on labels alone. It's the most expensive of the three since it requires a full training run, but it can get further below the original model's size than quantization or pruning alone.

Does quantization hurt accuracy?

Yes, some amount, but how much depends on which flavor you use. Post-training quantization (PTQ) converts an already-trained model's weights (and optionally activations) to INT8 using a calibration pass over sample data, with no retraining involved. It's fast to apply, usually a few lines of code, and the accuracy cost is generally small on models with well-behaved activation ranges.

Quantization-aware training (QAT) simulates the rounding error of low precision during fine-tuning, so the model's weights adjust around it. It costs a training run instead of a calibration pass, but it typically recovers most of the accuracy PTQ gives up, which matters when your model sits close to a decision threshold.

import torch

# Post-training dynamic quantization (CPU inference)
model_fp32 = torch.load("trained_model.pt")
model_fp32.eval()

model_int8 = torch.quantization.quantize_dynamic(
    model_fp32,
    {torch.nn.Linear},   # quantize only Linear layers
    dtype=torch.qint8,
)

torch.save(model_int8.state_dict(), "model_int8.pt")

Dynamic quantization like this is the cheapest entry point: it quantizes weights ahead of time and activations on the fly at inference, with no calibration dataset required. Static PTQ and QAT need a representative calibration or fine-tuning set but give better speedups on hardware that supports full INT8 compute paths.

Structured pruning ships speedups, unstructured pruning often doesn't

This is the detail most compression write-ups skip, and it's the one that actually determines whether pruning does anything for you in production. Unstructured pruning zeroes out individual weights based on magnitude, producing a sparse weight matrix. That matrix is smaller in a mathematical sense, but most standard GPU and CPU matrix-multiply kernels aren't built to skip zeroed entries, so you often get the same latency with a model that merely looks smaller.

Structured pruning removes entire channels, filters, or attention heads, which shrinks the actual matrix dimensions the hardware operates on. That's a real, dense, smaller model, and any standard inference runtime benefits from it without special sparse-matrix support.

import torch.nn.utils.prune as prune

# Structured pruning: remove 30% of output channels by L2 norm
prune.ln_structured(
    module=model.conv1,
    name="weight",
    amount=0.3,
    n=2,
    dim=0,   # prune along the output-channel dimension
)
prune.remove(model.conv1, "weight")  # bake the mask into the weights permanently

The practical rule: pick unstructured pruning only if your deployment target has sparse-kernel support (some newer NVIDIA GPUs and specialized inference libraries do). Otherwise default to structured pruning, even though it's coarser and costs a bit more accuracy per parameter removed, because it's the version that actually turns into lower latency.

When distillation is worth the extra training run

Distillation makes sense when you need a smaller architecture, not just a smaller version of the same architecture. Pruning and quantization keep the original model's structure and shrink it in place; distillation lets you target a genuinely different, cheaper architecture (fewer layers, smaller hidden dimensions) while still inheriting most of the teacher's accuracy.

The trade-off is cost. Distillation needs a full training loop, a teacher model to query, and enough representative input data to cover the cases your production traffic actually hits. It pays off when the target deployment is aggressive enough (edge device, strict latency SLA) that pruning and quantization alone can't reach it.

The order that actually preserves accuracy

Applying these techniques in the wrong order compounds errors instead of isolating them. The sequence that holds up in practice:

Establish a baseline eval harness first. Before touching the model, lock in the exact accuracy metrics, on the exact validation slices, that you'll compare every compressed version against.
Distill, if you're targeting a smaller architecture. Do this first because it's the biggest structural change; every step after should compress the student, not the original.
Prune next, structurally by default. Fine-tune for a few epochs afterward to let the remaining weights recover the accuracy the removed structure was carrying.
Quantize last. Apply PTQ first as a cheap check; if the accuracy drop against your baseline is unacceptable, move to QAT on the already-pruned model.
Re-run the full eval harness on the final artifact, not just the aggregate accuracy, before it goes anywhere near production traffic.

Quantizing before pruning means you're pruning weights whose "importance" was measured in a precision the model won't actually run in, which produces a pruning decision that doesn't match the deployed numerics.

Build an accuracy safety net before you ship

Aggregate accuracy hides the failures that matter most. A model that loses only a sliver of accuracy overall after compression can still lose much more on a specific input slice, like a minority class, a rare intent, or a particular sensor range, while everything else stays flat.

Before shipping any compressed model, break the eval down by the same slices your monitoring already tracks in production, not just a single overall number. Compare compressed-vs-original per slice, and set an explicit regression budget per slice rather than one global tolerance. This is the same train-serve mismatch problem that shows up in other places models silently degrade after leaving the validation set: aggregate metrics look fine while a specific segment quietly breaks.

Which technique should you reach for first?

If the goal is inference speed on hardware you already know supports low-precision math, start with post-training quantization. It's the cheapest to try, requires no retraining, and tells you within an afternoon whether you even need to go further.

If quantization alone doesn't hit your latency or memory target, add structured pruning with a short fine-tuning pass. Reach for distillation only when you need a smaller architecture outright, not just a smaller version of the current one, since it's the only technique of the three that changes what the model actually is.

Sources:

FAQ

Does quantization always hurt accuracy?

Not always, but it's not free either. Post-training INT8 quantization usually costs a small, measurable amount of accuracy on well-behaved models; quantization-aware training closes most of that gap by simulating the precision loss during fine-tuning. The failure cases are models with large activation outliers or thin classification margins, where INT8 rounding can flip predictions.

What's the difference between pruning and quantization?

Quantization reduces the numerical precision of weights and activations (for example FP32 to INT8), shrinking memory and speeding up math on hardware that supports low-precision ops. Pruning removes weights or whole structures (neurons, channels, attention heads) entirely, shrinking the model's parameter count. They target different bottlenecks and are usually combined, not chosen instead of each other.

Should I prune before or after quantizing?

Prune first, fine-tune to recover accuracy, then quantize last. Quantization is comparatively cheap to redo, so you want it applied to the final, already-pruned architecture rather than baking low-precision math into a network you're about to restructure.

Does knowledge distillation require the original training data?

It requires representative input data and access to the teacher model's outputs (or the ability to generate them), but not necessarily the exact original training set or its labels. Distillation trains the student to match the teacher's output distribution, which is why it works even with unlabeled or synthetically generated inputs.

What hardware do I need to see quantization speedups?

You need a runtime and chip that actually execute low-precision math natively: x86 CPUs with VNNI/AVX-512 support, ARM chips with NEON/dot-product instructions, or GPUs with INT8 tensor cores, paired with a runtime like ONNX Runtime or TensorRT that dispatches to those kernels. Quantizing a model and running it on hardware without INT8 support gives you a smaller file and no speedup.