DevsUnite Logo
DevsUnite

How to Make AI Models Smaller, Faster, and Cheaper?

TD
Team DevsUnite
model-compression
10 min read
Aug 11, 2026
How to Make AI Models Smaller, Faster, and Cheaper?

Three model compression techniques cover most production cases: quantization, pruning, and distillation. Quantization cuts memory and bandwidth with the least retraining effort. Pruning removes weights or structures outright, but usually needs fine-tuning to recover accuracy. Distillation trains a smaller model from scratch to mimic the big one. Pick based on what's actually slow or expensive, then validate every step against your real eval set, not a benchmark someone else published.

Memory, latency, and cost are three different problems

"The model is too big for production" usually means one of three distinct things, and each has a different fix. Memory footprint determines whether the model fits on the device or GPU at all. Latency determines whether a single inference finishes fast enough for the product. Serving cost determines how many GPUs or CPU cores you need per request at your actual traffic volume.

A 7B-parameter model in FP16 needs roughly 14GB just to hold the weights, before you count activations, KV cache, or batching overhead. Drop to INT8 and that's about 7GB; INT4 gets you to roughly 3.5GB. That's a memory win from quantization alone, and it's often enough to move a model from "needs an A100" to "runs on a single consumer GPU."

Latency is a different axis. A model that fits in memory can still be too slow if the bottleneck is compute-bound matrix multiplies or an attention mechanism that scales quadratically with sequence length. Structured pruning and distillation attack this directly by reducing the actual amount of computation, not just the storage format.

Before picking a technique, profile which of the three you're actually solving for. Compression techniques aimed at the wrong bottleneck will "work" in the sense that the model gets smaller, and do nothing for the metric your product actually cares about.

Quantization: the first thing to try

Quantization represents weights (and sometimes activations) in fewer bits: FP32 down to FP16/BF16, INT8, or INT4. Each step down roughly halves memory and bandwidth, and on hardware with native low-precision kernels, it speeds up compute too, since moving fewer bits per operation is the actual bottleneck on most modern accelerators.

There are two ways to get there. Post-training quantization (PTQ) takes a trained model and quantizes it afterward, using a small calibration dataset to pick good scale factors. It's cheap, usually a few minutes to hours of compute, and doesn't touch the original training pipeline. Quantization-aware training (QAT) simulates quantization noise during training or fine-tuning, so the model's weights adapt to it. QAT costs more but holds up better at INT4 and below, where PTQ starts to visibly degrade output quality.

For large language models specifically, two PTQ methods dominate because naive rounding-based quantization falls apart on transformer weight distributions. GPTQ quantizes layer by layer using approximate second-order (Hessian) information to correct for the error each rounding decision introduces, getting usable INT4 weights with a small perplexity hit. AWQ takes a different angle: it identifies the small percentage of weight channels that matter most based on activation magnitude, not weight magnitude, and protects exactly those from quantization error instead of treating every channel equally.

Here's dynamic quantization on a standard PyTorch model, no calibration data required:

import torch

model_fp32 = MyModel()
model_fp32.eval()

model_int8 = torch.quantization.quantize_dynamic(
    model_fp32,
    {torch.nn.Linear},
    dtype=torch.qint8,
)

And loading an LLM in 8-bit through Hugging Face's bitsandbytes integration:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "model-name",
    quantization_config=quant_config,
    device_map="auto",
)

Quantization is the first move because it's the cheapest to try and reverse. If PTQ at INT8 doesn't meet your bar, that's a signal to look at QAT, pruning, or distillation, not a reason to push straight to INT4 and hope.

Pruning: cut weights that don't earn their place

Pruning removes weights, or entire structures, that contribute the least to the model's output. The simplest version, magnitude pruning, zeroes out weights closest to zero, on the theory that small weights contribute little to the forward pass. It's a reasonable heuristic and cheap to apply, but it needs a fine-tuning pass afterward to recover the accuracy the removed weights were quietly contributing.

The distinction that matters for production is unstructured vs. structured pruning. Unstructured pruning zeroes individual weights anywhere in a matrix, which gives you the best accuracy-to-sparsity ratio on paper, but a dense hardware matrix multiply doesn't skip zeros unless the runtime has explicit sparse-kernel support. Without that support, an unstructured-pruned model is smaller to store and identical in speed to the dense one.

Structured pruning removes whole channels, attention heads, or layers, producing a genuinely smaller dense model that runs faster on any hardware, no special kernels required. It's less flexible and usually loses more accuracy per unit of size reduction than unstructured pruning, but it's the version that reliably translates into a latency win in production.

Pruning almost always needs a retraining or fine-tuning step afterward. Treat "prune once, deploy immediately" as a red flag; the standard workflow is prune, fine-tune to recover, measure, and repeat until you hit your sparsity target or your accuracy floor, whichever comes first.

Knowledge distillation: train a smaller model to imitate the big one

Distillation trains a smaller "student" model to match a larger "teacher" model's output distribution, not just its hard labels. The student learns from the teacher's full softmax output (often temperature-scaled to soften the distribution), which carries more signal than a one-hot label: it tells the student not just the right answer but how confident the teacher was and what it considered plausible alternatives.

DistilBERT is the reference example: distilled from BERT, it cuts parameter count by about 40% and runs about 60% faster, while retaining around 97% of BERT's language understanding performance on the GLUE benchmark, according to the original paper from Hugging Face researchers. That's the trade-off curve distillation aims for: most of the capability, a fraction of the cost.

Distillation is the most expensive of the three techniques to set up, since you need teacher inference during training and a student architecture to design, but it's also the one most likely to produce a model that's actually fast on commodity hardware, because you're not fighting kernel support for exotic sparsity or bit widths. It's the right call when quantization and pruning of the original architecture still don't hit your target, or when you want a fundamentally smaller model to build a product around from day one.

Blog image

How much accuracy will you actually lose?

There's no single number, because it depends on the technique, the bit width or sparsity level, and how well-conditioned your model and data are. But some patterns hold consistently across model families:

  • INT8 post-training quantization on a well-trained CNN or transformer typically costs well under 1% accuracy on standard benchmarks.

  • INT4 weight-only quantization on LLMs, done with GPTQ or AWQ rather than naive rounding, commonly holds perplexity increases to a small, often acceptable margin, but naive INT4 rounding can be dramatically worse.

  • Structured pruning above roughly 30-50% sparsity, without fine-tuning, usually produces a visible accuracy cliff, not a smooth degradation. Below that threshold, with fine-tuning, loss is often modest.

  • Distillation's accuracy retention depends heavily on the capacity gap between teacher and student. A student half the teacher's size retains more than one a tenth the size.

Every one of these numbers is model- and dataset-dependent. Treat published figures as a rough prior, not a guarantee, and always measure on your own eval set before shipping. A model that holds up on ImageNet or GLUE can still degrade in ways that only show up on your specific data distribution.

Putting model compression techniques into a production pipeline

  1. Profile first. Determine whether you're memory-bound, compute-bound, or cost-bound before choosing a technique. Don't compress blind.

  2. Establish a real eval set and baseline. Use production-representative data, not just the original training benchmark, and record baseline accuracy/latency/memory before touching anything.

  3. Try post-training INT8 quantization first. It's cheap, reversible, and often gets you most of the win with none of the retraining cost.

  4. If you need more, add structured pruning with fine-tuning. Target a sparsity level, fine-tune to recover accuracy, and re-measure against your baseline eval set, not just training loss.

  5. If the architecture itself is the bottleneck, consider distillation. This is the point to consider a smaller architecture entirely, rather than continuing to compress the original one.

  6. Push to INT4 or below only with QAT or activation-aware PTQ (GPTQ/AWQ). Naive low-bit rounding on an already-compressed model tends to be where accuracy quietly falls off a cliff.

  7. Validate against production traffic patterns before rollout, including edge cases and out-of-distribution inputs your offline eval set might not cover, and keep the uncompressed model available as a fallback during rollout.

Blog image

Confirm your target runtime actually supports the technique you picked before you invest in it. A sparsity pattern or bit width that has no corresponding kernel in your inference engine, whether that's ONNX Runtime, TensorRT, Core ML, or TFLite, will not translate into a real speedup, no matter how good the paper's numbers look.

FAQ

Should I quantize or prune a model first?

Quantize first. It's the highest-ROI move, often needs no retraining, and modern INT8 kernels give near-free speedups on both CPU and GPU. Reach for pruning when quantization alone doesn't clear your latency or memory bar and you're willing to spend retraining time to get further.

Does INT8 quantization always hurt accuracy?

Not necessarily. Many CNNs and transformers lose well under 1% accuracy from post-training INT8 quantization. Below INT8, or with badly calibrated activations, degradation becomes visible unless you use quantization-aware training or an activation-aware method like AWQ or GPTQ.

What's the actual difference between pruning and quantization?

Pruning removes weights or whole structures, making the network sparser or literally smaller. Quantization keeps every weight but represents each one in fewer bits, shrinking size and memory bandwidth without touching the network's structure. They solve overlapping but different problems, and most aggressive production pipelines use both.

Do I need special hardware to benefit from pruning?

Only for unstructured pruning. Structured pruning, removing whole channels, attention heads, or layers, gives you a genuinely smaller dense model that runs faster on any hardware. Unstructured pruning zeroes individual weights, and those zeros only translate into speed on runtimes with sparse-kernel support; otherwise you've only saved compressed storage, not compute.

Can I combine quantization, pruning, and distillation?

Yes, and pipelines that need aggressive compression usually do all three in sequence: distill to a smaller architecture, prune the distilled model, then quantize what's left. Validate against your eval set after each step, not just at the end, because the accuracy loss from each technique compounds.

The one thing to actually do this week

Before you touch a compression technique, build the eval set you'll measure every change against, one that reflects production traffic, not just the original training benchmark. Every technique in this post can look great on paper and still fail silently on the inputs your users actually send; the eval set is what catches that before your users do. This kind of production-vs-offline gap is exactly the judgment call interviewers probe for in ML engineer interview questions about training and efficiency, and it's the same instinct that separates a compression pipeline that ships from one that quietly regresses.

Sources