DevsUnite Logo
DevsUnite

Feature Engineering at Scale Without Losing Iteration Speed

TD
Team DevsUnite
feature-engineering
9 min read
Aug 14, 2026
Feature Engineering at Scale Without Losing Iteration Speed

Feature engineering at scale doesn't get slow because of the feature engineering. It gets slow because every idea gets tested against the full billion-row dataset before anyone knows if it's any good. Decouple iteration from production volume instead: prototype and select features on a cached, representative sample, and only run the full job on features that already passed selection and leakage checks.

Why feature engineering at scale slows down iteration

The math is multiplicative, not additive. Thousands of raw candidate features, each requiring a join against one or more source tables, run against billions of rows, and a naive workflow reruns that whole chain from raw data every time someone tweaks a window size or adds a new signal.

A single feature idea might be cheap. Testing two hundred of them against the full table, one at a time, in a notebook that recomputes everything from scratch each run, is not. That's the actual bottleneck teams hit: not the eventual production job, but the hundreds of throwaway iterations that happen before a feature earns a permanent place in the pipeline.

Blog image

The fix isn't a faster cluster. A bigger Spark cluster still charges you full-table cost for every experiment, it just charges you less time per experiment. The fix is making most experiments never touch the full table at all.

How do you keep the feedback loop fast when the data doesn't fit in memory?

You don't iterate on data that doesn't fit in memory. You iterate on a sample that does, and you make that sample honest enough that what works on it also works at full scale.

A representative sample for feature work needs two properties a naive random sample usually breaks. It needs the full time range your production data spans, not just a recent window, so date-based and seasonal features behave the same way they will in production. And it needs to preserve whatever field skews your data, user tier, region, device type, so a feature that only matters for 2% of traffic doesn't silently vanish from a 1% random sample.

A stratified, date-covering sample that fits on a single machine turns a query that takes minutes on the full table into one that takes seconds. Polars and DuckDB are both built for exactly this: in-process, columnar, vectorized engines that skip the cluster round-trip entirely for anything that fits on one box.

import polars as pl

sample = (
    pl.scan_parquet("data/events/*.parquet")
    .filter(pl.col("event_date") >= "2025-01-01")
    .filter((pl.col("user_id").hash() % 100) < 5)  # ~5% sample, stable per user
    .with_columns(
        pl.col("amount")
        .rolling_mean(window_size=30)
        .over("user_id")
        .alias("amount_30d_avg"),
    )
    .collect()
)

That query runs in seconds against a cached local sample and stays lazy until .collect(), so Polars can push the date filter down before it ever reads a full column into memory. Prototype every feature idea here first. The full-scale job only needs to run once a feature has already proven itself.

Select features before you compute them at scale, not after

With thousands of raw candidates, the expensive mistake isn't picking a bad feature. It's materializing all of them at full scale before finding out which ones are bad.

Screen on the sample with cheap, fast methods before anything touches the billion-row table: correlation against the target, mutual information for non-linear relationships, or a quick tree-based importance pass from a model trained on the sample alone. None of these need to be your final model or your final answer. They need to cut a list of two thousand candidates down to the thirty or so worth the cost of a full-scale join.

This ordering matters more at billion-row scale than at any smaller scale, because the cost of computing a feature nobody keeps scales with the table, not with how good the idea was.

Stop recomputing what hasn't changed

The other iteration killer is recomputing history that hasn't changed. A pipeline that reprocesses all billion rows every time new data lands, or every time a downstream feature definition changes, pays the full cost on every run regardless of how small the actual change was.

Partition raw and intermediate data by the field you'll filter on most, usually date, and store it in a columnar format like Parquet. Then scope every recompute to the partitions that actually changed instead of the whole table.

-- Recompute only today's partition, not the full history
COPY (
    SELECT
        user_id,
        event_date,
        amount,
        avg(amount) OVER (
            PARTITION BY user_id
            ORDER BY event_date
            ROWS BETWEEN 29 PRECEDING AND CURRENT ROW
        ) AS amount_30d_avg
    FROM read_parquet('data/events/event_date=2026-08-14/*.parquet')
) TO 'data/features/amount_30d_avg/event_date=2026-08-14.parquet' (FORMAT PARQUET);

Splitting the pipeline into a stable "raw signal extraction" layer and a cheaper, faster-changing "transformation" layer helps too. Extraction, the expensive joins against source-of-truth tables, changes rarely. Transformation, the rolling windows and ratios and bucketing on top of already-extracted signals, changes constantly during iteration. Cache the first layer aggressively and let the second layer stay cheap to rerun.

The leakage trap that only shows up at full scale

A feature that looks great on your sample and quietly leaks the future is the most expensive kind of bug in this workflow, because it's invisible until it either fails in production or, worse, doesn't fail and just makes your offline metrics lie to you.

The cause is almost always a join that isn't point-in-time correct: a feature computed using data that wouldn't have existed yet at the label's timestamp. This risk scales with feature count. With thousands of raw features pulled from dozens of source tables, each with its own update cadence and its own definition of "as of when," the odds that at least one join uses future information climb fast.

Blog image

The fix has to live in the sample, not just in the full-scale job. Build your iteration sample with a hard point-in-time cutoff per row, matching whatever timestamp your label uses, not a random cross-section of the table. Feature stores like Feast formalize this as a point-in-time join for exactly that reason: it's easy to get right in theory and easy to get wrong in a rushed feature-engineering loop. It's the same failure mode production feature platforms like Uber's Michelangelo were built around handling correctly at scale.

Putting it together

  1. Build one point-in-time-correct, stratified sample covering the full date range and preserving your data's skewed categories, cached locally.

  2. Prototype and screen every feature idea against that sample first, using a fast columnar engine, not the production table.

  3. Rank candidates with cheap selection methods (correlation, mutual information, tree importance) before any of them touch full-scale compute.

  4. Materialize only the shortlist at full scale, using the same transformation code path the sample used, not a rewritten version.

  5. Partition full-scale storage and recompute incrementally, scoping reruns to changed partitions instead of the whole table.

  6. Re-validate shortlisted features against production-scale data before shipping, since some skew or leakage issues only surface once you're past the sample.

FAQ

What's the fastest way to prototype ML features on billion-row datasets?

Don't prototype against the full table. Pull a representative, cached sample that fits on one machine, iterate with a fast columnar engine like Polars or DuckDB, and only push a feature into the full-scale job once it has already earned its place through selection and leakage checks.

How big should the iteration sample be?

Big enough to preserve your rare categories and full time range, small enough to fit in memory and return a query in seconds. A fixed row count matters less than stratifying by date and by whatever field skews your data, like user tier, region, or device. Resample if a feature behaves differently on the full run than it did on the sample.

Should feature selection happen before or after the full-scale computation?

Before. Screen thousands of raw feature candidates on the sample with cheap methods, like correlation, mutual information, or a quick tree-based importance pass, and only materialize the shortlist at full scale. Running a billion-row join for a feature you're going to drop next week wastes compute and wastes your iteration time.

Do I need Spark or a distributed engine for billion-row feature engineering?

Only for the final full-scale materialization, not for daily iteration. Single-node columnar engines handle far more data than they used to. Keep Spark, Dask, or your warehouse's compute for runs that need to touch every row, and reserve them for finalized features, not exploratory ones.

How do you stop the iteration pipeline from drifting away from the production pipeline?

Write the transformation logic once, as a shared function or SQL macro, and call it from both the sample-scale dev job and the full-scale production job with only the input source swapped. Maintaining two separate implementations, a fast dev version and a real production version, is exactly how training and serving quietly diverge.

The one thing to actually do this week

Build the point-in-time-correct sample once, wire it into a shared transformation layer that both your dev loop and your production job call, and every feature idea you test after that costs seconds instead of a cluster job. The same tension between iteration speed and production scale shows up again once training starts, where the distributed training strategy you pick trades the same kind of speed against the same kind of cost, and again at deployment, where model compression decisions revisit it one more time.

Sources