, ,

Polars’ Streaming Engine Is a Bigger Deal Than People Realize

Polars’ Streaming Engine Is a Bigger Deal Than People Realize

If there’s one tool that still doesn’t get enough attention in our strange little data world, it’s Polars. It gets some love, sure, but not nearly what it deserves. I’ve been using it on and off since around 2022, and it was actually the first tool I used to replace a Databricks Spark job in production. That alone earns it a permanent spot in my stack.

I’m still very much a believer in what I’ve been calling the “Single Node Rebellion,” especially as teams start taking a harder look at data platform costs. In a world where compute bills can spiral quickly, tools like Polars feel less like a niche option and more like an obvious direction forward. There’s no real reason it shouldn’t play a major role in the modern data stack.

That said, Polars could learn a thing or two from DuckDB and MotherDuck when it comes to community and messaging. Those folks have done an excellent job making their tools approachable and widely adopted. Meanwhile, Polars sometimes feels like it’s surrounded by gatekeepers rather than advocates, which doesn’t help adoption.

Still, curiosity always wins out for me. I recently asked how many people are actually using Polars in production, and while some are, there’s clearly a large group that either hasn’t tried it or hasn’t made the jump yet. That’s part of the reason for revisiting it here, especially as more people lean on tools like Claude to generate pipelines and workflows. If you’re describing a data pipeline to an AI, Polars should absolutely be part of that conversation.

  • One thing that tends to confuse people is that Polars operates in two execution modes: eager and lazy. In eager mode, every operation is executed immediately, returning intermediate results at each step. In lazy mode, operations are deferred and optimized as a whole before execution. The difference matters because lazy execution allows Polars to apply optimizations like predicate and projection pushdown, which can significantly reduce memory usage and CPU load. As the Polars documentation puts it, eager execution can be wasteful because it processes data that may not ultimately be needed, while lazy execution gives the query planner room to optimize.

In practice, lazy mode is what you want most of the time. You’ll typically see patterns like scan_* for reading data and a final collect() or sink_* call to trigger execution. Eager mode, by contrast, looks more like traditional APIs where data is loaded and processed immediately. The important thing here is not to confuse lazy execution with the newer streaming engine, which builds on top of lazy execution but introduces a different execution model altogether.

The newer streaming engine is where things get interesting. There have been hints that this could become the default execution model in the future, which would make sense given the performance characteristics. It’s essentially a more efficient way to process data without loading everything into memory at once, and early benchmarks suggest it delivers substantial improvements.

I ran a couple of quick tests to see how it behaves in practice. First, I used a relatively small dataset—the Divvy bike trip data—and ran a simple aggregation using standard Polars. Reading a few million records, grouping by date, and writing the results out took under a second. That alone is impressive, but switching to the streaming engine by enabling it and using scan_csv and sink_csv cut the runtime in half. The exact same logic, just a different execution model, and noticeably better performance.

We will do the Eagar one first.

import time
import polars as pl

start = time.perf_counter()

df = pl.read_csv('data_Q4_2025/*.csv')

failures_per_day = (
    df
    .filter(pl.col("failure") == 1)
    .group_by("date")
    .agg(pl.len().alias("failure_count"))
    .sort("date")
)

failures_per_day.write_csv("failures_per_day_eager.csv")

elapsed = time.perf_counter() - start
print(f"Done in {elapsed:.2f}s — written to failures_per_day_eager.csv")

Now our Lazy and Streaming engine mode. Go for the moon baby.

import time
import polars as pl

pl.Config.set_engine_affinity("streaming")

start = time.perf_counter()

df = pl.scan_csv('data_Q4_2025/*.csv')

failures_per_day = (
    df
    .filter(pl.col("failure") == 1)
    .group_by("date")
    .agg(pl.len().alias("failure_count"))
    .sort("date")
)

failures_per_day.sink_csv("failures_per_day_lazy.csv")

elapsed = time.perf_counter() - start
print(f"Done in {elapsed:.2f}s — written to failures_per_day_lazy.csv")

And the performance results are even more pronounced at this larger scale.

  - failures_per_day_lazy.py — streaming, 6s                                                                                                                        
  - failures_per_day_eager.py — eager, 27.65s
  • The difference becomes more obvious as the data size increases. Using a larger dataset, about 12GB of Backblaze hard drive data, I compared eager execution with lazy streaming. The eager version took around 27 seconds, while the streaming version completed in about 6 seconds. That’s not a marginal improvement; it’s a completely different performance profile.

At that point, it’s hard not to extrapolate what this means for real-world workloads. As data sizes grow, the benefits of streaming execution should become even more pronounced. This is exactly the kind of capability that makes single-node processing viable for workloads that would traditionally require distributed systems.

That’s really the bigger picture here. Polars isn’t just fast; it’s shifting the conversation around what kind of infrastructure is actually necessary. If you can process large datasets efficiently on a single machine, a lot of the assumptions about clusters, orchestration, and cost start to fall apart. That doesn’t mean distributed systems are going away, but it does mean they’re no longer the default answer to every problem.

The future for tools like Polars looks strong, especially as teams continue to push for simpler, more cost-effective architectures. If the streaming engine becomes the default and adoption continues to grow, it’s not hard to imagine Polars becoming a core piece of many modern data platforms. And if you’re not at least experimenting with it today, you’re probably going to be catching up later.