Cluster Fatigue. Polars and PyArrow to Postgres and Apache Iceberg (streaming mode)

I’ve been working lately, on moving expensive distributed compute jobs (that don’t need to be distributed) from Spark, to other single node tools and frameworks. To be honest, there are reasons that Data Platforms might pick Spark, for example, and just keep everything on Spark, even if it doesn’t need Spark.

Yes, it costs more. Yes, it makes things simple.

We try to balance the need for simplicity, fast development iterations, low mental ramps to get up to speed, reliability. It’s a difficult line to walk for some of us. Things change, we are asked to save costs … so I find myself kicking Spark to the curb in favor of Polars.

Yes it increases the lines of code written, yes there is more mental burden, but at the same time, costs come down. We get to do something new in production that is fun, keeps interest running.

Large datasets, small compute, streaming needs.

One of the struggles I have today, is the actual reality of using not tiny datasets, with smallish compute, in a production setting that requires certain integrations be done well. Most of all, providing the ability to keep memory pressure low, while keeping throughput high, and integrating with the modern Lake House architecture.

What do I mean?

It’s all fine and dandy to someone to write HelloWorld() in DuckDB or Polars, showing how you can read a CSV file in S3 and write out to a Parquet file(s) without OOM on a machine. Whatever. Duh, we aren’t that gullible, at least some of us aren’t.

What, you’re still running a Data Lake? That’s not a good luck or a good idea. The Lake House is here to stay, Apache Iceberg and Delta Lake are the baseline for modern storage in the Data Platform. I was recently working on this issue.

Delta Lake –> Transformation –> Postgres.

Fairly normal workflow I would say. Used to be done in PySpark … because … why not? The thing is that Spark has been around forever and made such things trivial. I mean is it worth the cost of the those DBU’s to Databricks? Maybe. Depends on your point of view.

Things are moving ahead.

I’m glad that the frameworks we use are catching up with the real world. No, we are all not using vanilla parquet files in S3. No we are not using unmanaged open-source Delta Lake tables. We are using Unity Catalog tables. We are using RDS. We are using Apache Airflow nodes that are not that big.

We just got too git ‘er done.

And so, we ask Polars to save the Spark day. Cluster fatigue is real, we need single node glory.

Unity Catalog Delta table? No problem my friend.

Easy enough, we have a lazy dataframe of Unity Catalog Delta table, know we can do our transformations and proceed as normal, no worries about blowing up memory with the streaming, memory friendly, lazyframe.

< insert all the transformations you want on a lazy frame >

Of course, Postgres is a supported Sink for every single tool under the sun.

Kind Polars, so nice and wonderful. Like a fall day with soft sunlight and a breeze on your face. That’s what it’s like to not blow up memory and compute with Polars.

.collect(engine=”streaming”)

Doesn’t get much better. Just like that … Spark gone, Databricks gone, DBU’s gone. Almost brings a tear to the ole’ eye ya know? Must be getting soft in my old age.

I did mention something about PyArrow in the title didn’t I? Indeed I did.

Twas’ the night before last and I was working on one of those little side projects that always keep you busy but never makes you rich. Classic. This time I did have a large 1TB batch of parquet files in S3 that needed to be streamed in Apache Iceberg (in s3), I figured PyArrow might come in handy this time.

Here we use PyArrow Datasets to be able to get 1TB of parquet data in S3 into a usable state. We then create an Apache Iceberg table (using boring catalog), with the schema thereof.

Simple.

Next, we use PyArrow to slow batch those records and push them into Apache Iceberg.

Mind you, Sunny Jim, this was for 1TB of data.

Don’t lie to me. I will tell your Mom … and Grandma. You would have just reached for Spark wouldn’t ya? Say it. Say it louder. I got you pegged.

Maybe someday you will end up getting Cluster Fatigue like me. Sure it’s a lot more code and more complex than PySpark. It’s like ripping the bandaid off your arm that’s covered in hair. It hurts the first time.

But to see it work my friend.

Those logs pumping across the terminal. Those bytes of 1TB screaming from one S3 bucket of parquet data over into an Iceberg table in S3. It makes a hobbit wonder. 

Have we been lied too? Are we like Neo stuck in the Matrix of the Modern Data Stack? Like that infamous man who said “What is truth?”

We lived in a world where we hear the words … “Terabyte”, or “Iceberg” or “Lake House” and we think Cluster. We … must … have … Cluster … give … me … cluster. DBUs start piling up. We smile, our boss smiles, Databricks smiles. Just a little pipeline, another one of a million pulsing through the network today. What’s just one more?

But does it need to be so?

Cluster Fatigue is real. I see it boiling and bubbling out there in the world, little splashes popping out of the cauldron of the Lake House splattering into spots and pipelines like mine. Just a little uv add polars is all it took. The dust cleared, the scales fell from eyes.

Born-again into a new world of possibilities, we have our new streaming messiah to worship. Oh what does the future hold? Is it full of DuckDB, Polars, Daft, and the like? Can they survive the crush of Snowflake and Databricks bearing down on them from upon high? Will the teaming mindless masses of Data Engineers throw off the shackles of their former idols, and pip install their way to freedom?

Time will tell.