I’ve been working lately, on moving expensive distributed compute jobs (that don’t need to be distributed) from Spark, to other single node tools and frameworks. To be honest, there are reasons that Data Platforms might pick Spark, for example, and just keep everything on Spark, even if it doesn’t need Spark.
Yes, it costs more. Yes, it makes things simple.
We try to balance the need for simplicity, fast development iterations, low mental ramps to get up to speed, reliability. It’s a difficult line to walk for some of us. Things change, we are asked to save costs … so I find myself kicking Spark to the curb in favor of Polars.
Yes it increases the lines of code written, yes there is more mental burden, but at the same time, costs come down. We get to do something new in production that is fun, keeps interest running.
Large datasets, small compute, streaming needs.
One of the struggles I have today, is the actual reality of using not tiny datasets, with smallish compute, in a production setting that requires certain integrations be done well. Most of all, providing the ability to keep memory pressure low, while keeping throughput high, and integrating with the modern Lake House architecture.
Need a gentle introduction to Databricks Asset Bundles, what they are, how to use them, why to use them? Look no further you hobbit.