At some point, every data engineer has to confront a slightly uncomfortable truth about how they work: the tools they use are not just tools but habits, and those habits quietly shape how they think, how they build, and ultimately what kind of systems they produce. That realization tends to hit hardest when someone points directly at a pattern and gives it a name, which is exactly what happened when I started talking about notebook engineering and received a very direct question in response, which was essentially this: if notebooks are the problem, then what does it actually look like to break free from them?
That question is much harder to answer than simply criticizing the behavior, because it forces you to move beyond pointing out flaws and into explaining what better looks like in practice.
I finally hit that point that every engineer eventually reaches with a tool they once loved, that moment where frustration quietly builds over time and then suddenly flips into a decision, not because of one catastrophic failure but because of the accumulation of too many small ones. That was me with Polars. After years of using it, promoting it, and even putting it into production workloads early on, I reached the point where I removed it entirely from a set of critical pipelines and replaced it with DuckDB without much hesitation.
Before anyone jumps in to defend Polars or call this an overreaction, it is worth understanding that this was not a spur-of-the-moment decision. This was the result of years of real-world usage, repeated friction, and a growing realization that what matters most in a production data platform is not theoretical performance or benchmark wins, but consistency, predictability, and trust that the tool behaves the same way every time it runs.
I’ve written before about the elusive “Semantic Layer,” that mythical construct every data team eventually talks about building. It’s the idea of pulling all business logic, calculations, and definitions into a single place so everyone agrees on what the numbers actually mean. Anyone who has worked in data long enough knows the pain this is trying to solve. Logic gets scattered across pipelines, dashboards, notebooks, and random scripts, and before long, no one can explain why two reports show different answers for the same metric.
Despite decades of industry experience, we still struggle with this. Data teams continue to fight their way through repos, documentation, and tribal knowledge just to understand how a number is calculated. It’s not that we don’t know better—it’s that systems naturally drift toward complexity and inconsistency over time.
Polars’ Streaming Engine Is a Bigger Deal Than People Realize
If there’s one tool that still doesn’t get enough attention in our strange little data world, it’s Polars. It gets some love, sure, but not nearly what it deserves. I’ve been using it on and off since around 2022, and it was actually the first tool I used to replace a Databricks Spark job in production. That alone earns it a permanent spot in my stack.
I’m still very much a believer in what I’ve been calling the “Single Node Rebellion,” especially as teams start taking a harder look at data platform costs. In a world where compute bills can spiral quickly, tools like Polars feel less like a niche option and more like an obvious direction forward. There’s no real reason it shouldn’t play a major role in the modern data stack.
Every once in a while, I catch myself wondering if I should sell everything, move to a cabin in the woods, and raise goats… or keep doing this Data Engineering thing a little longer. The problem with sticking around is that you start to recognize patterns, and lately it feels like we’re watching history repeat itself—just with better branding and cloud infrastructure.
With things like stored procedures showing up in Spark and Databricks, I probably should have seen this coming. Transactions were always going to follow. That’s on me.