I’ve written before about the elusive “Semantic Layer,” that mythical construct every data team eventually talks about building. It’s the idea of pulling all business logic, calculations, and definitions into a single place so everyone agrees on what the numbers actually mean. Anyone who has worked in data long enough knows the pain this is trying to solve. Logic gets scattered across pipelines, dashboards, notebooks, and random scripts, and before long, no one can explain why two reports show different answers for the same metric.
Despite decades of industry experience, we still struggle with this. Data teams continue to fight their way through repos, documentation, and tribal knowledge just to understand how a number is calculated. It’s not that we don’t know better—it’s that systems naturally drift toward complexity and inconsistency over time.
I recently spent some time poking around Agent Bricks from Databricks, and it’s a pretty good representation of where we are in the AI cycle right now. Whether you’re skeptical or all-in, it’s hard to ignore the fact that agent-based systems are no longer theoretical. They’re here, and they’re being used to automate real workflows.
I’m not particularly interested in the hype, though. What matters is whether something useful can actually be built. We’ve reached a point where building AI systems is no longer technically difficult. Tools like Claude can generate large portions of the code, frameworks are maturing quickly, and platforms like Databricks are packaging everything into approachable interfaces. The barrier to entry has dropped significantly. The real question now is not whether you can build an agent, but whether you can build one that delivers value.
Polars’ Streaming Engine Is a Bigger Deal Than People Realize
If there’s one tool that still doesn’t get enough attention in our strange little data world, it’s Polars. It gets some love, sure, but not nearly what it deserves. I’ve been using it on and off since around 2022, and it was actually the first tool I used to replace a Databricks Spark job in production. That alone earns it a permanent spot in my stack.
I’m still very much a believer in what I’ve been calling the “Single Node Rebellion,” especially as teams start taking a harder look at data platform costs. In a world where compute bills can spiral quickly, tools like Polars feel less like a niche option and more like an obvious direction forward. There’s no real reason it shouldn’t play a major role in the modern data stack.
Every once in a while, I catch myself wondering if I should sell everything, move to a cabin in the woods, and raise goats… or keep doing this Data Engineering thing a little longer. The problem with sticking around is that you start to recognize patterns, and lately it feels like we’re watching history repeat itself—just with better branding and cloud infrastructure.
With things like stored procedures showing up in Spark and Databricks, I probably should have seen this coming. Transactions were always going to follow. That’s on me.
