The future never shows up quietly. Just when you think you’ve tamed the latest “must-have” technology, a fresh acronym crashes the party. I’d barely finished wrapping my head around the Lakehouse paradigm when Databricks rolled out something new at the 2025 Data & AI Summit: Lakebase, a fully managed PostgreSQL engine built directly into the Databricks platform.
Cue the collective gasp—and the scramble—for data teams everywhere.
I’d be lying if I said a small part of me didn’t groan when I first read about SQL Scripting being released by Databricks.
Don’t get me wrong—I don’t fault Databricks for giving users what they want. After all, if you don’t feed the masses, they’ll turn on you. We data engineers are gluttons for punishment. We grumble, we fight with complexity, but somehow, we get the job done.
So why did this announcement sting a bit? Because I’ve been around long enough to remember the old days—the old days. When SQL Servers ruled the land, unleashing T-SQL, stored procedures, and SSIS packages everywhere. Some of you will remember those dark times, battling deadlocks and debugging cryptic errors in SSMS. Others were lucky enough to be born into the light.
I’ve been thinking about this for a few days now, and I still don’t know whether to cheer or groan. Some moments, I see DuckLake as a smart, much-needed evolution; other times, it feels like just another unnecessary entry in the ever-growing Lake House jungle.
Reality, as always, is probably somewhere in between.
MotherDuck and DuckDB have thrown their hat into the ring with yet another Lake House storage format. DuckLake joins an already crowded field, and here we are trying to make sense of it.
Let’s be honest: working with Apache Iceberg stops being fun the moment you step off your local laptop and into anything that resembles production. The catalog system—mandatory and rigid—has long been the Achilles’ heel of an otherwise promising open data format.
For a long time, you had two options: over-engineered corporate-grade solutions that require infrastructure teams to babysit or sketchy hacks no one wants to rely on.
Sometimes, we follow tradition because, well, it works—until something new comes along and makes us question the status quo. For many of us, Amazon S3 is that well‑trodden path: the backbone of our data platforms and pipelines, used countless times each day. If it vanished, our entire workflow would skid to a halt—and cost us a fortune in the process.
Every once in a great while, the question comes up: “How do I test my Databricks codebase?” It’s a fair question, and if you’re new to testing your code, it can seem a little overwhelming on the surface. However, I assure you the opposite is the case.
Building fun things is a real part of Data Engineering. Using your creative side when building a Lake House is possible, and using tools that are outside the normal box can sometimes be preferable. Checkout this video where I dive into how I build just such a Lake House using Modern Data Stack tools like AWS Lambda (for cheap and fast compute), DuckDB (for data processing) and Delta Lake for storage.
I’ve been playing around more and more lately with DuckDB. It’s a popular SQL-based tool that is lightweight and easy to use, probably one of the easiest tools to install and use. I mean, who doesn’t know how to pip install something and write SQL? Probably the very first thing you learn when cutting your teeth on programming when you’re wet behind the ears.
Well, another turkey day has come upon us all. I trust you are getting at least a day or two off from your overlords from writing code and taking names.
While the rest of you will be slicing up that turkey with your friends and family, clinking your glasses and giving toasts to each other, I will be stuck behind my glowing screen taping away at my keyboard producing juicy content for you all.
I appreciate your support in helping me to continue to bring interesting and sometimes controversial content into the Data Engineering space.
We recently reached over 14k subscribers to the newsletter and about 17k followers on Substack.
