I was recently working on a PySpark pipeline in which I was using the JDBC option to write about 22 million records from a Spark DataFrame into a Postgres RDS database. Hey, why not use the built in method provided by Spark, how bad could it be? I mean it’s not like the creators and maintainers of Spark aren’t probably our version of rocket engineers.
Well, a few hours later staring at my screen, I knew something had to change. Slower than your grandma on her way to the quilt shop.
I can no longer hold the boiling and frothing mess of righteous anger that starts to rumble up from within me when I hear the words “Medallion Architecture” in the context of Data Modeling, especially when it’s used by some young Engineer who doesn’t know any better. Poor saps who have been born into a Databricks world where that fresh, supple mind has been polluted and twisted by the machinations of a marketing department.
Look, I am a daily user of Databricks, I have no axe to grind with them in particular. But the false gospel of the “Medallion Architecture” wreaked havoc on a generation of Data Engineers.
I recently encountered a problem loading a few hundred CSV files, which contained mismatched schemas due to a handful of “extra” columns. This turned out to be not an easy problem for Polars to solve, in all its Rust glory.
That made me curious: how does DuckDB handle mismatched schemas of CSV files?
Of course, this can be a tricky problem to solve, and every creator of a new Data Engineering tool probably has a different take on how this should be handled. The perfectionist will probably say … “Puke the whole thing, schemas should match exactly if you’re reading multiple files.” The realist, who has worked in data for many years, might say, “No, you simply need to at least give the option to MERGE the schemas on read.”
So, you are happily using the new Rust GOAT dataframe tool Polars to mung messy data, maybe like me, messing with 40GBs of CSV data over multiple files. You are pretty much going to run into this error.
polars.exceptions.ComputeError: schema lengths differ
This error occurred with the following context stack:
[1] ‘csv scan’
[2] ‘select’
I recently used Polars … inside an AWS Lambda … to fill a novel and somewhat obtuse CSV formatting issue.
We were receiving CSV files that contained rows with specific columns that were empty because the following values matched the first one, until a different value finally appeared.
Let me show you.
The future never shows up quietly. Just when you think you’ve tamed the latest “must-have” technology, a fresh acronym crashes the party. I’d barely finished wrapping my head around the Lakehouse paradigm when Databricks rolled out something new at the 2025 Data & AI Summit: Lakebase, a fully managed PostgreSQL engine built directly into the Databricks platform.
Cue the collective gasp—and the scramble—for data teams everywhere.
I’d be lying if I said a small part of me didn’t groan when I first read about SQL Scripting being released by Databricks.
Don’t get me wrong—I don’t fault Databricks for giving users what they want. After all, if you don’t feed the masses, they’ll turn on you. We data engineers are gluttons for punishment. We grumble, we fight with complexity, but somehow, we get the job done.
So why did this announcement sting a bit? Because I’ve been around long enough to remember the old days—the old days. When SQL Servers ruled the land, unleashing T-SQL, stored procedures, and SSIS packages everywhere. Some of you will remember those dark times, battling deadlocks and debugging cryptic errors in SSMS. Others were lucky enough to be born into the light.
I’ve been thinking about this for a few days now, and I still don’t know whether to cheer or groan. Some moments, I see DuckLake as a smart, much-needed evolution; other times, it feels like just another unnecessary entry in the ever-growing Lake House jungle.
Reality, as always, is probably somewhere in between.
MotherDuck and DuckDB have thrown their hat into the ring with yet another Lake House storage format. DuckLake joins an already crowded field, and here we are trying to make sense of it.
Let’s be honest: working with Apache Iceberg stops being fun the moment you step off your local laptop and into anything that resembles production. The catalog system—mandatory and rigid—has long been the Achilles’ heel of an otherwise promising open data format.
For a long time, you had two options: over-engineered corporate-grade solutions that require infrastructure teams to babysit or sketchy hacks no one wants to rely on.