The Era of the YAML Engineer

You know, I did fight it for a long time, and I’m still fighting it. Look, no one wants to become a Terraform engineer; that is pain and suffering. But, we all understand the benefits of IAC (infrastructure as code), and SHOULD be using it in our daily tech lives, or pushing towards it.

But there is a thin red line, and it gets crossed a lot.

Oh, you want a new Workspace in Databricks? Terraform. Do you need a new RDS instance in AWS? Terraform. Want to give some group permissions to X, Y, Z? Terraform.

Read more

Full vs Incremental Data Loads Explained

The Fastest Way to Insert Data to Postgres

I was recently working on a PySpark pipeline in which I was using the JDBC option to write about 22 million records from a Spark DataFrame into a Postgres RDS database. Hey, why not use the built in method provided by Spark, how bad could it be? I mean it’s not like the creators and maintainers of Spark aren’t probably our version of rocket engineers.

Well, a few hours later staring at my screen, I knew something had to change. Slower than your grandma on her way to the quilt shop.

Read more

The Medallion Architecture Farce.

I can no longer hold the boiling and frothing mess of righteous anger that starts to rumble up from within me when I hear the words “Medallion Architecture” in the context of Data Modeling, especially when it’s used by some young Engineer who doesn’t know any better. Poor saps who have been born into a Databricks world where that fresh, supple mind has been polluted and twisted by the machinations of a marketing department.

Look, I am a daily user of Databricks, I have no axe to grind with them in particular. But the false gospel of the “Medallion Architecture” wreaked havoc on a generation of Data Engineers.

Read more

DuckDB … Merge Mismatched CSV Schemas. (also testing Polars)

I recently encountered a problem loading a few hundred CSV files, which contained mismatched schemas due to a handful of “extra” columns. This turned out to be not an easy problem for Polars to solve, in all its Rust glory.

That made me curious: how does DuckDB handle mismatched schemas of CSV files?

Of course, this can be a tricky problem to solve, and every creator of a new Data Engineering tool probably has a different take on how this should be handled. The perfectionist will probably say … “Puke the whole thing, schemas should match exactly if you’re reading multiple files.” The realist, who has worked in data for many years, might say, “No, you simply need to at least give the option to MERGE the schemas on read.”

Read more

polars.exceptions.ComputeError: schema lengths differ

So, you are happily using the new Rust GOAT dataframe tool Polars to mung messy data, maybe like me, messing with 40GBs of CSV data over multiple files. You are pretty much going to run into this error.

polars.exceptions.ComputeError: schema lengths differ
This error occurred with the following context stack:
[1] ‘csv scan’
[2] ‘select’

Read more

Solving a “Fill Forward” NULL problem with Polars

I recently used Polars … inside an AWS Lambda … to fill a novel and somewhat obtuse CSV formatting issue.

We were receiving CSV files that contained rows with specific columns that were empty because the following values matched the first one, until a different value finally appeared.

Let me show you.

Read more

Lakebase: Databricks’ Bold Play to Fuse OLTP and the Lakehouse

The future never shows up quietly. Just when you think you’ve tamed the latest “must-have” technology, a fresh acronym crashes the party. I’d barely finished wrapping my head around the Lakehouse paradigm when Databricks rolled out something new at the 2025 Data & AI Summit: Lakebase, a fully managed PostgreSQL engine built directly into the Databricks platform.

Cue the collective gasp—and the scramble—for data teams everywhere.

Read more

Databricks SQL Scripting: A Familiar Friend or an Old Foe?

I’d be lying if I said a small part of me didn’t groan when I first read about SQL Scripting being released by Databricks.

Don’t get me wrong—I don’t fault Databricks for giving users what they want. After all, if you don’t feed the masses, they’ll turn on you. We data engineers are gluttons for punishment. We grumble, we fight with complexity, but somehow, we get the job done.

So why did this announcement sting a bit? Because I’ve been around long enough to remember the old days—the old days. When SQL Servers ruled the land, unleashing T-SQL, stored procedures, and SSIS packages everywhere. Some of you will remember those dark times, battling deadlocks and debugging cryptic errors in SSMS. Others were lucky enough to be born into the light.

Read more

DuckDB Enters the Lake House Race: My Take on DuckLake

I’ve been thinking about this for a few days now, and I still don’t know whether to cheer or groan. Some moments, I see DuckLake as a smart, much-needed evolution; other times, it feels like just another unnecessary entry in the ever-growing Lake House jungle.

Reality, as always, is probably somewhere in between.

MotherDuck and DuckDB have thrown their hat into the ring with yet another Lake House storage format. DuckLake joins an already crowded field, and here we are trying to make sense of it.

Read more