Transactions Are Coming to the Lakehouse… Whether You Like It or Not
Every once in a while, I catch myself wondering if I should sell everything, move to a cabin in the woods, and raise goats… or keep doing this Data Engineering thing a little longer. The problem with sticking around is that you start to recognize patterns, and lately it feels like we’re watching history repeat itself—just with better branding and cloud infrastructure.
With things like stored procedures showing up in Spark and Databricks, I probably should have seen this coming. Transactions were always going to follow. That’s on me.
At this point, it’s useful to divide the data world into two groups.
- On one side, you’ve got the modern crowd, raised on Iceberg, Delta Lake, and the promise of the lakehouse. On the other hand, you’ve got the veterans, former DBAs and warehouse engineers who spent years working in systems like SQL Server, Oracle, or Postgres.
- The younger crowd will often point out that we already have transactions in the lakehouse: we can insert, update, and delete data just fine. And that’s true, but it’s not the full story.
What’s being introduced now is something more familiar to the old guard: coordinated, multi-statement transactions across tables. The kind of logic where multiple operations must succeed together or fail together. As Databricks puts it, “Transactions let you coordinate operations across multiple SQL statements and tables.” This pattern powered entire generations of applications—financial systems, inventory systems, and deeply embedded business logic wrapped in stored procedures. It wasn’t a niche feature; it was foundational.
Databricks bringing this capability into the lakehouse isn’t surprising when you think about it. A large portion of the data world still runs on these patterns, and making migration easier from legacy systems is simply good business. Features like temp tables and stored procedures were early signals. Transactions are the natural next step.
From an implementation standpoint, Databricks supports both interactive and non-interactive transaction models, which reflects how people actually use the platform today. Some workflows live in automated pipelines, while others happen in notebooks. The syntax and behavior differ slightly between the two, especially around commit and rollback behavior, but the underlying concept remains the same.
What makes this all work in a lakehouse architecture is the use of snapshot isolation. When a transaction begins, the system captures a consistent snapshot of the data at the point of first access, and all subsequent reads operate against that snapshot. This allows for consistency without relying on the traditional locking mechanisms you’d see in a relational database, which is a key distinction.
- Of course, just because something works doesn’t mean it can’t be abused. It didn’t take much effort to reproduce a classic transaction conflict scenario: two sessions, two open transactions, one commit followed by another. The second one fails, just as you’d expect. The mechanics are sound, but the implications are familiar to anyone who has spent time debugging transactional systems.
That’s really the heart of it. This is a powerful feature and a meaningful step forward for the lakehouse, particularly for organizations migrating complex workloads. At the same time, it reintroduces a class of problems that many modern data teams have managed to avoid. Long-running transactions, hidden business logic in SQL, and increasingly complex debugging scenarios were all part of the old world. There’s no reason to believe they won’t show up again.
So yes, this is progress. It’s also a reminder that the data world doesn’t move in straight lines. We revisit ideas, refine them, and rebuild them on new foundations. Transactions in the lakehouse are a perfect example of that. The technology is newer, the architecture is different, but the core concept is something we’ve seen before. And like before, it will reward teams that use it carefully and punish those that don’t.
