Databricks SQL Scripting: A Familiar Friend or an Old Foe?

I’d be lying if I said a small part of me didn’t groan when I first read about SQL Scripting being released by Databricks.

Don’t get me wrong—I don’t fault Databricks for giving users what they want. After all, if you don’t feed the masses, they’ll turn on you. We data engineers are gluttons for punishment. We grumble, we fight with complexity, but somehow, we get the job done.

So why did this announcement sting a bit? Because I’ve been around long enough to remember the old days—the old days. When SQL Servers ruled the land, unleashing T-SQL, stored procedures, and SSIS packages everywhere. Some of you will remember those dark times, battling deadlocks and debugging cryptic errors in SSMS. Others were lucky enough to be born into the light.

What is SQL Scripting in Databricks?

Enough nostalgia—what are we dealing with here?

Databricks SQL Scripting allows you to write procedural logic using SQL/PSM standard-based syntax. Scripts start with a compound statement (BEGIN ... END) where you can declare variables, conditions, and handlers for exceptions. And yes, you can wrap this procedural logic into stored procedures. These procedures can be persisted in Unity Catalog, shared with other users, and invoked using CALL.

If this sounds familiar, it’s because it is. It’s the same pattern that’s been around for decades.

The Risks We Know Too Well

So why does this raise alarm bells for some of us? Because we’ve seen how stored procedures can go wrong. In classic SQL Server environments, stored procedures often became dumping grounds for business logic that should have been handled elsewhere. The result? Pipelines that were:

  • Hard to trace — the logic was buried inside invisible procedures
  • Difficult to maintain — debugging required spelunking through layers of hidden code
  • Easy to abuse — it became tempting to jam too much logic into a single place

And let’s be honest: these challenges weren’t caused by stored procedures themselves. The problem was (and is) human—bad design decisions, poor documentation, lack of observability.

Can We Do Better This Time?

Databricks SQL Scripting is undeniably powerful. Like any powerful tool, it can be used well or misused terribly. It will face the same risks stored procedures have always faced:

  • Too much hidden logic = future pain
  • Obscured pipelines = operational headaches
  • Overly fancy solutions = costly technical debt

If you plan to use SQL Scripting, here’s my advice:

✅ Keep logic clean and focused
✅ Use source control—everything
✅ Make your pipelines observable and easy to follow

Final Thoughts

What do I think of Databricks SQL Scripting? Honestly, I’m neutral. It’s just another tool—how we use it will determine whether it helps or hurts. Some engineers will build elegant, maintainable solutions. Others will create a mess. The difference isn’t in the tool—it’s in the choices we make.