The Era of the YAML Engineer

You know, I did fight it for a long time, and I’m still fighting it. Look, no one wants to become a Terraform engineer; that is pain and suffering. But, we all understand the benefits of IAC (infrastructure as code), and SHOULD be using it in our daily tech lives, or pushing towards it.

But there is a thin red line, and it gets crossed a lot.

Oh, you want a new Workspace in Databricks? Terraform. Do you need a new RDS instance in AWS? Terraform. Want to give some group permissions to X, Y, Z? Terraform.

This makes sense. You know what gets annoying? When it becomes not just “infrastructure” that turns into YAML, but into things that should be done programmatically… with code, an API, a CLI, a SDK. Going YAML crazy can actually make your Tech Stack, Job, Debugging, and Development more …

  • complicated
  • less flexible
  • harder to develop
  • harder to debug

And you stop winning at a certain point. But we have no choice; the winds of software development change over time, and there is little the peons can do about it. Things get hot, the CTOs froth and foam over such things, and we do them. Whatever happened to a good old API call? What happened to a little code and some SDK calls?

They aren’t so bad after all, much easier to read than 50 Terraform modules.

I call for keeping the YAML configs for the infrastructure and out of the day-to-day code that actually runs our thing. Whatever that thing is.

I want the old days back, you know. But this old dog can learn new tricks. I’ve spun up my fair share of infrastructure with Terraform this year. I can follow all the lemmings down their dirt paths to ignominy. Besides, Cursor is not too bad at writing those YAML files, just give em’ a little nudge with the docs now and again.

This too shall pass, just like the AI overlords taking away our code from us. Infrastructure is next; pretty soon, we will all be prompt engineers.

Tread the YAML path lightly

Listen to me, you wicked DevOps Engineer, keep on your side of the playground. You want to codify every little bit of consistent architecture? Fine, go for it. Don’t touch my data pipelines, keep away, you sneaky rats.

We live in the real world; our data and pipelines change daily. We face new requirements and challenging problems to solve. We must be creative at midnight when Slack goes off. Do you think I want to look at a Databricks Pipeline defined like this?

I do not.

I want my Apache Airflow, I want my PySpark before my sad eyes. Who wants to decipher 50 pipelines that look like this junk? I mean, I will if I have to, sullenly with a frown on my face. But we all march to the beat of drums pounded by those over us. We are all just peons doing the bidding of those in power, who are in turn held captive by the SaaS Lords.

I have embraced the YAML, I can like that famous man, able to “Become all things to all people.“

Don’t bite the hand that feeds you. Maybe the good Lord will save us from our YAML fate.