,

The Difficulties of Senior Engineer …. are not Engineering

Well, I hate to break the news to you. I was the same when I first started, writing code that is. I was a zealot. I was zealous for every new thing I learned, every new language, every new approach, I would find the preacher who was preaching the message I wanted to hear … OOP, functional, Kimball, this, that, the other thing.

You’re young and full of life. You think that your Software career revolves around … software. The pinnacle of your mountain seems to be becoming that “perfect programmer” who can write anything without any bugs.

Yet, when you get to the top of the mountain you find you’ve been deceived. Not all that glitters is gold.

Read more

,

Engineering Lessons Learned from LLM Fine Tuning

Well, I finally got around to it. What you say? Fine-tuning an LLM, that’s what. I mean all the cool kids are talking about and caring on like it’s the next thing. What can I say … I’m jaded. I’ve been working on ML systems for a good few years now, and I’ve seen the best, and worst.

Most of Machine Learning is Data Engineering. That’s the truth. Is the LLM gold rush any different?

Read more

,

Polars vs Spark

, ,

SQL Bad, Reddit Mad

, ,

SparkSQL is Destroying your Pipelines

It’s true, even if you don’t want it to be. SparkSQL is destroying your data pipelines and possibly wreaking havoc on your entire data team, infrastructure, and life. In your heart of hearts, you’ve probably known it for years. With great power comes great responsibility. We all know that even us Data Engineers are human and fallible.

Once those tentacles of SparkSQL get their hold on you, the probability of survival is low. Sure, there are a few wizened old engineers with enough battle scars to make it through unscathed. The rest of us will be maimed.

Read more

, ,

Datafusion SQL CLI – Look Ma, I made a new ETL tool.

Sometimes I just need something new and interesting to work on, to keep me engaged. A few days ago I was lying by the river next to a fire, with the cold air blowing on my face and the eagles soaring above. Thinking about and contemplating life and data engineering … something flitted across my mind, just a little fragment of an idea someone had written about.

The little fragment had to do with Datafusion, a Rust-based query engine, and something about it having a SQL CLI interface.

What an interesting thing. I’ve used Datafusion a few times, here and there, I love Rust because it’s fast. I’m a Data Engineer so I’m eternally enslaved to SQL whether I like it or not. This whole thing just seemed like an interesting little tidbit to poke at.

It basically made me wonder if I could combine the Datafusion SQL CLI with bash into a new ETL tool. Simple, small, fast, and maybe fun? Just because I can?

Read more

, ,

How to be Better Than Everyone Else

Ok. Get off your high horse. You are human just like the rest of us. Just like your ancient ancestors who were throwing rocks and sticks at each other a thousand years ago … you are looking for a leg up on the competition. Isn’t that the world we live in? At the end of the day, no one is looking out for you, besides you.

Do you know what all those famous “influencers” have over you? Those you look at with boiling bitterness buried somewhere inside you. Besides a smattering of luck here and there, there is one other trait they have that most people don’t.

There are general branches and streams of actions that come off that main branch. But without that central drive, all else is useless.

What is it that apparently 1% of the data population has that the 99 don’t? Simply put, in the words of your grandparents, hard work.

Read more

,

Fleetclusters for Databricks + AWS to reduce Costs.

Show me the money. That’s what it’s all about. I have a question for you, to tickle your ears and mind. Get you out of that humdrum funk you are in. Here is my question, riddle me this all you hobbits.

“Of what use is, and what good does the best and most advanced architecture provide if your bosses boss starts to look at the cost and starts to ask questions?”

The thing is, as engineers and developers we love to do fun stuff. We follow the crowds, don’t we? We are like those lemmings, we just do what is cool.

Read more

, ,

Date and DateTime Manipulation in Polars

One thing all Data Engineers are doomed to do in purgatory will be to solve different date and datetime problems in an endless loop. I’m sure of it. I can’t imagine anything worse, so that must be it. Either way the constant need to manipulate dates and datetimes are just a way of life, something that never ends and never changes. Also, it appears Polars is here to stay from what I can tell. Not a fad like that Data Mesh. Since Polars is here to stay, (I’ve already got it running in production at my company, (don’t mind if I bow)), we should probably take a gander at how to manipulate date and datetime objects from both the Dataframe and (if I have time) SQL perspective. See if we can find anything to complain about. I like to complain.

Read more

, , ,

The Ultimate Data Engineering Chadstack. Running Rust inside Apache Airflow.

Is there anything more Chad than Apache Airflow … and Rust? I think not you whimp. What two things do I love most? At the moment Rust and Airflow are at least somewhere at the top of that list. I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling soup and witches concoction from the depths. Then I had a strange thought while laying in bed one night.

What would happen if I ran my Rust inside my Apache Airflow? What would happen? Would the sun go dark? Would SQL Servers everywhere puke up their log files and go to Davey Jones’s locker? Birds fall from the sky? Why hasn’t anyone done this before, why isn’t anyone making this happen in real life?

Read more