, ,

Delta Kernel (Rust) and the future of the Lakehouse.

I recently spent some time with my old love, that is, Rust; fawning over the old days when I learned to fight the borrow checker without my own two worn-out hands, no LLMs to do the work for me. I still think it’s probably one of the most fun languages to learn, for many reasons. Yes, it can be a bit verbose, but it also teaches you to think about your programs and problems in a different way. It’s fast, way fast, and simply brings joy. It’s also a great tool to write building blocks with, CLIs and other crates that can be consumed by others.

I’ve also been an avid user of the Lakehouse Architecture since before it was cool.

Read more

,

How to Orchestrate LLM Calls in Your Data Pipelines

,

Solution > Problem.

, ,

Cloudflare as a Data Platform? Yeah. No.

,

Don’t say “I found a problem.” Say … “I found a problem, here’s a solution .. and proof.”

A few things have not changed in the age of AI, where the LLM has replaced 3/4 of the brain power of a whole generation of engineers who once found their meaning in the code they wrote. I mean, I hate to say it’s true, but a lot of folk should have been finding their meaning in the code they decided NOT to write. It’s the classic mix-up of smart engineers and developers thinking that their “use” or “importance” to a company or product was the amount and cleverness of the code they put out. They worshipped at the feet of their own fingers, enamored with staring at themselves in the mirror, convincing themselves they were smarter and could write better code than those around them.

Read more

, , , ,

The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering | Wes McKinney

, ,

AWS Lambda MicroVMs: The Compute Nobody Asked For?

When AWS announced Lambda MicroVMs, I was genuinely excited. After years of incremental services and endless feature releases, it finally looked like AWS had built something that filled the awkward gap between Lambda and EC2. On paper, it sounded perfect. You get VM-level isolation, more memory, persistent state, and startup times measured in milliseconds, all without managing infrastructure. If you’ve ever wished Lambda had fewer limitations or EC2 required less babysitting, this looked like the answer.

Then I actually built something with it. Needless to say, I felt betrayed, lied to, and taken for a ride. Not a happy camper am I.

Read more

, ,

Delta Lake + DuckDB. Catalog Commits with Unity Catalog. Unlocking Concurrent Ingestion.

Why isn’t anyone talking about this?

Sometimes I am genuinely amazed by what captures the attention of the broader data community and what gets quietly pushed off to the side. I suppose I understand why it happens. New models arrive, vendors announce shiny features, and everyone rushes toward the next big thing. Still, as someone who has spent the last few years championing what I affectionately call the Single Node Rebellion, it is difficult not to wonder why certain developments receive so little attention. This is one of those moments.

Read more

, , ,

From Failure to AWS: What Actually Makes a Great Engineer