, ,

Delta Kernel (Rust) and the future of the Lakehouse.

I recently spent some time with my old love, that is, Rust; fawning over the old days when I learned to fight the borrow checker without my own two worn-out hands, no LLMs to do the work for me. I still think it’s probably one of the most fun languages to learn, for many reasons. Yes, it can be a bit verbose, but it also teaches you to think about your programs and problems in a different way. It’s fast, way fast, and simply brings joy. It’s also a great tool to write building blocks with, CLIs and other crates that can be consumed by others.

I’ve also been an avid user of the Lakehouse Architecture since before it was cool.

Read more

, ,

Cloudflare as a Data Platform? Yeah. No.

, , , ,

The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering | Wes McKinney

, ,

AWS Lambda MicroVMs: The Compute Nobody Asked For?

When AWS announced Lambda MicroVMs, I was genuinely excited. After years of incremental services and endless feature releases, it finally looked like AWS had built something that filled the awkward gap between Lambda and EC2. On paper, it sounded perfect. You get VM-level isolation, more memory, persistent state, and startup times measured in milliseconds, all without managing infrastructure. If you’ve ever wished Lambda had fewer limitations or EC2 required less babysitting, this looked like the answer.

Then I actually built something with it. Needless to say, I felt betrayed, lied to, and taken for a ride. Not a happy camper am I.

Read more

, , ,

From Failure to AWS: What Actually Makes a Great Engineer

, , ,

What is Apache Arrow Flight?

I don’t know what it is about Apache Arrow, that GOAT of data engineering, that snuck in like a weasel through the backdoor, we all woke up one day and found out Arrow is the Atalas of the data world, holding up the systems we depend on and take for granted. One name you might, or might not have heard rattling around is “Apache Arrow Flight” or “Arrow Flight.”

It pops up a little in blogs, talks, and the READMEs of some GitHub repos… I imagine all the vibe-coding Chads just nodding furiously on Zoom calls, like wise old owls who know exactly what’s being talked about. Admit it, you don’t know what Arrow Flight is.

  • Apache Arrow Flight is a high-performance data transport framework built on top of Apache Arrow and gRPC that allows applications to move large datasets across networks much faster than traditional technologies like JDBC, ODBC, CSV exports, or REST APIs

Read more

, ,

Databricks Zerobus: Event Streams + The Lakehouse

I had not been thinking much about Kafka lately, but depending on who you ask, Kafka is either sitting comfortably at the top of the streaming world or beginning a slow decline into abstraction. The truth is probably somewhere in the middle. I’ve seen a number of newish streaming tools, many Rust-based, come in like a splash and leave nothing but a few bubbles.

Kafka is entrenched in the streaming world, and this has created a problem for the Lake House architecture. The integration isn’t “as easy as pie,” … far from it.

Read more