, ,

Rust + DuckDB – Fast meets Faster. Yowza.

There are many things I love in this life. God. Family. The Great Outdoors. Writing code. Reading good books. Writing on Substack. Friends. Pizza. Fishing. Ah well, you get the point. What a time to be alive, the world in tatters, AI upending everything we have taken for gospel for the last … well forever. Because I’m an old curmudgeon, I learned to write Perl as my introduction to code, used to run LAMP stacks on macs underneath my desk and watching everyone to try hack it. That being said, while everyone else is ringing their hands in the error and retching like Gollum at how AI has stolen our precious, our code; I just ignore it all, and do both, use AI for coding when required or expected, and still writing Rust by hand when I get chance for fun.

Read more

,

Writing Rust … by Hand. No LLMs.

I recently had a bit of a self-identity crisis; of course, it was tied to spending way too much time on social media. I do try to avoid the worst of it, got rid of Twitter (X) this year, and mostly stick to Substack, LinkedIn, YouTube, occasional Reddit, and Hacker News. Probably still too much.

I’ve always seen it as a form of self-preservation, probably due to growing up the youngest of five in the middle of the prairie next to the river, in the only county without a working stoplight.

I try to be pragmatic and never dogmatic, except when someone says “Production” and “Notebook” in the same sentence.

Read more

, , ,

Introduction to Polars in 2 Minutes

Polars is the hot new Rust based Python Dataframe tool that is taking over the world and destryoing Pandas even as we speak. You want the quick and dirty introduction to Polars? Look no farther.

, , ,

Reading and Processing JSON with Rust vs Python.

Have you ever wondered about being explicit in your code vs being vague? I think about this a lot as I’m writing code on a daily basis. I’ve found I like being explicit and verbose when writing code, rather than being vague in what I’m doing most of the time.

Read more

, , , , ,

Why did Golang lose to Rust for Data Engineering?

A few years ago I wasn’t sure, who was going to win, Golang seemed to be popular, and still is for that matter. When I first wrote a little Golang (~2+ years ago) I was just trying to see what the hype was all about. The funny thing is, at the time, and today, it seems like the Golang syntax is much simpler than Rust, easier to learn and pick up by far.

Read more

, , ,

Writing Apache Spark with Rust! Spark Connect Introduced.

I never thought I would live to see the day, it’s crazy. I’m not sure who’s idea it was to make it possible to write Apache Spark with Rust, Golang, or Python … but they are all genius.

As of Apache Spark 3.4 it is now possible to use Spark Connect … a thin API client on a Spark Cluster ontop of the DataFrame API.

You can now connect backend systems and code, using Rust or Golang etc, to a Spark Server and run commands and get results remotely. Simply amazing.  A new era of tools and products is going to be unleashed on us.  We are no longer chained to the JVM. The walls have been broken down. The future is bright.

, ,

Why I Love Rust, but Deploy Python

I’m not sure if others have this same problem, maybe they are lucky, they get to build in their favorite language 24/7, it’s their tool of choice. I feel like I have a great burden to bear, a heavy one. I love to write Rust … but I deploy Python. Even when I know I could write Rust … Python gets deployed.

Read more

, , ,

The Ultimate Data Engineering Chadstack. Running Rust inside Apache Airflow.

Is there anything more Chad than Apache Airflow … and Rust? I think not you whimp. What two things do I love most? At the moment Rust and Airflow are at least somewhere at the top of that list. I wring my hands sometimes, wishing that things and technologies somehow come together into some bubbling soup and witches concoction from the depths. Then I had a strange thought while laying in bed one night.

What would happen if I ran my Rust inside my Apache Airflow? What would happen? Would the sun go dark? Would SQL Servers everywhere puke up their log files and go to Davey Jones’s locker? Birds fall from the sky? Why hasn’t anyone done this before, why isn’t anyone making this happen in real life?

Read more

,

Introduction to using Rust Libraries (cargo and crates)

So perhaps you’re thinking it’s time to use Rust on your next project. You’ll find plenty of primers on how to get your feet wet in the language (and if you somehow made it this far without that much, The Book is that starting point), but maybe you’re feeling a bit lost amidst the seas of opportunity. While still growing, the Rust ecosystem has many great existing options to pull from, and you’re now asking “How do I even?”

If you’re wondering how to make sense of your third-party library ecosystem, I hope to get you thinking like a Rustacean.

If you’re stuck on trying to find the right library for you, jump down to the section on crates for a primer on discoverability. Otherwise, if you already know what you’re working with and are looking to understand it, keep reading to see how I do it.

Read more