, ,

Rust + DuckDB – Fast meets Faster. Yowza.

There are many things I love in this life. God. Family. The Great Outdoors. Writing code. Reading good books. Writing on Substack. Friends. Pizza. Fishing. Ah well, you get the point. What a time to be alive, the world in tatters, AI upending everything we have taken for gospel for the last … well forever. Because I’m an old curmudgeon, I learned to write Perl as my introduction to code, used to run LAMP stacks on macs underneath my desk and watching everyone to try hack it. That being said, while everyone else is ringing their hands in the error and retching like Gollum at how AI has stolen our precious, our code; I just ignore it all, and do both, use AI for coding when required or expected, and still writing Rust by hand when I get chance for fun.

Read more

DuckDB Labs acquired by AWS. What does it mean?

Ok, I’m just going to give it to you straight. No beating around the bush, you came here for the DuckDB Labs bought by AWS news right? Wondering what that means? What does it mean for DuckDB? What is DuckDB Labs, even? Never fear, I got you boo.

What is DuckDB Labs?

This is a good question, many people might be confused from the start what exactly is DuckDB Labs? You’ve probably heard of the widely popular DuckDB open-source project, I’m sure.

Read more

, ,

Delta Lake + DuckDB. Catalog Commits with Unity Catalog. Unlocking Concurrent Ingestion.

Why isn’t anyone talking about this?

Sometimes I am genuinely amazed by what captures the attention of the broader data community and what gets quietly pushed off to the side. I suppose I understand why it happens. New models arrive, vendors announce shiny features, and everyone rushes toward the next big thing. Still, as someone who has spent the last few years championing what I affectionately call the Single Node Rebellion, it is difficult not to wonder why certain developments receive so little attention. This is one of those moments.

Read more

, ,

The Truth About The Modern Data Stack | DuckDB Insider

, , , ,

Apache Arrow + DuckDB (the GOAT + the GOAT)

It’s hard to find the bright, shining stars amid the doom and gloom the tech world seems to be floundering in. When the going gets tough, I like to remind myself that there are lots of new and exciting tools released in the last few years, most of which, when combined, have not been part of the great LLM training material, leaving some fun left to explore.

Two of my newest favorite tools, DuckDB and Apache Arrow, have been around a while but are now becoming more integrated, starting to stand more firmly on their own and together.

Read more

, , ,

Embeddings and Vector Databases – Lance + DuckDB + LangChain

It’s an interesting time to be in software and data; the world of generative AI is changing the landscape beneath our feet. I don’t see this as a bad thing for software folk, but as an opportunity to learn new technologies and BUILD / UNDERSTAND the technologies used in an LLM and AI context.

You can’t expect an LLM trained two years ago to be up-to-date on what the new and best approaches are for X, Y, Z tech.

Sure, they can do a decent job given enough context, Agents, etc, but if you’re working on the cutting edge of AI and LLM infrastructure, you are going to have to be active in the community and reading about what others are doing, who’s releasing new tools, and what those tools do.

Don’t forget, there is the whole architectural and systems design piece. One part of the LLM and AI infrastructure is vector and embedding representations.

Read more

, , , ,

DuckDB beats Polars for 1TB of data.

I’ve been a Polars bro for most of the last few years. Why? It’s Rust-based, fast, DataFrame-centric, just the way I like it. It also had the excellent feature, right from the start, of Lazy Execution. A few years ago, maybe two, I actually put Polars into production, running on Airflow, working with S3 and reading Delta Lake tables.

I was in love.

Read more

, , ,

DuckDB vs Polars. Wait. DuckDB and Polars.

So, the classic newbie question. DuckDB vs Polars, which one should you pick?

This is an interesting question, and actually drives a lot of search traffic to this website on which you find yourself wasting time. I thank you for that.

This is probably the most classic type of question that all developers eventually ask at some point in their sad and depressing lives. Isn’t that the same story that is as old as time? This stick is better than that rock. Rust is better than C. Databricks better than Snowflake. You know, Delta Lake better than Iceberg.

And so the world keeps turning and grinding away.

DuckDB vs Polars? That’s the wrong question.

Read more

, , ,

Apache Iceberg Writes with DuckDB (or not)

Well, all the bottom feeders (Iceberg and DuckDB users) are howling at the moon and dancing around a bonfire at midnight trying to cast their evil spells on the rest of us. Apache Iceberg writes with DuckDB? Better late than never I suppose.

Your witchy ways won’t work on me.

Not going to lie, Iceberg writes with MotherDuck is an interesting concept. MotherDuck is lit and Iceberg only puts a little ice on the fire.

Many other tools like Polars or Daft have been offering Iceberg writes for ages now, it’s about time DuckDB preened its feathers and added write support. Up until now the DuckDB Iceberg Extension has all about the read. But, that is pretty much good for HelloWorld() crap pumped and dumped on Redditors.

We need write support in the real production world. Oh, and not on some Iceberg table stored on your laptop you ninny.

Read more

, , , , ,

AWS Lambda + DuckDB + Polars + Daft + Rust

When it comes to building modern Lake House architecture, we often get stuck in the past, doing the same old things time after time. We are human; we are lemmings; it’s just the trap we fall into. Usually, that pit we fall into is called Spark. Now, don’t get me wrong; I love Spark. We couldn’t have what we have today in terms of Data Platforms if it wasn’t for Apache Spark.

Read more