Apache Arrow + DuckDB (the GOAT + the GOAT)
It’s hard to find the bright, shining stars amid the doom and gloom the tech world seems to be floundering in. When the going gets tough, I like to remind myself that there are lots of new and exciting tools released in the last few years, most of which, when combined, have not been part of the great LLM training material, leaving some fun left to explore.
Two of my newest favorite tools, DuckDB and Apache Arrow, have been around a while but are now becoming more integrated, starting to stand more firmly on their own and together.
I think it’s valid to ask, “Why use Apache Arrow as a data interchange layer?” Especially when we have so many choices as far as data processing tools, PySpark, Polars, DuckDb, Daft, Pandas, blah, blah, blah. This is more of a philosophical idea than a technical problem, though it does give rise to more technical arguments at the ground level.
- Why use Arrow for data representation?
Because we can achieve good performance and zero-copy semantics while increasing the interoperability of data processing tooling and architecture.
Arrow gives us data processing speed and, more importantly, options.
Arrow is the perfect tool to use with tabular data, like the Divvy Bike Trip dataset (open source).
It’s not that this code is particularly wonderful; it may or may not be that. But in the larger picture of a Data Platform made up of various processing tools, where we want to use the right tool for the right job at the right time, using Arrow as a base, we have interoperability and the ability to interchange tools.
Maybe we work on a primary Databricks Spark platform, but we have numerous datasets that could be better handled with DuckDB, because at a certain scale and size, it’s much more cost-efficient and, overall, performs better to use a non-distributed tool like Spark for various data processing.
- Using Arrow, we can easily move fluidly between data tools, without any performance hits; in fact, any performance difference will likely be positive.
I mean, this is all sorta made up, but the point I’m trying to make is the world is a large place with all sorts of wonderful tools to use, many new ones that an LLM will never ever tell you to use alone, or together, to solve a problem unless specifically told to do so. The All-Knowing AI is kinda boring, trained on boring old data from 5 years ago.
Moving data across multiple, varied systems using Arrow as the base opens up a world of tools at your fingertips. It makes it easy to just “drop in DuckDB” or some other tool that others may see as a burden or just another tool to add to the already-packed list they deal with on a daily basis.
The simplicity of Arrow datasets, combined with the simplicity of SQL via DuckDB, is just what the doctor ordered for the SaaS-overextended Modern Data Stack.
