Apache Arrow ADBC Database Drivers

Anyone who’s been around for more than a decade or so in the programming, development, and data world might get a slight eye twitch when the word database driver appears. Before the modern times we live in came along, the entire data world was driven by SQL Server, Oracle, with just a sprinkling of Postgres and MySQL … a-la AWS. That’s just the way it was.

Part of that joy was dealing with database drivers, such as JDBC and ODBC, which are language-neutral, OS-level standards for accessing databases.

How it works:

Your application → ODBC Driver Manager → ODBC Driver → Database

• API based on C

• Very common on Windows (but exists on Linux/macOS)

• Used by tools like Excel, Power BI, Tableau, and many BI/ETL tools

• Requires configuring DSNs (Data Source Names) or connection strings

Ahhh … the good (bad) old days.

Apache Arrow has quietly eaten the data world, small bite by small bite. It’s lightweight, columnar, fast, and has bindings across all popular programming languages. Low or no serialization and deserialization, depending on the tools that are used and passing around Arrow data.

I’m sure you fine folk can think of another area that pulls and pushes a lot of tabular data, eh?

Yeah, database drivers.

A tale as old as time… we push data into a database, we pull it back in, we push it, we pull it. Forever till you’re dead and buried.

Apache ADBC (Arrow Database Connectivity) is a modern database driver standard built on Apache Arrow’s columnar in-memory format. It’s designed to replace or complement ODBC/JDBC in environments where Arrow-native tools are already in use.

Instead of row-by-row marshaling, ADBC moves Arrow RecordBatches end-to-end, drastically reducing copies, conversions, and overhead.

At a high level:

Application → Arrow / ADBC → Database
(no row materialization in between)

Arrow is already everywhere in the data landscape, yet many people are simply unaware of it. Arrow underpins: DuckDB, Polars, Spark, DataFusion, Pandas (interop), Flight / Flight SQL – you can use Arrow with all these tools. So it would make sense to add Arrow at the database driver layer … this is a big part of what we deal with today.

3. Performance without exotic tricks

This is what my benchmark shows clearly: