Databricks Spark adds Excel data source.

Read more
, , , ,

DuckDB beats Polars for 1TB of data.

I’ve been a Polars bro for most of the last few years. Why? It’s Rust-based, fast, DataFrame-centric, just the way I like it. It also had the excellent feature, right from the start, of Lazy Execution. A few years ago, maybe two, I actually put Polars into production, running on Airflow, working with S3 and […]

Read more

Scott Haines on the Future of Data Engineering

Read more
,

The Age of Agentic AI | LangChain and LangGraph style.

It’s a fast-paced and ever-changing world we live in; nothing we can do about it. I grew up in the middle of the prairie, when the internet became mainstream, the age of Doom, Myst, MSN Messenger, Yahoo Pool, and that irreplaceable Goldeneye, let’s be honest, World of Warcraft on a PC was game-changing. I suppose […]

Read more

PySpark vs DuckDB vs Polars: The Results

Read more

The Evolution of Databricks Compute – Serverless is Winning

Read more

Parquet Killer? Introduction to the Lance File Format.

Read more

SQL Data Modeling with One Big Table (OBT)

Read more
,

Migrate (hundreds) Delta Lake Partitioned Tables to Liquid Clustering

Recently, I had to migrate a few hundred Delta Lake tables that were partitioned over to Liquid Clustering. It seems straightforward on the surface, but everything usually does, until it isn’t. There was no rocket science involved here, but I did want to write this up to help the myriad of others who will probably […]

Read more
, ,

PyArrow for Large Dataset Processing

  Over the last few years, I’ve found myself using PyArrow more and more for everyday data engineering things. Data ingestion, reading, and writing from various data sources and sinks. Most of us are familiar with Arrow and how it underpins a lot of new tech like DataFusion, and Arrow is used as an internal […]

Read more