,

Migrating to Databricks – A Guide

So you’re thinking about moving to Databricks. Maybe you’re frustrated with your current stack. Maybe leadership wants “AI readiness.” Maybe you’re just tired of duct-taped pipelines and brittle warehouses. Databricks is powerful. It is not magic. Before you migrate, you need clarity. Not excitement. Not feature envy. Clarity. This guide walks through how to approach […]

Read more
, ,

Why Declarative (Lakeflow) Pipelines Are the Future of Spark

Every few years, Spark reinvents itself. First, it was Scala and RDDs. Then DataFrames. Then Python took over. Now we’re entering the era of declarative pipelines. You can complain about abstraction if you want. There’s always a small but passionate group that mourns the loss of some lower-level construct. They cling tightly to bespoke implementations […]

Read more

Robin Moffatt on the Evolution of Data Engineering: From Batch Jobs to Real-Time | Podcast Interview

Read more

Data Engineering Career Path: From Circuits to Pipelines

Read more

This One Polars Pattern Makes Code 10x Cleaner

Read more
, , ,

Embeddings and Vector Databases – Lance + DuckDB + LangChain

It’s an interesting time to be in software and data; the world of generative AI is changing the landscape beneath our feet. I don’t see this as a bad thing for software folk, but as an opportunity to learn new technologies and BUILD / UNDERSTAND the technologies used in an LLM and AI context. You […]

Read more