Will AI Kill Data Engineering (or Software Engineering)?

Short answer? No.Long answer? Also, no — but let’s talk about why. If you spend any amount of time on LinkedIn, X, YouTube, or your favorite tech newsletter, you’ve probably been told that software engineering, and especially data engineering, is on the brink of extinction. The headlines are loud, and the thumbnails are dramatic, heck […]

Read more

Introduction to Databricks SQL Temporary Tables

There was a moment when Databricks announced support for temporary tables, and a certain segment of the data world collectively blinked and did a double-take. If you came up through the ranks of SQL Server, Oracle, or any traditional warehouse, your first reaction may have been somewhere between nostalgia and disbelief. “Wait… we’re celebrating temp […]

Read more
,

Migrating to Databricks – A Guide

So you’re thinking about moving to Databricks. Maybe you’re frustrated with your current stack. Maybe leadership wants “AI readiness.” Maybe you’re just tired of duct-taped pipelines and brittle warehouses. Databricks is powerful. It is not magic. Before you migrate, you need clarity. Not excitement. Not feature envy. Clarity. This guide walks through how to approach […]

Read more
, ,

Why Declarative (Lakeflow) Pipelines Are the Future of Spark

Every few years, Spark reinvents itself. First, it was Scala and RDDs. Then DataFrames. Then Python took over. Now we’re entering the era of declarative pipelines. You can complain about abstraction if you want. There’s always a small but passionate group that mourns the loss of some lower-level construct. They cling tightly to bespoke implementations […]

Read more

Robin Moffatt on the Evolution of Data Engineering: From Batch Jobs to Real-Time | Podcast Interview

Read more