, ,

Lazy Execution with Polars and DuckDB

Something I’ve taken for granted for a long time, and then suddenly discover others are discovering for the first time, leaves me a little baffled. It makes me wonder how many folks are living under a proverbial rock. Recently, I saw a post on that infamous LinkedIn about someone excited about Polars’ lazy execution.

Read more

Cluster Fatigue. Polars and PyArrow to Postgres and Apache Iceberg (streaming mode)

I’ve been working lately, on moving expensive distributed compute jobs (that don’t need to be distributed) from Spark, to other single node tools and frameworks. To be honest, there are reasons that Data Platforms might pick Spark, for example, and just keep everything on Spark, even if it doesn’t need Spark. Yes, it costs more. […]

Read more

Introduction to Databricks Asset Bundles.

Need a gentle introduction to Databricks Asset Bundles, what they are, how to use them, why to use them? Look no further you hobbit.

Read more

Fivetran buys DBT. People get mad.

Well, we all knew that open source wasn’t a real thing anymore. This just confirms it. I don’t use DBT much, I think it’s for whimps and script kiddies. Anywho, I love watching Linkedin and Reddit explode with anger at Fivetran buying DBT. Everyone things dbt core is done. Who cares. Babies.

Read more
,

Running Llama 3.1 8B Locally (LangChain and SQLite)

Things have changed a lot in the last year related to LLMs and AI; on the one hand, it seems the AI skeptics for coding are increasingly confined to the corners of the internet. Everyone is dancing around in the middle, not sure of where everything should fall. Clearly, if we don’t use AI at […]

Read more

DuckDB on Mother Duck. Lake House Glory?

Read more

The Semantic Layer – Real or Not?

Read more

The Era of the YAML Engineer

You know, I did fight it for a long time, and I’m still fighting it. Look, no one wants to become a Terraform engineer; that is pain and suffering. But, we all understand the benefits of IAC (infrastructure as code), and SHOULD be using it in our daily tech lives, or pushing towards it. But […]

Read more
, , ,

DuckDB vs Polars. Wait. DuckDB and Polars.

So, the classic newbie question. DuckDB vs Polars, which one should you pick? This is an interesting question, and actually drives a lot of search traffic to this website on which you find yourself wasting time. I thank you for that. This is probably the most classic type of question that all developers eventually ask […]

Read more

Full vs Incremental Data Loads Explained

Read more