I recently wrote on my Substack (Data Engineering Central) about how I used the new OpenAI o1 model to do some basic Data Engineering tasks surrounding PostgreSQL. It did ok. I’ve also been using CoPilot and ChatGPT for over a year now to assist me with my daily code that I have to write for one reason or another.
It is a Brave New World out there these days. The new tools and features come out faster than your mom on Sunday morning getting you ready for church. The same goes for the context and advice being produced on a myriad of platforms, the ole’ Like and Subscribe, and all that bit. It does make you wonder after a while, what you can trust, who has your best interest in mind, and who is selling you a bottle of snake oil, doesn’t it?
Today we talk about Data Modeling. Specifically Data Modeling in the new world we all live in christened The Lakehouse by our benevolent Vender Overlords.
Did you know there are only 3 types of Data Engineers? It’s true. I hope you are the right one.
Over the many years I’ve been pounding my keyboard … Perl, PHP, Python, C#, Rust … whatever … I, like most programmers, built up a certain disdain for what is called Low Code / No Code solutions. In my rush to worship at the feet of the code we create, I failed, in the beginning, to recognize some important axioms …
Polars is the hot new Rust based Python Dataframe tool that is taking over the world and destryoing Pandas even as we speak. You want the quick and dirty introduction to Polars? Look no farther.