Hmm … data types. We all know they are important, but we don’t take them very seriously. I mean we know the difference between boolean, string, and integers, those are easy to get right. But we all get sloppy, sometimes we got the string and varchar route because we don’t spend enough time on the data model to care.
Can a string versus a int or bigint in Delta Lake with Spark have a big impact on performance? Data size? Does it matter? Let’s find out.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2023-02-11 04:45:072023-02-11 13:38:21Data Types in Delta Lake + Spark. Join and Storage Performance.
I remember those days, oh so long ago, it seems like another lifetime. I haven’t used Pandas in many a year, decades, or whatever. We’ve all been there, done that. Pandas I mean. I would dare say it’s a rite of passage for most data folk. For those using Python, it’s probably one of the first packages you use other than say … requests?
You know, Pandas feels like Airflow, everyone keeps talking about its demise, but there it is everywhere … used by everyone. Sure it’s old, wrinkled, annoying, slow, and obtuse, but it’s ours, and that makes it the words of Gollum … precious.
We should probably get to the point already. Everyone is talking about Polars. Polars is supposed to replace Pandas. Will it? Maybe 10 years from now. You can’t untangle Pandas from everywhere it exists overnight. Do you still want to replace Pandas with Polars and be one of the cool kids? Ok. Let’s take a look at a practical guide to replacing Pandas with Polars, comparing functionally used by most people. My code is available on GitHub.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2023-01-19 22:14:122023-01-19 22:14:12Replacing Pandas with Polars. A Practical Guide.
Anyone who’s been roaming around the forest of Data Engineering has probably run into many of the newish tools that have been growing rapidly around the concepts of Data Warehouses, Data Lakes, and Lake Houses … the merging of the old relational database functionality with TB and PB level cloud-based file storage systems. Tools like Delta Lake, lakeFS, Hudi, and the like.
Sure, these tools have been around for some time, but the uptake and adoption of them all have been rapidly growing. I use Delta Lake on a daily basis, taking advantage of the many wonderful features it provides to simplify and reduce complexity in data pipelines. But, I’ve been sitting around for a long time waiting for the plethora of “add-on” tooling to come out, stuff that will make my life easier. I recently saw one of the first tools like that for Delta Lake, namely mack.
Mack appears to have the ability to “do the hard work for you,” a concept that appears to be growing in popularity, but which I have a fraught relationship with. Double-edged sword? Let’s find out.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2023-01-12 22:11:152023-01-12 22:21:12Simplify Delta Lake Complexity with mack.
We’ve all been in that spot, especially in tech. You wanted to fit in, be cool, and look smart, so you didn’t ask any questions. And now it’s too late. You’re stuck. Now you simply can’t ask … you’re too afraid. I get it. Apache Arrow is probably one of those things. It keeps popping up here and there and everywhere.
“Back then it seems like no one used Arrow much, no one was writing about it, using it, or talking about it. At least not that I saw. But oh how times have changed. Arrow seems to be showing up everywhere and is starting to become a backbone for many other tools.”
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-12-27 14:12:562022-12-27 14:23:25What is Apache Arrow? Asking for a friend.
There once was a day when no one used DataFrames that much. Back before Spark had really gone mainstream, Data Scientists were still plinking around with Pandas a lot. My My, what would your mother say? How things have changed. Now everyone wants a piece of the DataFrame pie. I mean it tastes so good, doesn’t it?
Would anyone like a nice big slice of groupBy, maybe agg is what you need? No? Can you say distributed data set? Whatever it is you’re looking for, I’m quite sure a nice old DataFrame can give it to you. With so many options to choose from … what do you choose? I don’t know, whatever works best for you. But, it does set the stage nicely for a clash of the titans per see.
Let’s do this just that. Straight out of the box performance test. Bunch of CSV’s, a little aggregation, just some simple stuff. Mirror mirror on the wall, who is the fastest with DataFrames of them all?
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-12-10 16:46:122022-12-12 21:15:18Dataframe Showdown – Polars vs Spark vs Pandas vs DataFusion. Guess who wins?
I’ve often wondered what purgatory would be like, doing penance for millennia into eternity. It would probably be doing data migrations. I suppose they are not all that dissimilar from normal software migrations, but there are a few things that make data migrations a little more horrible and soul-sucking. Data migrations are able to slow teams down to a crawl, take at least twice as long as planned, and be way more difficult than imagined.
Can’t it be made easy, shouldn’t Data Migrations have been conquered by now? I mean just put together the perfect plan, break up the work, make a bunch of tickets, estimate the work, and the rest falls into place? If only.
Nothing captures the imagination and heart like a tale of betrayal and heartbreak, and that is a tale I want to bring to you today. It’s a tale of Databricks Workflows and Jobs, version changes, new features, API’s, and insidious little hidden gems that will make you pull your hair out when you find them. It’s a tale of what not to do, a tale of how to put developer and customer experience first, instead of forcing unwanted solutions down the throats of the little birdies feeding at your nest.
As a Data Engineering simplicity and ease of use is something close to my heart, something that Databricks did well, or maybe I should say used to do well … before recent releases like Jobs 2.1 API. I hope you can hear the bitterness oozing from my words.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-12-01 15:48:102022-12-01 15:48:40A Tale of Betrayal and Heartbreak – Databricks Workflows and Jobs.
There are probably few things in life that will strike more fear and tumult in the heart of the Data Engineer than historical loads. You know, on the surface it seems like such an innocent thing. How could it possibly be, just take a bunch of data stored somewhere and shove it into a table. If only. Life never works that way, and neither does the historical load. You would think after decades we all would have figured it out you know. Is it because we don’t do it enough? Maybe it’s like regex, you just figure it out as you go every single time, telling yourself you’ll do it right next time.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-11-05 22:57:382022-11-06 23:24:18Introduction to Historical Loads – for Data Engineers.
An interesting topic of late that has been rattling around in my overcrowded head is the idea of Big Data vs Not Big Data, and the intersection thereof. I’ve been thinking about SAAS vendors, the Modern Data Stack, costs, and innovation. A great real-life example of all these topics is Delta Lake. Delta Lake is the child of Databricks, officially or not, and at a minimum has exploded in usage because of the increasing usage of Databricks and the popularity of Data Lakes.
Delta Lake, Hudi, Iceberg, all these ACID/CRUD abstractions on top of storage for Big Data have been game changers. But, as with any new popular tech, it comes with its own set of challenges. Specifically for Delta Lake … if you want to use it 99.9% of people are going to have to use Spark to do so, which can be costly, in terms of running clusters, and add complexity, in terms of new tooling, data pipelines, and the like. Anytime you only have one path to take with a tool, innovation is stifled, and barriers arise. Enter delta-rs the Standalone Rust API for Delta.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-10-25 19:30:022022-10-25 19:32:13Delta Lake without Spark (delta-rs). Innovation, cost savings, and other such matters.
Do this, not that. Well, I’ve got my own list. With everyone jumping on the PySpark / Databricks / EMR / Glue / Whatever bandwagon I thought it was long overdue for a post on what to do, and not to do when working with Spark / PySpark. I take the pragmatic approach to working with Spark, it’s honestly very forgiving well and far into the 10s of TBs of data. Once you wander past that point things tend to get a little spicy if you don’t have it all dialed in. As with most things in life if you get a few things right, and of course don’t do some things, that will get you a long way, the same applies to Spark.
https://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.png00Danielhttps://www.confessionsofadataguy.com/wp-content/uploads/2019/03/DG_logo450-300x104.pngDaniel2022-10-13 00:51:502022-10-13 20:55:05New to PySpark? Do this, not that.
