Thoughts on Saint Augustine, Rust vs Golang. Complexity, verbosity, and other matters.

Image: Saint Augustine of Hippo | Line engraving by P. Cool after M. de Vos | Wellcome Images

I’ve always enjoyed reading Mr. Augustine of Hippo, particularly “Confessions.” Ahead of his time in many ways. Although, you have to be into that sort of thing to find such topics interesting. It can be sort of dry, drawn out, verbose, and not for the faint of heart. Much like learning new programming languages. I’ve been messing with Golang off and on and here and there. Recently I added Rust to that list, more out of curiosity and to see what’s new in the world.

I’ve spent a lot of time thinking about the theology of programming in the space of Data Engineering. It’s such a wide area that encompasses so many different skills, Data Engineering that is. Why do we do what we do, write what we write? Like Augustine I see both old and new all around me, some things change, but many things stay the same.

People find hills like Python, Scala, Golang, Rust, and then promptly decide to die on them. I enjoy different things simply because of the way they teach you things about yourself and the world.

Read more

Exploring Delta Lake’s ZORDER, and Performance. On Databricks.

Photo by Joel Holland on Unsplash

I think Delta Lake is here to stay. With the recent news that Databricks is open-sourcing the full feature-set of Delta Lake, instead of keeping the best stuff for themselves, it probably has the most potential to be the number one go-to for the future of Data Lakes, especially within those organizations that are heavy Spark users.

One of the best parts about Delta Lake is that it’s easy to use, yet it has a rich feature set, making it a powerful option for Big Data storage and modeling. One of those features that promise a lot of performance benefits is something called ZORDER. Today I want to explore more in-depth what ZORDER is, when to use it, when not to use it, and most importantly test its performance during a number of common Spark operations.

Read more

Thoughts on HTTP and JSON with Golang. And other Headaches.

Photo by Nik Shuliahin 💛💙 on Unsplash

I’ve been playing with Golang off and on for a few weeks, when I find the time, which is every few weeks between kids and fishing. I have become a little bit of a fan, wishing for more projects to take on with Go. It seems like a fairly straightforward language to pick up, the learning curve isn’t that bad, and it’s fast and powerful. I’ve found it a little more intuitive than Scala for example. I mean don’t get me wrong, nothing will take the place of Python in my life, but there’s always room for one more.

That being said, “But I have this against you…” when it comes to Go, and it has to do with JSON. All code is on GitHub.

Read more