Approaching Software as a Craft, then as a Engineer.

Craft first, engineering second. There’s probably a lot of software programmers, developers, and engineers who will take issue with this. That’s kinda the point. Software should be approached as a craft first, then a engineering problem second. There are so many ways this is true, it’s going to be hard to touch them all. I […]

Read more
, ,

3 (Or More) Ways to Open a CSV in Python

Ah. What a classic. The one piece of code that I end up writing over and over again, you would think I would have stashed it away by now. Not going to lie I usually have to Google it, while thinking, is this the right way? Should I just open the csv file and iterate […]

Read more

How Smart Engineers Create Bad Software

You ever wonder how a room full of what appears to be smart engineers manage to build software that doesn’t work? Given more time and money, it appears to only get worse or no better. It doesn’t make that much sense does it? As someone who writes software it’s hard to see how bugs that […]

Read more
, , ,

Thunderdome for Geospatial Tools in Python. It’s to the Death.

It’s a fight to the death people… that’s why it’s called Thunderdome. This will be no different. Last time we talked about the very basics of the strange world of geo-spatial tools for data engineering. The next most obvious thing do of course is to see what tool is the best. By best I mean […]

Read more
, ,

Gentle Introduction to Geospatial for Data Engineers

What does a data engineer need to know about working with geospatial data? I’m going to give my two cents on what is and is not important. First, prepare to be annoyed as you will most likely spend hours debugging strange and not obvious errors and bugs. You should run screaming the other way, but […]

Read more
, ,

Python Async File Operations – Juice Worth the Squeeze?

What I’ve greatly feared has come to pass. I’ve come to love on of the most confusing parts of Python. AysncIO. It has this incredible ability for data engineers building pipelines in Python to take out so much wasted IO time. It saves money. It’s faster. People think you’re smarter than you are. Tutorials are […]

Read more
,

Dask – To Distribute or Not To Distribute..Ahh..This Thing Sucks.

Raise your hand if you’ve every used Dask? ……. Me either. With tools like Spark, I’ve only recently started seeing some articles and podcasts pop up around Dask. Written in Python and claiming to be a distributed data processing framework I figured it was about time to check it out. Suprisingly when reading up on […]

Read more
,

Data Engineering a Machine Learning Pipeline In Real Life.

It doesn’t take long reading articles on Medium or Towards Data Science to become enamored with Machine Learning. Especially the people and companies who do it in “production.” I always read about the big picture, the fancy algorithms, the cloud computing, but you have that feeling there is something missing. It’s all the details that […]

Read more
, ,

Concurrently Download Large Files from GCS.

There is nothing more annoying than sitting around waiting for files to download. That was true while I was in high school staring at LimeWire, it’s still true today. Especially when you’re a data engineer who’s supposed to make data pipelines fast. You’re in luck! Yes, it is possible to download a large file from […]

Read more
,

Data Engineering vs Data Science – Where’s The Love??

It seems like a never ending battle for supremacy. Articles about Data Science being the bee’s knees, then more articles about how Data Engineering holds up the world of Data Science like Atlas. Whenever I read something in one of these two categories on Medium or wherever, it just seems more like ego clash to […]

Read more