My Journey from Data Analyst to Senior Data Engineer

My newsfeed these days is chock-full of “how to break into Data Engineering” these days. It’s made me a bit nostalgic, to say the least. I’ve been dreaming about those days gone by when I started out in the data world. I would say my experience was not so much “breaking in”, but more of a “weasel my way into” Data Engineering.

I didn’t get a Computer Science degree, not even close. I think there are many ways to get into Data Engineering, it’s probably easier than it ever has been in the past. We will fulfill our destiny in different ways, and that journey gives us a unique perspective and makes us “good” at certain things. This is my story.

In The Beginning …

Photo by Alexander Andrews on Unsplash

In the beginning, God said, “Let There Be Daniel.” Fast forward 16-18 years. I’m in high school living in the middle of nowhere doing many things kids these days only see in movies. Then something fortuitous happened in circa 2000, my brother bought me something.

Things were never quite the same after that. I lived two lives … one in the woods and gravel woods.

If you don’t like the picture of me and a gun, go read someone else’s blog or start your own. We are all humans and can have different perspectives and backgrounds without being idiots to each other.

The other life was with my precious computers.

I wasted a lot of time in high school playing World of Warcraft, Doom, and then eventually Halo and Battlefield 1948 etc. Still no programming yet.

College

After starting out with a major in Forestry, I switched to Agriculture Systems Technology, mostly because it involved two things I knew about … farming and technology. This is where my love for computers started to flourish. I was introduced to Perl, PHP, and MySQL, mostly for web development. I ended up working my way through college writing code, even becoming the WebMaster for the student council, running and building websites.

I remember being blown away by the power of PHP on the web, I would stay up all night writing PHP for the fun of it.

Post-College – Circa 2008

After college went to work for a large agriculture equipment company. I worked in manufacturing, it was grueling and hard work, although I got to travel the country and overseas. I still did SQL and PHP on the side, a few contract jobs here and there. Then things started to slowly change.

Being “that person” who knew anything about code at work started me down a long path. This started out as writing VBA and all sorts of macros to automate analyst stuff. Circa 2009-2010. I quickly ended up working on SQL Server managing and querying all sorts of data for manufacturing and writing lots of VBA to make my life easier.

Photo by kate.sade on Unsplash

By the time 2013 came around, I was done with manufacturing and endless cubicles run by giant multi-national corporations who wanted to suck every last drop of life out of me. I knew I had to do something drastic because I couldn’t imagine the rest of my life like this. I decided I was going to do what it took to break into the Data world.

I started practicing the crap out of SQL, became an expert, to the point I could write recursive CTE’s to explode bills-of-material. I knew I would be behind the 8-ball with no degree in technology, but I figured I could work harder than most people, and that combined with a little luck I might be able to break into a job.

The Switch

I applied to every Data Analyst job I could find. I was living in Wisconsin between Milwaukee and Chicago, so circa 2014 there were more Data Analyst jobs than you could throw a stick at on Indeed. I threw sticks. All the while I lived and breathed SQL and data outside of work, and at work when I could. I read everything I could online about data analysis and SQL Server specifically.

Then I got my big break. I got an interview for a Data Analyst job at a regional bank. By this time I could write SQL like a bandit, and I proved it to them in the interview. I wanted that job so bad I could taste it, I was also scared out of my mind.

Photo by Hello I’m Nik on Unsplash

I followed up after the interview like you are supposed to, answering questions more in-depth that they had asked me in the interview. I got the job. I couldn’t believe I had done it, it seemed crazy to me. Switched into a whole new industry, from manufacturing to tech. That created in me a deep-seated will to strive and prove myself … something that never has gone away.

Data Analyst – Circa 2014

The bank I ended up as a Data Analyst at was a SQL Server + SAP Business Objects shop, I dove in headfirst. My first boss …. was a boss of SQL, he taught me to write SQL that was hundreds of lines long … that’s just what we did. I worked with smart ETL and Warehouse Developers who were writing SSIS packages and writing C# to get things done. I worked with SAP Business Objects experts and Report Writers.

I didn’t just stay in my lane. I learned it all, I ate it all, ingested it.

I learned out to write SSIS packages, learned ETL and data transformations, I learned how to write the most complex SQL possible. I learned about analytics and report writing. I learned about what made SQL Server tick under the hood. I learned how to manage SAP Business Objects servers.

I learned how to talk to people and translate business requirements into code and eventually usable output.

Around this time I picked up some basic Microsoft SQL Server certifications, I learned more C# discovered, and started playing around with programming more.

Senior Data Analyst – Circa 2015

After 1.5 years at the bank, I was promoted to Senior Data Analyst. I worked hard, was picking up C# and Python on the side, a little at work. Was proficient at SQL Server, SSIS, writing business intelligence reports, SQL wizard, and SAP and Business Objects wizard.

At this point I was married and had a kid … I wanted more. Sometimes when you grow up with nothing, you work hard for everything. What wouldn’t we do for our families?

Circa 2015 Business Intelligence was a hot new topic, I knew this was my chance. Two birds with one stone. Move back to Iowa closer to family, more help with raising the kid, try to land a Business Intelligence Developer type role. I loved programming, data, and ETL, SQL Server, and Data Warehouses. I was always learning more and reaching for the best at what I was doing.

It didn’t take long this time around to make the switch.

Business Intelligence Engineer – Circa 2016

Within a 6 week span, I started applying for jobs in Iowa and got a few interviews, I landed a job shortly after. My first Business Intelligence role. I drank the Kool-Aid and went down the Data Warehouse and Python rabbit hole. I was blessed to land a job at a small manufacturing company. The great thing about moving from the rigid Banking sector to this small business what that I got to do everything.

  • Kimball style Data Warehousing
  • Running and managing SQL Server
  • Running and managing SAP Business Objects services and tools.
  • Writing Python scripts to scrape data from API’s.
  • Writing SSIS packages to ETL data.
  • Gathering business requirements.
  • Working closely on analytics with different business units.
  • Data Modeling
  • Tableau/Analytics

At this point, I had taken a few TalkPython classes and was obsessed with Python. Circa 2018 I was hardcore all things data, data warehousing, Python and the like. I started a blog and GitHub and was doing the stuff 24/7.

Data Engineer – Circa 2018

By this time the Data Engineer was the new popular thing, and I felt like the Business Intelligence Engineer had lost its luster. Although, depending on the role, including mine, I was doing the exact same thing as any Data Engineer, maybe even more so. I was writing Python for production use, building Data Warehouses from scratch, providing Analytics, managing servers, and building ETLs.

I could see the future was with Big Data and cloud tools like GCP and AWS, as well as Python. So, once again I wasn’t going to let fate have its way with me … I went a-hunting. I landed my first Data Engineer job at a startup.

  • AWS
  • DevOps
  • in-depth SQL and tuning
  • Analytics and Data Warehousing
  • Spark/Glue
  • Python best practices, APIs.
  • Dealing with terrible leaders and principal engineers

By this time I had been faithfully blogging, and putting stuff into my personal GitHub account, I was learning a lot about software engineering best practices, OOP, testing etc. I also was diving hard into Apache Spark and other big data tools, learning as much about AWS and GCP as possible. Also, picking up things like Docker and learning CI/CD best practices.

Photo by James Lee on Unsplash

I learned a lot about software design and working the cloud working at my first startup. I also learned a lot about toxic culture and terrible engineering leaders and principal engineers who were not very kind, nice, and all around horrible to be around. I learned what and who to avoid, which was an important lesson for anyone in tech.

Second Data Engineer Job – Circa 2019

After barely a year I jumped ship from that terrible and toxic culture to a more established startup, again as a Data Engineer. I found a company that treated people with respect and where people were actually happy to be there and work. It was my first taste of the true startup culture that you always read about, beer, ping-pong, free meals, and wonderful people.

Photo by Andre Hunter on Unsplash

I grew a lot in this second Data Engineering job. I started to learn about Machine Learning and MLOps, picked up Kuberentes and Airflow. I started to be able to deliver massive projects a sole engineer, enabling Data Scientist to bring their dreams to life and production. I basically honed all the skills I already had and picked up some new ones around Machine Learning and MLOps.

  • DevOps
  • Big Data
  • Kubernetes
  • hardcore Python
  • Docker
  • Airflow
  • Planning and executing big projects.
  • Machine Learning and MLOps

I was still writing lots of blogs and putting a lot of stuff on GitHub. After getting a little bored and wanting to do more Spark I started looking again. I knew I was performing at the level of Senior Data Engineer, even though my title didn’t say it. I wanted to the title.

Senior Data Engineer – Circa 2020

By this time the market was super hot for Data Engineers, and it was really a matter of just finding the right job. Which I did without too much problem, another startup. I wanted to be pushed to the do the impossible, learn new skills and provide a ton of value. I get a chance to do that every day, to this day. The role of a Senior Data Engineer has a lot to do with being able to solve complex problems and write a lot of code. But it’s so much more than that.

Photo by Ivan Diaz on Unsplash

Sure, technically I have to do a lot of things other Engineers do.

  • Airflow
  • Databricks + Spark
  • Delta Lake / Data Lake / Warehouse
  • Python
  • AWS
  • CI/CD
  • Machine Learning Ops
  • Architecture

But, it also includes a TON of non technical skills, that appear to be more important than the technical.

  • Project/JIRA planning and managment
  • Communicating with other people often
  • Training and upskilling other folks to do difficult work
  • Documentation
  • Developer experience improvments
  • Creating/documenting and implementing Standards and Procedures

And the list goes on. To be a good Senior Data Engineer requires a lot of technical skills, but it also requires more technical skills. Being able to deliver projects on time in a Startup requires give and take. It’s about thinking ahead, planning, making tradeoffs, the right tradeoffs. It’s about implementing best practices and procedures and sticking to them yourself, holding yourself accountable.

It’s about continuous learning and growth. It requires a lot of reading … documentation, blogs, books, keeping up on the trends and every changing directions and tooling. It’s learning to hold those technical hills you want to die on more loosely, seeing the big picture and finding the middle ground.

How you can become a Senior Data Engineer.

I get a lot of questions and messages from other engineers on either breaking into Data Engineering or growing up in levels. Everyone’s journey is different and I wouldn’t suggest anyone follow any prescribed formula, if someone is trying to sell you one, it’s probably not going to work out like you think.

Here are some suggestions from my point of view and my experience. Take them as you will. If you are a Data Engineering or want to be one, do things things ( at your own risk).

  • Change jobs often (1.5 years and then move on). It’s easier to get title change by moving rather than staying.
  • Have a personal GitHub account where you write code for yourself.
  • Create and maintain a free blog where you document your thoughts and learnings.
  • Read a lot. Blogs, books, and other free material.
  • Learn, or try to learn, 1 new language a year. Just for fun.
  • Work hard when you are at work. Harder than anyone else around you.
  • Take the weekends and nights off. Avoid burnout.
  • Be ok with the fact there will always be people smarter than you. Who cares. Control the controllable, you. Learn from them.
  • Kick imposter syndrome to the curb. Understand that you bring a unique set of experiences and skills to any job.
  • Be ok starting as a Data Analyst and working your way up the chain.
  • Love what you do.
  • Practice the soft skills, just as much as you practice the technical.

That’s my story and I’m sticking to it.