So, You Want To Be a Data Engineer

Then You Better Be Pedantic. B E Pedantic

Not a Medium member? No problem. Read free here.

Photo by Jennifer Uppendahl on Unsplash

Actually, no one wants to be that guy or that gal who always corrects their coworkers and business partners. It sucks. Yet, assuring correct definitions of metrics and processes is critical in data engineering. It separates leaders from ticket takers and prevents confusion and inaccurate metrics. As a data engineer you must take opportunities to be pedantic.

What does that mean? Well dictionary.com defines pedantic as “overly concerned with minute details.” It is generally used in a negative connotation. “Stop being a pedant.” With that in mind, let me ask you a question. Shouldn’t we be concerned with minute details? Of course we should. Details are what make the difference between directionally correct and accurate. Details are important.

Let’s Discuss A real example

Here’s a real situation I ran into many years ago. Working with the capacity manager for Intel’s internal IaaS platform, we managed a global VMware environment with compute clusters spread around the globe. While this might sound glamourous, it was mostly munging data in Excel and attending meetings. ): Senior leadership measured us on cost efficiency and wanted to know our average…