Transformational Data Science
One data scientist I follow on LinkedIn is Eric Weber, Head of Data Product for Yelp. In a recent post, he wrote, ‘An interesting problem may have no business value, while a boring problem may be transformational for your company.’ This has been my experience and it really stuck with me, although I might substitute the word ‘solution’ for ‘problem.’ I have often labeled complex modeling as ‘interesting,’ and basic exploratory data analysis (EDA) or basic descriptive statistics as ‘boring.’ As a more experienced data scientist now, I’ve changed my view substantially.
I think it’s easy to assume that a complex solution has more value. Because it takes more time to implement, right? So it must be better? That often isn’t the case. In business, it’s not just the solution itself, but other ecosystem factors come into play, such as time to market, existing solutions in the market from competitors, explainability requirements of solutions, and others. As a trainee, it’s important to shed assumptive labels for different problem solving approaches. More complex doesn’t always mean better. More complex doesn’t always equate to interesting, either.
For data scientists entering the job market, I would encourage you to think about the ‘why’ behind your solutions for different projects. Often, it might be because that was the appropriate model for the type of data you had available. But if you had it to do over again, would you implement a different solution? What is the incremental value gained by a more complex model over a more simplistic solution that might take less time? Talking through this reasoning is one of the hallmarks of an experienced data scientist.
Several years ago, I was watching Food Network, and a baker mentioned that a very elaborate cake was very easy to make because you could hide your errors very easily. Basic cakes with little decoration were much more difficult. I feel the same way about data science—doing the basics is very hard because you must be disciplined. You must do basics somewhat flawlessly. And that is indeed transformational.
