The Fallacy of the Data Scientist Shortage
There is no question that the USA (in fact, most of the world) would be well-served with more quantitatively capable people to work in business and government. However, the current hysteria over the shortage of data scientists is overblown. To illustrate why, I am going to use an example from air travel.
On a recent trip from Santa Fe, NM to Phoenix, AZ, I tracked the various times:
There is no question that the USA (in fact, most of the world) would be well-served with more quantitatively capable people to work in business and government. However, the current hysteria over the shortage of data scientists is overblown. To illustrate why, I am going to use an example from air travel.
On a recent trip from Santa Fe, NM to Phoenix, AZ, I tracked the various times:
As you can see, the actual flying time of 60 minutes represents only 19% of the travel time. Because everything but the actual flight time is more or less constant for any domestic trip (disregarding common delays, connections and cancellations which would skew this analysis even farther), this low percentage of time in the air is a reality. For example, if the flight took 2 hours and fifteen minutes, it would still work out to 135/386 = 35%. The most recent data I have, from 2005, shows the average non stop distance flown per departure was 607 miles, so we can add about 25 minutes to the first calculation and arrive at 85/336 = 25%.
Keep in mind, again, these calculations do not account for late departures/arrivals, cancelled and re-booked flights, connections, flight attendants and pilots having nervous breakdowns, etc. It’s safe to say that at most 25% of your travel time is spent in the air. Just for fun, let’s see how this would work out if we could take the (unfortunately retired) Concorde. We would reduce our travel time by flying at Mach 2.5 by 40 minutes, trimming out journey from five hours and eleven minutes to four hours and 31 minutes, about a 13% improvement.
What’s the point of all of this and what does it have to do with the so-called data scientist shortage?
Based on our research at Constellation Research, we find that analysts that work with Hadoop or other big data technologies spend a significant amount of time NOT requiring any knowledge of advanced quantitative methods – configuring and maintaining clusters, writing programs to gather, move, cleanse and otherwise organize data for analysis and many other common tasks in data analysis. In fact, even those who employ advanced quantitative techniques spend from 50-80% of their time gathering, cleansing and preparing data. This percentage has not budged in decades. Keep in mind that advanced analytics is not a new phenomenon; what is new is the volume (to some extent) and variety of the source data with new techniques to deal with it, especially, but not limited to, Hadoop.
The interest in analytics has risen dramatically in the past two or three years, that is not in dispute. But the adoption of enterprise-scale analytics with big data is not guaranteed in most organizations beyond some isolated areas of expertise. Most of the activity is in predictable (commercial) industries – net-based businesses, financial services, and telecommunications, for example, but these businesses have employed very large-scale analytics, at the bleeding edge of technology for decades. For most organizations, analytics will be provided by embedded algorithms in applications not developed in-house and third-party vendors of tools and services and consultants.
The good news is that 80% of the expertise you need for big data is readily available. The balance can be sourced and developed. “The crème-de-la-crème of data scientists will fill roles in academia, technology vendors, Wall Street, research and government.
There are related and unrelated disciplines that are all combined under the term analytics. There is advanced analytics, descriptive analytics, predictive analytics and business analytics, all defined in a pretty murky way. It cries out for some precision. Here is how I characterize the many types of analytics by the quantitative techniques used and the level of skill of the practitioners who use these techniques.
“Data Scientist” is a relatively new title for quantitatively adept people with accompanying business skills. The ability to formulate and apply tools to classification, prediction and even optimization, coupled with fairly deep understanding of the business itself, is clearly in the realm of Type II efforts. However, it seems pretty likely that most so-called data scientists will lean more towards the quantitative and data-oriented subjects than business planning and strategy. The reason for this is that the term data scientist emerged from those businesses like Google or Facebook where the data is the business; so understanding the data is equivalent to understanding the business. This is clearly not the case for most organizations. We see very few Type II data scientists with the in-depth knowledge of the whole business as, say, actuaries in the insurance business, whose extensive training should be a model for the newly designated data scientists (see my blogs “Who Needs Analytics PhD’s? Grow Your Own” and “What is a Data Scientist and What Isn’t.”)
Trending Now1  SmartData Collective Exclusive 51 shares2,150 views1 51 shares2,150 views1 4 Ways R Developers Are Solving Business Analytics ChallengesRehan Ijaz - September 12, 20182  Big Data 45 shares2,950 views 45 shares2,950 views An API Bridge Between Big Data And Digital MarketingPhilip Volna - September 10, 20183  Artificial Intelligence 63 shares2,582 views 63 shares2,582 views Examining The Positive And Negative Impacts of AI On EducationPhilip Piletic - September 10, 20184  Analytics 71 shares3,247 views 71 shares3,247 views Big Data Helps You Use LinkedIn Sponsored Content CompetitivelyRyan Kh - September 8, 20185  Big Data 51 shares3,020 views 51 shares3,020 views Fascinating Ways Big Data Is Reshaping The Future of FootballSean Mallon - September 8, 20186  Big Data 31 shares2,694 views 31 shares2,694 views How Big Data And Education Can Work Together To Help Students ThriveRyan Ayers - September 7, 20187  Blockchain 77 shares2,733 views 77 shares2,733 views Bringing Together Blockchain, eCommerce and ContentDaniel Matthews - September 6, 2018Hot Posts2  Artificial Intelligence 78 shares2,407 views 78 2,407 Why Small Businesses Shouldn’t Think Twice Before Embracing AIJennifer Warren - September 20, 20183  Analytics 66 shares2,265 views 66 2,265 Predictive Analytics Makes DasCoin And Other Currencies MainstreamDiana Hope - September 19, 20184  Machine Learning 31 shares2,578 views 31 2,578 Machine Learning Is Driving The New Digital Marketing RenaissanceRyan Kh - September 19, 20185  Big Data 53 shares2,267 views 53 2,267 4 Data-Backed Strategies To Drive More Traffic To Your eCommerce SiteManish Dudharejia - September 18, 2018Latest Developers’ Outlook On Artificial Intelligence In 2018September 20, 2018 Why Small Businesses Shouldn’t Think Twice Before Embracing AISeptember 20, 2018 Predictive Analytics Makes DasCoin And Other Currencies MainstreamSeptember 19, 2018Popular User Adoption – Resistance Is Futile, We HopeJanuary 10, 2012 Data Visualization: A Scientific TreatmentNovember 7, 2014 |
You must log in to post a comment.