How Walmart is Tackling the Big Data Skills Crisis

As the amount of digital information generated by businesses and organizations continues to grow exponentially, a challenge – or as some have put it, a crisis – has developed.

There just aren’t enough people with the required skills to analyze and interpret this information – transforming it from raw numerical (or other) data into actionable insights – the ultimate aim of any Big Data-driven initiative.

One survey recently carried out by researchers at Gartner found that more than half of the business leaders they queried felt their ability to carry out analytics was restricted by the difficulty in finding the right talent.

Overcoming this problem is a challenge that all companies will have to face, and market leaders – aware that they have more to lose than many by falling behind in the race to keep up with technology – have come up with some innovative solutions.

Walmart decided to apply one of the fundamental weapons in the Big Data arsenal – crowdsourcing – to the problem, with positive results.

Last year, they turned to crowdsourced analytics competition platform Kaggle. At Kaggle, an army of “armchair data scientists” apply their skills to analytical problems submitted by companies, with the designer of the best solution being rewarded – sometimes financially, in this case with a job.

Mandar Thakur, senior recruiter for Walmart’s Technology division, told me “The Kaggle competition created a buzz about Walmart and our analytics organization. People always knew that Walmart generates and has a lot of data, but the best part was that this let people see how we are using it strategically.”

Crowd-sourced talent

Candidates were provided with a set of historical sales data from a sample of stores, along with associated sales events, such as clearance sales and price rollbacks. They were asked to come up with models showing how these events would affect sales across a number of departments.

As a result, several people were hired into the analytics team, and the competition was held again this year. This time candidates were asked to predict how weather would impact sales of different products.

This crowdsourced approach led to some interesting appointments of people who, as Thakur says, wouldn’t have been considered for an interview based on their resumes alone. One for example had a very strong background in physics but no formal analytics background. “He has a different skillset – and if we hadn’t gone down the Kaggle route, we wouldn’t have acquired him.”

Kaggle’s competitions are open to anyone – so what kind of person does an organization look for, when considering how to fill vacancies which could be vital to the future growth of the business?

Communication is key

“Fundamentally,” says Thakur, “we need people who are absolute data geeks – people who love data, and can slice it, dice it and make it do what they want it to do.

“Having said that, there is one very important aspect we look for, which perhaps differentiates a data analyst from other technologists. It exponentially improves their career prospects if they can match this technical, data-geek knowledge with great communication and presentation skills.”

In other words, as well as being able to wring unexpected and game-changing insights from the unlikeliest of data, they must be able to explain them to a room full of (often non-technical) business executives and marketing suits.

“Someone who has this combination of skills can rise to the top very quickly”, explains Thakur.

And by “the top”, he does mean the top – career pathways are in place for a new hire beginning an entry-level role as a data science associate to rise to senior director level, becoming an Enterprise Decision Scientist guiding the overall company direction through statistical analysis.

Physicists aside, most new recruits still come with a “traditional” background in the academic disciplines required for data analysis – statistics, mathematics, computer science and business analytics.

A working knowledge of Python or R – two of the programming languages most commonly used for analyzing large digital datasets, is also usually expected. The biggest challenge can be finding candidates with experience in the most cutting edge analytics applications, such as those involving machine learning. Many people will not have the opportunity to learn this at school, and experts are often self-taught.

Walmart also recently ran a recruitment campaign across social media using the Twitter hashtag #lovedata, to raise its profile among the online data science community and avert their eyes away from Silicon Valley and towards Bentonville, Arkansas, when scoping for job opportunities.

Analytics rotation program

Analytics and big data is now integrated into every vertical within Walmart, so once on the team, every new staff member with these responsibilities takes part in the Analytics Rotation Program – spending a period within every department to get an overview of how analytics is used across the company.

“This allows them to combine their analytical knowledge – whether they have gained it in education or in work experience – and helps to assimilate that knowledge with what Walmart is doing in different pockets of their business. Once they are on the ground running, they can run in the right direction.” Says Thakur.

Another valuable source of job applicants is referrals. Data fans – academic, industrial or armchair – are as active in online communities and social networks as any other breed of techies. Provide one with a great job, and he or she is likely to spread the word to their associates.

Unprecedented demand

The company’s ramping-up of recruitment into its analytical team coincides with the launch of Walmarts attempt to build the largest ever private cloud. This huge database will hold all of the data gathered from its online operations and more than 11,000 bricks and mortar stores. Once completed it is estimated that more than 40 petabytes of information will be collected, and available for analysis, every day.

This will undoubtedly lead to an unprecedented demand for people with the skills to turn this data into actionable insights, and to explain to others how to do so.

Thakur says “The staffing supply and demand gap is always there, especially when it comes to emerging technology.

“We have found innovative and creative ways to go about finding talent to carry out our data science and analytics, however we are always, always, always on the lookout for people who can come in, contribute and catapult our organization even further.”