The Transformational Power of Big Data For HR

Big Data is one of the most used tech buzzwords of the last five years, and the interest is growing, yet only 20% of companies use it for various operations. The laggards cite lack of knowledge (51%) or absence of data (30%) as the main reasons for not using this new tool.

Predictive analytics is slowly replacing experience and gut feeling in areas like investment, crime-fighting and human resources. Over half of HR departments use some type of Big Data powered tool in their daily operations. The rising popularity is also due to a more agile approach to work and the affordability of cloud-computing solutions.

Addressing top HR challenges

Big Data is just a tool, albeit a popular one, with numerous powers, but only useful if the underlying problem is well-defined. In HR, the main challenges are related to candidate filtering, retention, evaluation and reducing bad hiring. The good news is that algorithms can at the very least assist in each of these steps, if not perform them entirely.

The most time-consuming part of HR is filtering. Deep learning and natural language processing can select promising candidates based on the content of their resumes. Those who provided relevant key-words and have the required qualifications are chosen first, but those who would fit in with the team, based on their choice of words are also given a chance. Sentiment analysis can indicate a more assertive personality or dangerous levels of pugnacious behavior.


Big Data can also show the sources (schools, competitors, etc.) that yield the best candidates based on their skill level, the estimated time it takes to train them and dedication to the company. Retaining staff is done by analyzing the risk for them to change employers based on their job descriptions, salaries, involvement in team-work and other predictors. Creating mathematical models is an objective way of assessing HR and more accepted by employees since they don’t suspect personal management bias.

Collecting data regarding questions and answers during hiring and exit interviews helps companies create a repository of relevant inquiries and eliminate those subjects that don’t have a correlation to the later performance of the employee. An example of a traditional validation tool that is slowly being abandoned is the IQ test.

Setting the right metrics

The real question is what KPIs are still valid today for the workforce? In the past, the corporate environment was plagued with vanity metrics, that looked good on annual reports, but brought little value to the company, such as the percentage of middle managers with a college education.

Data should offer actionable insights and the only metrics that deserve a place on the company’s dashboard are those that show progress or some which act as red flags.


The simple test in keeping a metric or not is: “Can we connect this to measurable output or ROI?” Even in the case of soft skills, the right metrics will pass the test by showing an increase in client retention rates or customer relationship improvement.

The power of analytics

Having numbers as the foundation for various decisions instead of hunches and customs is the primary advantage of using analytics versus classical HR. Now cloud-based tools offer access to information and HR managers can take a more strategic approach to their problems.

Since a full-time employee means more costs to a company than the salary and benefits, there is enough incentive for businesses to protect their investment. This is the motivation behind long and dreadful interview rounds. Big Data can offer real-time information about employees’ behaviors, performance rates, and even mood changes (think voice stress analysis). This is as close as possible to reading their minds and knowing if they intend to keep the current job, are motivated to perform, or they are already thinking about an exit strategy.

Create a unitary system

Having such a solution in place can seem like an overwhelming investment. Yet, big data consulting firm InData Labs highlight the advantages of using a “Data lake” solution, if money is an issue and creating a new system is considered too expensive or unfeasible due to various compliance regulations.


The difference from a data warehouse consists of the fact that a data lake acts more like a centralizing unit, a hub for information stored in different places, not necessarily structured, while a data warehouse demands a tabular arrangement of information. The data-lake can be easily scaled and upgraded and can be added on top of existing legacy systems.

Manage data for new insights

These tools can evaluate the current situation, but also create forecasts related to staffing levels, the expected job openings and the necessary skill sets of future employees. One of the most pressing problems of the HR department is to find excellent employees to fill IT and Data science positions. These are highly skilled specialists in high demand, and there is currently a shortage of such employees.

Another direction which is currently under development and is even regarded as controversial by some is the territory of sensors and IoT. Installing such devices in the workplace is supported by work security opportunities, especially in high-risk environments, like oil platforms, mines and production plants. Yet, some employees could consider this an intrusion into their privacy and a Big Brother approach to management. The role of the HR department is to inform potential candidates about such practices and only proceed with those ready to accept it. The data collected can also serve to improve the quality of work.


Time for a change?

If your company is not currently using a Big Data solution for HR purposes, although even LinkedIn counts as one, it is time to start thinking about ways of leveraging the data you already have in files, applications, e-mails and even recordings of interviews. This 6-step guide to organizing data is a good place to kick-off the data project if you are just learning the basics. Remember, that the quality of the analysis and the relevance of the algorithms’ outputs are never better than the training data. Security and compliance to handling personal data are top priorities and should be in place before launching the program.


Solution Architect for 8+ years with experience in software consulting. Focused on It solutions for marketing, healthcare, financial sector and a few others.