Will Dwinnell: 6 Reasons You Hired the Wrong Data Miner

December 19, 2012
12 Views

By Will Dwinnell

By Will Dwinnell

As is in any discipline, talent within data mining community varies greatly.  Generally, business people and others who hire and manage technical specialists like data miners are not themselves technical experts.  This makes it difficult to evaluate the performance of data miners, so this posting is a short list of possible deficiencies in a data miner’s performance.  Hopefully, this will spare some heartache in the coming year.  Merry Christmas!

1. The data miner has little or no programming skill.

Most work environments require someone to extract and prepare the data.  The more of this process which the data miner can accomplish, the less her dependence on others.  Even in ideal situations with prepared analytical data tables, the data miner who can program can wring more from the data than her counterpart who cannot (think: data transformations, re-coding, etc.).  Likewise, when her predictive model is to be deployed in a production system, it helps if the data miner can provide code as near to finished as possible.

2. The data miner is unable to communicate effectively with non-data miners.

Life is not all statistics: Data mining results must be communicated to colleagues with little or no background in math.  If other people do not understand the analysis, they will not appreciate its significance and are unlikely to act on it.  The data miner who can express himself clearly to a variety of audiences (internal customers, management, regulators, the press, etc.) is of greater value to the organization than his counterpart who cannot.  The data miner should should receive questions eagerly.

3. The data miner never does anything new.

If the data miner always approaches new problems with the same solution, something is wrong.  She should be, at least occasionally, suggesting new techniques or ways of looking at problems.  This does not require that new ideas be fancy: Much useful work can be done with basic summary statistics.  It is the way they are applied that matters.

4. The data miner cannot explain what they’ve done.

Data mining is a subtle craft: there are many pitfalls and important aspects of statistics and probability are counter-intuitive.  Nonetheless, the data miner who cannot provide at least a glimpse into the specifics of what they’ve done and why, is not doing all he might for the organization.  Managers want to understand why so many observations are needed for analysis (after all, they pay for those observations), and the data miner should be able to provide some justification for his decisions.

5. The data miner does not establish the practical benefit of his work.

A data miner who cannot connect the numbers to reality is working in a vacuum and is not helping her manager (team, company, etc.) to assess or utilize her work product.  Likewise, there’s a good chance that she is pursuing technical targets rather than practical ones.  Improving p-values, accuracy, AUC, etc. may or may not improve profit (retention, market share, etc.).

6. The data miner never challenges you.

The data miner has a unique view of the organization and its environment.  The data miner works on a landscape of data which few of his coworkers ever see, and he is less likely to be blinded by industry prejudices.  It is improbable that he will agree with his colleagues 100% of the time.  If the data miner never challenges assumptions (business practices, conclusions, etc.), then something is wrong.

You may be interested

3 Indispensable Ways to Reach Your Marketing Potential with Data
Business Intelligence
0 shares194 views
Business Intelligence
0 shares194 views

3 Indispensable Ways to Reach Your Marketing Potential with Data

Rohan Ayyar - May 23, 2017

Today, all marketing is done on the back of data. Marketers know their decisions and actions must have a rationale…

How Did Big Data Create a Modern Day Manufacturing Workforce?
Big Data
0 shares292 views
Big Data
0 shares292 views

How Did Big Data Create a Modern Day Manufacturing Workforce?

Megan Ray Nichols - May 23, 2017

Despite its relatively brief lifespan thus far, big data is poised to reinvigorate the U.S. manufacturing workforce. Some of the…

4 Easy Steps How to Conduct IT Security Audit of Your Own Company
IT
0 shares339 views
IT
0 shares339 views

4 Easy Steps How to Conduct IT Security Audit of Your Own Company

GoganMarcell - May 22, 2017

Businesses often view data security audit as a stressful and intrusive process. Auditor walks around distracting everybody and meddling in…