Managing Data Scientists

With the rise of the ‘Data Scientist’, a lot has been said about the definition, role, qualifications and skills of the Data Scientist, and how to hire them. A somewhat neglected topic is how to manage data scientists. Indeed, data scientists, by their very nature, are hard to manage.

They love to resolve problems, but those problems are not always the business problems you want them to tackle. They are ace players, but they’re not always the best team players and some of them can sometimes have difficulty in dealing with (higher) management. They can have bright ideas, but they often lose interest when it comes to implementing those ideas in a profit making activity. They will find clever solutions for you, but they don’t always excel in making sure that a structured process is place, let alone the administrative follow up that comes with it. Some of them were hired as ‘rock-stars’ and have developed an ego that goes with that…

On the other hand, they are (sometimes) the ‘heroes’ of the company so you need to deal with it, it comes with the territory, as they say. Also, very often you can’t apply the usual bag of tricks that ‘ordinary managers’ can use, simply because these tricks don’t always work with them.

If your data scientists are all well behaved in this respect, this blog post is not for you. If you have experienced the issues I described above, read on!

One of the things I picked up early on as a manager was that a good manager should help his people rather than command them. Often I found myself doing things that my reports were asking me to do rather than doing what my manager was asking me to do. Mind you that I would take the general strategy and direction from my manager or people above her/him, but to make it happen I found it often more useful to listen what people who were closer to reality were saying. I would help them to make them more efficient in achieving my goals. And my goals were generally the goals of my boss. I’ve always tried to avoid micro-management and over reliance on procedures. But I will admit that in some cases I did micromanage and I did emphasize procedures. The thing is that I only did that when a certain unit was in problems, not when it was successfully achieving its goals.

Another thing I noticed is that data scientists, but also statisticians and some top coders, often have difficulties in accepting orders from managers who don’t have technical skills themselves. This does not mean that they would publicly disobey, but rather they would use some technical excuse to do whatever they wanted to do, knowing very well that the manager didn’t have the technical knowledge to challenge them. Coming from an IT and statistics background gave me (just enough) credibility to be taken seriously, and that gave me a head start compared to other managers.

But nonetheless, I had my share of problems managing data scientists.
When I was working for a large market research company a few years ago, I had to work with a lot of statisticians and the like. Some of them were direct reports, some of them indirect and sometimes, horror oh horror, we were acting in a matrix organization. I believe I had some credit with them because I was able to speak the same (technical) language as they did. But still I had difficulties in making sure standard procedures and administrative follow up was done correctly. Now there are two opposite ways to react in such a situation. On the one hand, you can put all your energy in making sure the administrative procedures are followed, or, you can let go of any administrative follow up completely. The former will make it very hard for you to get your ace players on board, because they generally hate this stuff, and the latter might cause problems with higher management, might create chaos and is seldom sustainable. So, as most things in live, the truth is somewhere in the middle. But how do you prioritize?

When I tried to explain my vision on these things I found it useful to use the following schema:

This rule has helped me in focusing on the priorities by not trying to force successful people and groups in a very rigid process driven structure, but on the other hand it was also a warning for those people and groups that they could only get away with it as long as they were successful. This rule also took some of the fear out my teams that were in trouble. If they were in trouble but they followed the normal procedures, there was no reason for fear. On the contrary, I would help them in resolving the problem. I’m sure this might have led to some situations that you might call micro management, but at least it was micro management applied on disfunctional groups and it would leave the successful ones doing whatever they were doing. Essentially there’s nothing new with this rule and I guess you can’t apply it to all situation or in all industries.

But for me, it worked.

image: complicated data science/shutterstock