Copyright © 2011 http://jtonedm.com James Taylor
I have been thinking about IBM’s Watson for a while now. I met some of the team very early in their development and here we are today with Watson slugging it out on TV. To do this it must decompose the question, generate multiple hypotheses and then score them before synthesizing an answer and associated confidence. Watson has a number of interesting elements of relevance to those of us working with analytics and decision management but the linkage is not as direct as you might think. First, remember that IBM has not listed its predictive analytics technology (SPSS) or its optimization technology (ILOG) or business rules (ILOG) as part of the Watson solution stack – the focus has been on the hardware on which it runs and the natural (or, given we are talking about Jeopardy questions here, unnatural) language processing to figure out the question. Nevertheless Watson has some critical lessons to teach us about applying analytics more generally.
- Watson uses many different techniques to assess the evidence and data it has. This is analogous to the use of ensemble models in analytics (combining the analytic predictions of multiple techniques to create an ensemble model that is more accurate than the component pieces). This is increasingly widely seen as an analytic best practice as Dean Abbott discussed in the webinar he and I did recently – 10 Best Practices in Operational Analytics.
- The confidence of its predictions is key to Watson’s success – it calculates the confidence of many answers and uses this information in synthesizing an answer. Given that predictive analytics “turn uncertainty about the future into usable probabilities” it is important to always understand how confident you are in a prediction before you use it. Indeed one of the reasons for using a business rules management system to implement analytics is that it allows you to selectively implement only those elements, of a decision tree say, in which you are suitably confident (as Dean discussed in this presentation at Predictive Analytics World a little while back.
- Speed – Watson demonstrates that real time is possible, even for very complex problems. The days when analytics could only be applied in batch or by scoring the database overnight are long gone. You should be thinking in terms of real-time scoring with the data available at the point of decision and nothing less. Your problems are likely to be much simpler than Watson’s so you won’t need anything like the hardware involved either.
- Incremental improvement over time is critical. When Watson was first put together it could not compete with Jeopardy winners. But the great thing about analytics, especially analytics applied to a decision you make a lot (like answering Jeopardy questions if you are Watson) is that allows you to test new approaches, learn from those tests, accumulate data about what works and what does not and get better. This continuous improvement mindset is what you need in any application of analytics to operational systems
A final note. There are those who say that because analytics are not right 100% of the time and because the future is not completely predictable, predictive analytics should not be used. Remember though that even the most successful Jeopardy winners don’t get every question they answer right nor do they even try to answer every question – even grand champions are only managing 90% accuracy on about 65% of the questions. If you could come up with an analytic decision that was better than the human who would otherwise make the decision two thirds of the time then you too could be a winner.
There’s lots more about Watson at ibm.com/innovation/us/watson/
