Man vs. Machine Contests: Forget “Level” Playing Fields

Just got back from Seattle on a red-eye, and I take off for Paris on another red-eye Sunday night. I’m not complaining, but maybe I should buy myself one of those pillowy donuts to wear around my neck. I had a great time in Seattle, four great interviews, a good talk at Microsoft, and an event last night at Third Place Books. As an added bonus, Jeopardy all-time great Ken Jennings showed up toward the end, with his son Dylan in tow. (After we’d been talking for a while, my cousin, who lives nearby, asked Ken if he’d ever been on Jeopardy...)

Just got back from Seattle on a red-eye, and I take off for Paris on another red-eye Sunday night. I’m not complaining, but maybe I should buy myself one of those pillowy donuts to wear around my neck. I had a great time in Seattle, four great interviews, a good talk at Microsoft, and an event last night at Third Place Books. As an added bonus, Jeopardy all-time great Ken Jennings showed up toward the end, with his son Dylan in tow. (After we’d been talking for a while, my cousin, who lives nearby, asked Ken if he’d ever been on Jeopardy...)

Anyway, one issue that came up at nearly every stop: If Watson had a faster finger on the buzzer, was it a “level playing ground”? My feeling is that when you put people into competition with machines, no such field is possible. Each side has different strengths. Yes, Watson was faster than the humans on the buzz. If Watson had confidence in a response, it pressed the button within 10 milliseconds. To beat that, humans had to anticipate the activation of the light. This is a technique that Ken and Brad mastered while annihilating fellow humans. But it was tough to beat Watson to the buzz.

However, the humans enjoyed other advantages. English comes naturally to them. They didn’t have to parse every sentence, going for a mad hunt to figure out what they were supposed to be looking for. During one of countless discussions leading up to the match, IBM scientists told Jeopardy execs that if they wanted a “level playing field” the humans should have to field a at least a few clues in Watson’s native “language” of ones and zeros.

Watson has a vast database and nearly instant recall of facts and figures. But it doesn’t “know” or understand any of this information. It can only calculate its responses through statistical correlations. That’s a sizeable handicap. It also has no life experience and no body, not to mention a sense of humor.

In short, there’s no way to make the match truly fair and even. It’s impossible to add strengths to each side. So the only way to do it would be to impose handicaps. Yes, they could slow down Watson’s finger. But could they handicap the humans to take away some of their unfair proficiency in language? I don’t think that would be much fun.

You could argue that Watson wouldn’t have won without the fast buzzer, and I’m sure it’s true. But then again, five years ago, a fast buzzer wouldn’t have made any difference, because no machine on earth could discipher Jeopardy clues, hunt down facts, formulate answers and calculate its confidence in them within three seconds. A fast buzzer only mattered because Watson, after four years of research, was finally coming up with lots of correct answers. The value of the match, from my perspective, was that it showed the world just how far this type of question-answering machine has come–while also giving us a glimpse of its vulnerabilities.