First Test Of Computer Jeopardy Player Goes Well; Watson Beats Mere Humans

from the show-off dept

Fri, Jan 14th 2011 1:32am — Mike Masnick

Last month, we wrote that the IBM computing project Watson was ready to take on real Jeopardy contestants in February of this year (just a few years after the project first came to life). While the big test isn't until Valentine's Day, apparently they had a dry run, and things are looking pretty good for Watson, who beat Jeopardy champions Ken Jennings and Brad Rutter. It wasn't a complete domination. Apparently Watson just narrowly edged out Jennings, though Rutter couldn't keep up. Of course, between now and the real test, Watson can be tweaked. Jennings' and Rutters' brains are pretty much set, and I'd imagine that their ability to cram more useless trivia in their brains is outmatched by Watson.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: ai, jeopardy, watson
Companies: ibm

44 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Bonzo, 14 Jan 2011 @ 3:25am

It's surprising how far AI has come. Not sure how "true" this AI is though, as it's specially configured for the jeopardy contest.

Still, steps in the right direction. I wonder how long it will be before fully fledged AI computers will be interacting with us on a regular basis!
[ link to this | view in thread ]
Richard (profile), 14 Jan 2011 @ 4:06am

From My Introductory AI Lecture this morning
Before project:

"We will construct a program/robot that will perform task X (e.g. play Chess). The resulting program/robot will thus have artificial intelligence - since humans use intelligence to perform task X."

After project

Scenario 1

"We failed"

Scenario 2

"We succeeded - however on closer examination it turns out that the method our machine uses is really some form of brute force algorithm that can't really be called intelligent. It seems that you don't really need intelligence to perform task X after all."
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 4:23am

When they started losing to Jennings, they probably just tweaked the questions. "And now for another question about Justin Beiber..."
[ link to this | view in thread ]
John Doe, 14 Jan 2011 @ 4:38am

You seem excited by this
Don't you know once SkyNet gets started, it won't be stopped? We will be eradicated. Or worse, turned into copper top batteries. ;)
[ link to this | view in thread ]
bob563, 14 Jan 2011 @ 5:20am

Re: You seem excited by this
You will now be sued since "copper top battery" is owned by Duracell.
[ link to this | view in thread ]
Sean T Henry (profile), 14 Jan 2011 @ 5:48am

Re:
I do not really consider this being a big thing. This is like saying a 6th grader beat Ken Jennings using only his skills to type the question in Google.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 5:52am

This is no real contest I'm afraid. This machine is so different from Big Blue that competed in chess. This is simply a search and seizure operation, you ask a question and it searches it's enormous database and finds the answer. For all we know it may just Google it to get the answer. The computer has an unfair advantage unless you give the humans the same access to the data.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 5:57am

Re:
if you even bothered to read the article, you'd notice that it says it doesn't have access to the internet
[ link to this | view in thread ]
antimatter3009 (profile), 14 Jan 2011 @ 6:17am

Re: Re:
Jeopardy questions are not straightforward. What's impressive here isn't finding the answer, it's figuring out the question. Natural language processing is very difficult, and Jeopardy provides a particularly difficult version of it, too, with all the puns and such.
[ link to this | view in thread ]
Ryan Diederich, 14 Jan 2011 @ 6:53am

Much more complex...
The programming here is much more complex than many realize. This thing has to be able to define the categories, and then figure out what the definition of the category means to each question.

For example, a question might read: 1812

The AI would have to remember that the category was "wars in history" and then apply that to its database search.

Of course that one was simple, there are many more complex ones that I cant think of.

I do remember one category that was, in other words, "Poets with the same letters for their first and last name"

Now, unless you were expecting this category, how on earth could you program something to realize what that means.
[ link to this | view in thread ]
J\, 14 Jan 2011 @ 7:41am

Re: Much more complex...
[ link to this | view in thread ]
Jason, 14 Jan 2011 @ 7:44am

More difficult than that would be the intuition type categories like Rhyme Time and Before and After. In these cases you have to get the idea of how the answers work before you can start answering correctly.

What I'm wondering: Are they trying to anticipate common categories and programming this type of information into the computer or simply relying on some kind of intuition algorithm (?).
[ link to this | view in thread ]
Chronno S. Trigger (profile), 14 Jan 2011 @ 7:54am

Re: Re:
A 6th grader can find the question with a Google search, but that 6th grader has the intelligence to take that answer and turn it into a keyword search, taking into account the word play, and sifting threw the many, many responses for the question that fits.

Computers aren't good at figuring out what information can be ignored and what words are being used in ways that aren't in the dictionary.
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 8:00am

Re:
It's really not that simple. I can see how it seems that way at first, but in fact this was a massive AI challenge. It's not having the knowledge that is difficult (and no, it has no internet access) - the challenge is parsing the wording of the Jeopardy questions to figure out what the answer is. Remember: computers don't understand English. They just fake it, and usually not very well.

There are lots of articles out there explaining why Watson was a very real AI challenge - and no they are not all just articles by IBM. You should definitely look more into it because it's actually pretty fascinating and it truly is a major breakthrough, if not in genuine "artificial intelligence" then at least in the natural language processing (which is one of the core building blocks of artificial intelligence)
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 8:02am

Re: Re:
A 6th-grader's brain comes pre-installed with the near-magical capability of abstraction. THAT'S what the challenge was here. Perfect recall is no huge feat - which is why even people like Jennings are only really impressive in a very limited way. But a computer learning to abstract and recognize complex patterns in order to answer natural english questions - that is tough
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 8:19am

Re:
My thoughts exactly. They asked the kinds of questions that computers are good at and worded them in a way that would help the computer answer correctly. They probably also chose questions that the computer was designed to be able to easily answer ahead of time.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 8:22am

Re: Re:
(and they installed preprogrammed answers ahead of time designed to answer those pre-planned questions. So it's kinda like allowing someone to study a test bank ahead of time and then testing how well that person knows the answers to the questions in the test bank. Very little conceptualization required).
[ link to this | view in thread ]
Esahc (profile), 14 Jan 2011 @ 8:57am

I'll take 'destroy all humans' for for $500 Alex.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 9:06am

Re: Re:
If I chose the questions, or set of questions to be asked, ahead of time then even I can create a computer program that can answer each question flawlessly and beat almost anyone who is being exposed to those questions for the first time. That's hardly impressive. I will just write a test bank of questions, program them into the computer, program the answers into the computer, and ask those pre - programmed questions.

Were the people who created the program independent of those who chose the questions? Were those who chose the questions given any guidelines as to what kinds of questions should be asked, what kinds of questions the computer has answers to, and how the questions should be worded? This test run was done at IBM's headquarters and so it was done in a controlled environment. Practically any beginner programmer can create an excellent Jeopardy contender in a controlled environment if the set of questions and answers were predetermined. We must know how prepared the computer was to answer the set of questions that were being asked ahead of time in opposed to how prepared it was to answer any random set of questions.
[ link to this | view in thread ]
teka (profile), 14 Jan 2011 @ 9:44am

Re: Re: Re:
do at least the tiniest, most minimal fact-checking before you spout off that they were somehow cheating or making things easy for the Watson system.

otherwise you can come out sounding petulant and foolish.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 10:35am

Re: Re: Re:
You mix some good concerns in with concerns such as "maybe they cheated entirely". Why don't you give them some credit and analyze it intelligently?
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 10:35am

Re: Re: Re:
What? Are you really accusing the whole shebang of being completely rigged?
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 10:38am

Re: Re: Re:
Where does this whole "pre-chosen questions" concept come from? You are just making that up.

That's not what happened. And yes, the people creating the program ARE independent. Maybe you didn't notice but this is an IBM computer competing at Jeopardy - IBM does not own Jeopardy and Jeopardy has no interest in helping them win. Besides, both sides have made it clear that the challenge is not altered in any way from a typical Jeopardy match - so unless you are accusing everyone involved of lying to our faces, I have no idea what you are talking about.

This is a genuine challenge, but you for some reason insist on believing it's a massive cheat. What has given you this idea?
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 10:48am

Re: Re: Re:
Watson's been around for several years now. It wasn't made just so it could answer Jeopardy questions.
[ link to this | view in thread ]
Chris Rhodes (profile), 14 Jan 2011 @ 10:50am

Re: Re: Re: Re:
Well, they have exactly zero evidence of any cheating having occurred, except for the idea they just manifested in their own minds, so clearly the unethical nature of the developers is a solid fact.

Clearly.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 11:16am

Re: Re: Re: Re:
I'm not saying the above is what happened, the above is just an oversimplification of how a computer can beat a person.

My point is that if the computer has been asked these questions before and the answers that the computers algorithms will provide are known in advanced (and the computer has been tweaked to answer these questions correctly), and the people contending against the computer have never been asked these questions, then it's not as surprising that the computer won. Sure, that still maybe a slight oversimplification, but the point is that the computer will naturally be more familiar with and tuned to answer the specific types of questions that its programmers will tend to ask along with being able to understand the specific wording that the programmers will tend to use (and respond with the expected wording).
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 3:45pm

Re:
Yah! bug you neglect the concept behind it, you take for granted the human ability to do so, is not that simple to make a computer do it, because we don't understand how we work.

If you doubt please try to make a search engine that understands the meaning of words and can keep track of the many many double meaning things have.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 3:52pm

Re: Re: Re: Re:
Hmmm...may I interject here?

Jeopardy was known to fix their games for a long time, this is history not story, so I would be careful not believing they could just lie to our faces they would if they could get away with it LoL

Other then that I doubt IBM would want that, because that system would be eventually be open to scrutiny and the truth would come out and destroy their good reputation that helps them sell billions in hardware. That is why I don't believe they are cheating here, they don't do that kind of thing because it would cost them a lot of money.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 3:59pm

Re: Re: Re: Re: Re:
If they did, their AI business that administer things like red lights, cameras for traffic control etc would be in serious jeopardy(pun intended) here.
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 5:22pm

Re: Re: Re: Re: Re:
Hmm - I didn't know that about Jeopardy. But definitely looking at this situation there are no benefits to either side for cheating: Jeopardy would rather have a human win, because the humans sort of represent them in this rivalry - and a computer beating their game would trivialize it in a lot of people's eyes. Meanwhile IBM, as you say, gains nothing by pretending to have made a breakthrough in AI, except perhaps a momentary stock bump before they get found out and torn to shreds.
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 5:29pm

Re: Re: Re: Re: Re:
You are making this all up. IBM has no idea what questions the computer will face - sure it has some concept of the type of question, just like any human practicing to be on Jeopardy. But the questions Watson faces are new and unexpected.

And the whole concept of a "type" or "nature" of question is exactly the point here. Humans can abstract and recognize the "type" of a bunch of complex and varying things like trivia questions. To a computer, each one is a completely unique string of characters that can only be parsed with strict logical conditions - same goes for every single one of the facts it has in its database of knowledge.

This is obvious even at basic levels of computing: spellcheckers work well, grammar checkers do not. That's because spelling is a (for the most part) strictly defined set of rules - the computer simply checks each word against its dictionary, seeking an exact match. Grammar, on the other hand, is highly complex with varying degrees of strictness and countless structural options: there are "rules" but they are not nearly as simple or clear, always being full of exceptions and intricate conditions. Now, even though most people cannot explain all the rules of grammar, we instinctively know when a lot of things are right or wrong just based on how it "sounds" to us - and we can't even explain what we mean by that, or how that process works in our head. Computers have no such skill.
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 5:33pm

Re: Re: Re: Re: Re:
Wait, I just figured it out - I think he's thinking of the Jeopardy! video game
[ link to this | view in thread ]
Marcus Carab (profile), 14 Jan 2011 @ 5:34pm

Re: Re: You seem excited by this
Another companies really needs to market their batteries upside-down and call them Copperbottoms.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 6:35pm

Re: Re: Re: Re: Re: Re:
"I didn't know that about Jeopardy."

Many game shows have been known to do that, not just Jeopardy, though that wasn't my point here. My point here, from the article

"was a test run this morning at IBM Research's headquarters in preparation for a televised weekend challenge"

From my understanding, this wasn't done by Jeopardy, it was done by IBM. To what extent were the selected questions created by people not involved in the development of the computer hardware, software, and database info and to what extent did those people creating the questions receive guidance as to how to word the questions, how accepted answers should be worded, and what kinds of questions to ask and which ones to steer away from.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 7:00pm

Re: Re: Re: Re: Re: Re: Re:
(and guidance as to what expected answers to expect so that questions can be biased towards ones with those answers). I'm not saying it's necessarily fixed, just that those developing the system will naturally have a tendency to ask certain kinds of questions, even if they're not conscious aware of it.
[ link to this | view in thread ]
Anonymous Coward, 14 Jan 2011 @ 7:00pm

Re: Re: Re: Re: Re: Re: Re: Re:
consciously aware *
[ link to this | view in thread ]
vivaelamor (profile), 15 Jan 2011 @ 6:50am

Re: Re:
"I do not really consider this being a big thing. This is like saying a 6th grader beat Ken Jennings using only his skills to type the question in Google."

Another point worth making is that Watson is completely offline. While they feed it presumably more books than any human could read in their lifetime, it cannot just rely on the question having already been answered in an easy digest form on Google.
[ link to this | view in thread ]
vivaelamor (profile), 15 Jan 2011 @ 6:56am

Re: Re: Re: Re: Re: Re:
'And the whole concept of a "type" or "nature" of question is exactly the point here. Humans can abstract and recognize the "type" of a bunch of complex and varying things like trivia questions. To a computer, each one is a completely unique string of characters that can only be parsed with strict logical conditions - same goes for every single one of the facts it has in its database of knowledge.'

Two terms worth throwing in here for anyone wanting to learn more about the sort of processes I assume Watson uses: fuzzy logic and neural network.
[ link to this | view in thread ]
vivaelamor (profile), 15 Jan 2011 @ 7:04am

Re: Re: Re: Re: Re: Re: Re:
"From my understanding, this wasn't done by Jeopardy, it was done by IBM. To what extent were the selected questions created by people not involved in the development of the computer hardware, software, and database info and to what extent did those people creating the questions receive guidance as to how to word the questions, how accepted answers should be worded, and what kinds of questions to ask and which ones to steer away from."

Why would IBM cheat on the not actual Jeopardy test? It's obvious why they wouldn't cheat: they can test it against the actual competitors and tweak the systems in preparation for the real test. Are you seriously suggesting that they're not confident of their chances in the real test and are using this trial run to make it look like they did well?
[ link to this | view in thread ]
vivaelamor (profile), 15 Jan 2011 @ 8:05am

Re:
"What I'm wondering: Are they trying to anticipate common categories and programming this type of information into the computer or simply relying on some kind of intuition algorithm (?)."

I'm guessing that if they needed to then they could a combination. I think it is more likely that such a step would be redundant because such a thing should be easily and more effectively learned by the machine anyway, e.g. by inputting previous actual questions. If every question that had the word 'dig' in the title turns out to be about archeology then the computer will learn that.
[ link to this | view in thread ]
vivaelamor (profile), 15 Jan 2011 @ 8:07am

Re: Re: Re: Re: Re: Re: Re: Re:
"I'm not saying it's necessarily fixed, just that those developing the system will naturally have a tendency to ask certain kinds of questions, even if they're not conscious aware of it."

What do you base that on? If their goal is to improve the system then they may well ask it questions that it is less likely to get right, a completely opposite bias to the one you assume.
[ link to this | view in thread ]
Marcus Carab (profile), 15 Jan 2011 @ 9:18am

Re: Re: Re: Re: Re: Re: Re:
Yes! But then, why learn all these fascinating new things about cognitive processes and artificial intelligence when you can just claim the whole thing is a hoax, right?
[ link to this | view in thread ]
nasch (profile), 15 Jan 2011 @ 5:05pm

Re:
Anything already invented is just a program, artificial intelligence is always whatever we don't know how to do yet. 20 years ago if you had described a computer that can beat a Jeopardy champion without assistance, that would probably be thought of as AI. Now to be AI it has to, I don't know, be able to both drive a car and cook a meal. And after that's invented, it won't really be AI until it can understand human emotions. And so on. ;-)
[ link to this | view in thread ]
nasch (profile), 15 Jan 2011 @ 5:10pm

Re: Re:
Remember: computers don't understand English.

I think you could make a pretty good argument that this one does. Which is what's so awesome about it.
[ link to this | view in thread ]