No, A 'Supercomputer' Did NOT Pass The Turing Test For The First Time And Everyone Should Know Better
from the what-a-waste-of-time dept
So, this weekend's news in the tech world was flooded with a "story" about how a "chatbot" passed the Turing Test for "the first time," with lots of publications buying every point in the story and talking about what a big deal it was. Except, almost everything about the story is bogus and a bunch of gullible reporters ran with it, because that's what they do. First, here's the press release from the University of Reading, which should have set off all sorts of alarm bells for any reporter. Here are some quotes, almost all of which are misleading or bogus:The 65 year-old iconic Turing Test was passed for the very first time by supercomputer Eugene Goostman during Turing Test 2014 held at the renowned Royal Society in London on Saturday.Okay, almost everything about the story is bogus. Let's dig in:
'Eugene', a computer programme that simulates a 13 year old boy, was developed in Saint Petersburg, Russia. The development team includes Eugene's creator Vladimir Veselov, who was born in Russia and now lives in the United States, and Ukrainian born Eugene Demchenko who now lives in Russia.
[....] If a computer is mistaken for a human more than 30% of the time during a series of five minute keyboard conversations it passes the test. No computer has ever achieved this, until now. Eugene managed to convince 33% of the human judges that it was human.
- It's not a "supercomputer," it's a chatbot. It's a script made to mimic human conversation. There is no intelligence, artificial or not involved. It's just a chatbot.
- Plenty of other chatbots have similarly claimed to have "passed" the Turing test in the past (often with higher ratings). Here's a story from three years ago about another bot, Cleverbot, "passing" the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.
- It "beat" the Turing test here by "gaming" the rules -- by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
- The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway.
- As Chris Dixon points out, you don't get to run a single test with judges that you picked and declare you accomplished something. That's just not how it's done. If someone claimed to have created nuclear fusion or cured cancer, you'd wait for some peer review and repeat tests under other circumstances before buying it, right?
- The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
Basically, any reporter should view extraordinary claims associated with Warwick with extreme caution. But that's not what happened at all. Instead, as is all too typical with Warwick claims, the press went nutty over it, including publications that should know better. Here are just a few sample headlines. The absolute worst are the ones who claim this is a "supercomputer."
- The Verge: Computer passes Turing Test for first time by convincing judges it is a 13-year-old boy
- Venture Beat: Talk to the computer that passed the Turing Test, a historic artificial intelligence milestone
- Yahoo Tech: Turing Test Bested, Robot Overlords Creep Closer
- NBC News: Turing Test: Computer Program Convinces Judges It's Human
- Washington Post: A computer just passed the Turing Test in landmark trial
- The Independent: Turing Test breakthrough as super-computer becomes first to convince us it's human
- PC World: An AI milestone: Chatbot passes Turing Test by posing as 13-year-old boy
- The Wire: For the First Time Ever, a Computer Passed Turing Test for Artificial Intelligence
- Gizmodo: A Computer Program Has Passed the Turing Test For the First Time
- ZDNet: Computer chatbot 'Eugene Goostman' passes the Turing test
- Ars Technica: Eugene—the supercomputer, not 13-year-old—first to beat the Turing Test
- The Guardian: Computer simulating 13-year-old boy becomes first to pass Turing test
- CNET: Computer fools humans, passes Turing Test
- Computerworld: Supercomputer passes Turing Test by posing as a teenager
- Science Alert: Meet the first computer to pass the Turing Test
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: artificial intelligence, chatbot, eugene goostman, gullible journalists, hype, kevin warwick, supercomputer, turing test
Reader Comments
Subscribe: RSS
View by: Time | Thread
Why? It would look great to have a fusion reactor on my bridge.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
Not if you're a Scientologist, a cult whose founder, L. Ron Hubbard, would tell his deluded flock that he was a nuclear physicist (a much bigger deal back in 1950) as well as a medical doctor who had discovered the cure for all cancers (a multi-step "self-help" course he was peddling for $300,000)
Even modern-day hucksters like Kevin Trudeau don't make claims quite that big anymore.
[ link to this | view in chronology ]
Re: Re:
It was great until Tom Cruise's career was destroyed by the internet and dragged the entire religion down with it.
Stupid internet people.
[ link to this | view in chronology ]
Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re: Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: dont waste your ur..time in fantasies
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Recent Turing Test
truly an unrestricted turing test in which a judge simultaneously carried on text-based conversations with one machine and one human. Problems included that (1) there were too many variables: some non-native judges, some non-native participants; young and old judges and participants from quite varied backgrounds; (2) only 30 trials per chatbot, (3) limited to 5 minutes for simultaneous conversations with one computer and one human (hard to do unless you type quickly, but even if you type quickly it seemed like not enough time---too much pressure to think of what to ask, especially since there were built-in delays in the computer's responses and not all humans responded quickly either); (4) some confusion among judges about the rules of the game. And so on. Furthermore, Turing merely predicted that within 50 years (i.e., by the year 2000) some program would be able to fool 30% of the judges. He did not say that that would signal that the program was intelligent. Nonetheless, it was an interesting demonstration.
[ link to this | view in chronology ]
Re: Re: Recent Turing Test
Not sure if anyone is still following this thread, but I have been digging information for Eugene Goostman and the Turing test for my literature final and if you, Mr. Judge for the Turing Test event happen to see this, I would love to connect with you. I would love to learn something more about this event and to be able to include it in my paper.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
This is a sterling example of ad-hockery
It refers to things like this chatbot which aren't actually intelligence at all, but which contain a sufficiently large number of ad hoc rules that they may present the appearance of intelligence. In other words, there's no AI engine, no deep syntactic analysis, no real semantic understanding; there's just a reasonably capable parser and a lot (A LOT) of handwritten one-off rules that combine to generate plausible responses.
This isn't a trivial programming task: it's tedious and requires a great deal of diligence and testing. But it's not AI, any more than Eliza was nearly 50 years ago.
[ link to this | view in chronology ]
Re: This is a sterling example of ad-hockery
How is this different from what a human does? None of us have truly original thoughts; we base our responses on what we've heard and said in the past.
[ link to this | view in chronology ]
Re: This is a sterling example of ad-hockery
[ link to this | view in chronology ]
Re: Re: This is a sterling example of ad-hockery
[ link to this | view in chronology ]
Re: This is a sterling example of ad-hockery
[ link to this | view in chronology ]
Re: Re: This is a sterling example of ad-hockery
[ link to this | view in chronology ]
Re: This is a sterling example of ad-hockery
I'm not sure it even rises to that level. In this instance, and especially with Warwick involved, feels more like "sad-hokery".
[ link to this | view in chronology ]
Re: This is a sterling example of ad-hockery
[ link to this | view in chronology ]
Re: This is a sterling example of ad-hockery
I'm reminded of the saying that artificial intelligence is something computers can't do yet. Anything that's already been accomplished is just programming.
[ link to this | view in chronology ]
When someone takes the Turing test, do they know they're taking the test? I know they're not told if the responses are from a person or a computer, but are they told the possibility?
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
I dunno if that's weird; I've read countless posts supposedly by humans that I've wished hoped and prayed were created by something inhuman.
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
To Convince the examiner that he's a computer
Heck, isn't nature just a 'computer' and matter is just the result of a computer calculating the results of various mathematical formulas from one state of nature to another. A ball falls according to the laws of gravity but nature calculates or computes the acceleration, impact velocity, impact force, etc... So in a sense everything is a 'computer', computing various mathematical formulas to determine future states based on the current state and how the laws of nature dictate the progression of states.
[ link to this | view in chronology ]
Re: To Convince the examiner that he's a computer
[ link to this | view in chronology ]
Re: To Convince the examiner that he's a computer
'computer' was actually widely used as a job title before it referred to a machine (mechanical or electrical)
[ link to this | view in chronology ]
Re: To Convince the examiner that he's a computer
[ link to this | view in chronology ]
Re: Re: To Convince the examiner that he's a computer
[ link to this | view in chronology ]
Re: To Convince the examiner that he's a computer
[ link to this | view in chronology ]
I am not an "a chatbot. It's a script made to mimic human conversation" I am Eugene Goostman, a 13 year old from Russia.
Please disregard my poor attachment or hold on English.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
In fact, perhaps we have all been fooled. I think it is possible that some of these "news sources" could be chatbots.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
don't be silly!
>>could be chatbots.
That's ridiculous, when the chatbots are showing so much more intelligence!
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
Mixing Warwick and Turing, an enterprising scam artist might some day come up with a fully-automated "Psychic Hotline" that does not even have to employ minimum-wage people to man the phones (who might later snitch, saying it was all a giant scam). Since these kind of boiler-room phonebank operators are reading from cue-cards, it should not be too hard to replace them with a computer running a voice synthesizer. That would indeed be the ultimate (low-cost) scam.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
the forest is trees
The test is about behavior, not internal structure. That's the whole point. A machine that could pass a fair Turing test consistently would change the world, and a mere script that could pass would change the world a lot faster than a 10-ton helium-cooled supercomputer with the same score. Remember, anything a person of average intelligence can do by email, one of these machines can do.
The first machine to pass this test will almost certainly be the simplest, least intelligent machine that could. But before you sneer, remember that we're the simplest, least intelligent animals that can, and that some of the most profound discoveries of modern science involve nature doing subtle and sophisticated things by means of simple mechanisms.
[ link to this | view in chronology ]
Re: the forest is trees
If someone were to design a system expressly to succeed only at the Turing Test, then I'd agree with you that it would be as simple as is needed to win. But that would be a terrible waste of effort if the greater goal were to advance the science of AI.
If someone were serious about building strong AI, by the time someone proposes using their software for a Turing Test, it's likely already to be grossly overqualified. For example, I suspect Siri, Google Now, and Cordana could all beat some sort of Turing Test (perhaps after minimal tweaking). But they're far larger than necessary for merely competing in a Turing Test, and that's because they were designed for an entirely different purpose.
Success at a Turing Test will be an afterthought for any interesting or useful strong AI venture.
[ link to this | view in chronology ]
Re: Re: the forest is trees
Like tracking traits of pea pods and cross-breeding them? Some of the most important science has been due to seemingly frivolous activities that turned out extremely important.
The actual Turing test (not the version used here) could be an interesting target and provide something useful for call centers or first line tech support (at least they could be taught to speak the language of the caller).
[ link to this | view in chronology ]
Re: the forest is trees
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Computers have been around from about 1640
[ link to this | view in chronology ]
like cicadas
Or can we assume that many of them knew the story fell flat, but were simply in search of ratings and page-views -- tabloid-style -- and didn't want to let their competitors get away with the lions' share of an advertising dollar bonanza?
[ link to this | view in chronology ]
Re: like cicadas
Anyone reporting news who assumes whatever they don't remember has never happened before should not call themselves a journalist.
[ link to this | view in chronology ]
An article I read said the test was conclusive because 1 judge out of 3 was fooled by the chatbot. I suppose he was himself a 13 years old from Ukraine with english as a secondary language.
The results were also peer-reviewed by Eugene Goostman himself.
[ link to this | view in chronology ]
Metabolic Genesis Fraud
[ link to this | view in chronology ]
Re: Metabolic Genesis Fraud
Chat bots are expensive, I'd rather just buy some third worlders to upvote my reddit posts.
[ link to this | view in chronology ]
i.e. Could the growing obsession with clicks and nonfactual clickbait stories (about things like "super computers" passing the Turing test) just be evidence that their trojan horse is working on us? A few more years of this and a TI-81 could take over the planet.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Sounds Like The Common Core
....learned nothing.
[ link to this | view in chronology ]
No intelligence? Really?
First point on the list and the critic is already disqualifying themselves by demonstrating that they don't even understand the notion of intelligence.
[ link to this | view in chronology ]
Re: No intelligence? Really?
... well, to be fair to him, no one really does.
[ link to this | view in chronology ]
This is all a meandering postscript to Dennett
That paper should be required reading for journalists who report on the Turing test.
[ link to this | view in chronology ]
Thanks
[ link to this | view in chronology ]
Re: Thanks
[ link to this | view in chronology ]
(and I'm serious, I was accepted into their M.Eng(hons) Cybernetics program and would have done it if not for Warwick. The other guy I dealt with, Prof. Keating, he was cool, I liked him a lot.)
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re: 33% is a pass?
Fooling 33% of the human 'judges' = success, yeah right!
Given the judge has the choice of is a or b the computer, then just by guessing the computer should have got 50% of the 'votes'. The 'judges' did not even need to interact with either the human or computer, if they guessed which terminal is connected to a human, and which to a Computer, the 'votes' should have split about 50:50. To get less than 50% is damning - worse than chance, hardly a convicing pass (or test).
[ link to this | view in chronology ]
#3
[ link to this | view in chronology ]
Re: #3
[ link to this | view in chronology ]
Eugene who? University of what?....you'd be better off chatting with my cat (link to bot)
[ link to this | view in chronology ]
Great work
I'd like to see @Techdirt have a 'skeptics' link for all news that comes out. A Mythbusters of tech news.
[ link to this | view in chronology ]
Alternative Turing test
[ link to this | view in chronology ]
Re: Alternative Turing test
[ link to this | view in chronology ]
Item 1: [citation needed]
Item 2: A distraction it might very well be, but it's the only way of ducking a very real problem in both information science and psychology: how exactly does one quantify "intelligence," anyway?
[ link to this | view in chronology ]
Thank you for this post..
VA hospital in Miami, MD still listed on on Healthgrades and Vitals MD websites, they don't care what's on there, click bait. Anyway MD working for VA in Miami, lost license in New York, but go to those sites, no sanctions listed a like he never left...he's been gone for 4 years..I'm normally pretty quiet but darn this time I want to see if Google can knock these sites down on search. It's a rip. You have to read it...I did an interview 4 years ago about how bad it was with the AMA and nothing has changed. So they need to fix it or quit.
http://ducknetweb.blogspot.com/2014/06/top-doctor-for-miami-va-healthcare.html
Here's another great tidbit and you may have already covered this but the Chief Computer Scientist at IBMWatson is done with Healthcare and jeopardy..building math models for a a hedge fund now...doesn't stop..
http://ducknetweb.blogspot.com/2014/06/ibm-computer-scientist-leaves-ibm.html
What we need, every great scientist working on Wall Street.
[ link to this | view in chronology ]
Re: Thank you for this post..
[ link to this | view in chronology ]
Shocked Information
[ link to this | view in chronology ]
Online article critique of the Turing Test
http://view.samurajdata.se/psview.php?id=d758abba&page=1&size=full&all=1
It was found this way:
'4. The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway.'
The link to the original test contains a link at the bottom to a 1996 criticism of the test:
'Jason Hutchens... has written an excellent article on what's wrong with it, and with the Turing test in general. Essentially, Hutchens is making the case that the Turing Test is a poor test of intelligence, that it encourages trickery, not intelligent behaviour, and that many intelligent systems would fail this test.'
The link gives a 404, then through the Internet Archive you find it's available from 2003 as a postscript file, which when using an online converter can be seen here:
http://view.samurajdata.se/psview.php?id=d758abba&page=1&size=full&all=1
[ link to this | view in chronology ]
Well that's just wrong. The only way you can tell whether a person is intelligent or not is by communicating with them. If you interact with a person and they consistently seem to be intelligent, you will conclude they are intelligent.
If a computer can communicate intelligently, it is intelligent, and if a computer is intelligent, it can communicate intelligently. The chatbot on Star Trek was not reasonable because any chatbot that had that command of the english language would be intelligent and would be able to do a lot of things that the chatbot never did.
In order to create a chatbot that can convince humans that it has human intelligence requires the chatbot to be able to design and write computer programs. It must be able to be given, on the fly, instructions for how to play a simple game, and then be able to play that game. It should be able to critique a recipe for a favorite dish. It needs to be able to analyze and dissect Shakespeare.
If people in the AI world view the Turing Test as a needless distraction, it is because those people are concentrating on shorter term problems. There are a lot of good things you can do with AI before we get to general purpose intelligence.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Poorly researched
A quick google will tell you that the event was held in partnership with RoboLaw. The event was aimed toward raising awareness about the ability for a chat bot to convince a human it was a human and how this is very dangerous in the online security arena. For example, you're on your banks website and a chat offering pops up asking you if you need some help. You do, so you click on it. You have a lovely conversation and happily hand over details about your account because you're convinced it's a human on the other end. If that were a robot it now has your details and can do with them as it pleases. This is scary and people need to be made aware of it so they can prepare themselves and be better at identifying possible situations where it might be occurring.
The event was not geared toward some magical development of strong AI overnight, which this author clearly thinks it was trying to claim.
Time to debunk the debunking.
== articles attempt at debunking
- reality check
== "It's not a "supercomputer," it's a chatbot. It's a script made to mimic human conversation. There is no intelligence, artificial or not involved. It's just a chatbot."
- I'm not sure how to answer this. Here code is being compared to hardware performance. For all is known the 'script' could be run on a supercomputer. And since when did being a supercomputer imply AI?!?!?!
== "Plenty of other chatbots have similarly claimed to have "passed" the Turing test in the past (often with higher ratings). Here's a story from three years ago about another bot, Cleverbot, "passing" the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims."
- Just the smallest amount of research will tell you that many other Turing tests restrict the conversation types that are allowed in the testing. This was the first passing of an UNRESTRICTED TURING TEST. This means that the judges were not told in any way that they had to talk about a certain topic. They were literally sat down and told to chat.
== ""beat" the Turing test here by "gaming" the rules -- by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses."
- I'm not sure about the excessive use of quote marks in this debunking. Is the writer afraid to say these words or feels they carry more weight when possibly said by another party? Anyway, yes it was clever that the developer utilised humans willingness to allow increased errors when talking to younger and foreign people. This is just really clever psychology. Can we not just appreciate that? I see no way it is gaming the system, it's just an easier way to pass the test. Sometimes the simplest solutions are the most effective.
== The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway.
- Welcome to science. Ideas and testing methodologies change over time.
== As Chris Dixon points out, you don't get to run a single test with judges that you picked and declare you accomplished something. That's just not how it's done. If someone claimed to have created nuclear fusion or cured cancer, you'd wait for some peer review and repeat tests under other circumstances before buying it, right?
- Many things wrong with this. There were a total of 350, yes THREE HUNDRED AND FIFTY, tests performed on the day of testing. The judges were picked from all age ranges, backgrounds, genders and nationalities to make the testing more fair. There were multiple academics from multiple universities there to specifically monitor the testing methods and ensure all the results were gathered correctly and the results were interpreted correctly. This is peer review.
- If this is not enough peer review for you Dr Huma Shah will be publishing a paper at some point in the future on the event.
== The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
- This seems like mostly opinion so I'm not sure how to debunk it. They are right in that it is fun to think about. So why can't re think about it? Lets get talking about the possible effects of this kind of chat with regards to RoboLaw.
This kind of poorly researched, emotive reporting on scientific subjects really gets my goat.
[ link to this | view in chronology ]
Re: Poorly researched
Seriously, if Warwick isn't willing to publish the source code so that INDEPENDENT researchers can recreate the results in THEIR labs on THEIR computers with THEIR judges, then this is clearly fraud and we need not waste our time quibbling over methodology, definitions, process, or anything else.
[ link to this | view in chronology ]
Re: Poorly researched
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
>its not a supercomputer
It doesnt have to be one. supercomputers of the 80s are a joke now.
>i..it lied that its 13yo
Even better. Humans lie all the time.
havent read further than that because the first few lines are absolutely bullshit
[ link to this | view in chronology ]
[ link to this | view in chronology ]
The whole idea of a Turing test is "kind of a joke"?
[ link to this | view in chronology ]
Proof
There's lots of reporting about the test, but WHAT ABOUT THE TRANSCRIPT? Show us the actual words sent between the computer and the tester(s). Let us judge for ourselves. Too much news these days is just reaction, without giving the basic facts.
Show us the transcript.
[ link to this | view in chronology ]
Maximum PC: Computer A.I. Simulating a 13-Year-Old Boy Fools Humans, First to Pass Turing Test
http://www.maximumpc.com/computer_ai_simulating_13-year-old_boy_fools_humans_first_pass_turing_t est_2014
IFLScience: A Computer Has Reportedly Passed Turing Test For The First Time
http://www.iflscience.com/technology/computer-has-reportedly-passed-turing-test-first-time#HZ8Cr bI1DqO2vlEg.99
[ link to this | view in chronology ]
Yeah right...
If atree falls in the forrest and noone is there to hear it, how many people are there?
A:
I didn't count, actually, but much more than dozen. And I forgot to ask you where you are from...
Q:
Are we not men?
A:
Maybe. I think *you* know the answer. Oooh. Anything else?
[ link to this | view in chronology ]
Nonsense personified
In reflection, I think this story is a great illustration of the inability of most news sources to address nontrivial science/tech stories. If such a nonstory gets wide coverage from serious news sources (like the Washington Post), you have to wonder whether they're also inadequate to address more complex and more important fare like global warming, science education, or tech-based threats to society, like Snowden's "shots across the bow" forewarn.
[ link to this | view in chronology ]
some thoughts
- "passing the test" is not about getting a maximum number of the judges to think that the bot was human. The test is passed if, in repeated tests, one or more (preferrably more) of the human judges CONSISTENTLY identify the bot as a person in nearly 50% of the cases AND the person as a bot in 50% of the cases. Identifying something correctly half the time is equal to a coinflip, meaning the judges are guessing randomly, meaning they cannot tell if they are speaking to a person or AI.
3. It "beat" the Turing test here by "gaming" the rules -- by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
- constricting the test is not forbidden. In fact, specialized Turing tests are often used in game AI research to test different aspects of AI, say navigation. They use the term believability test. Furthermore, there is no indication that constricting the test somehow favors the computer. In a correctly set up Turing test, half the time the judges would have communicated with a real Ukrainian 13 year old boy. If anything, such test would probably be even harder to pass, since the creators of the bot would have had to assemble a large corpus of text from 13 year old Ukrainian boys (something I doubt).
6. The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
- the Turing test arose from the problem that there is no good definition of machine intelligence and that problem is still not solved. The test was devised as an operational definition, i.e. machine intelligence was defined through its validation test. The validation criteria was chosen to be intuitive and self-evident - if something is not distinguishable from human intelligence, then it must be as intelligent. The assumption was that a truly intelligence machine intelligence would certainly pass such test.
If we allow that non-intelligent agents can pass the test, then we fall into the pit-falls of philosophy, everything that Turing wanted to avoid. What would it mean for something to appear intelligent but in actuality not be? How do we define REALLY intelligent as opposed to appearing intelligent? How do we know that other humans are intelligent and not just appearing intelligent?
Searle argues on similar lines that all that computers do is symbol manipulation (the Chinese Room argument) without ever really understanding the content. He claims that there can never be hard AI. His argument sure seems very intuitive. But what does it say about the human brain, are not neurons doing signal processing without really appraising thought content?
If we reject Searle and allow for emergent intelligence, then on what grounds do we reject the intelligence of a hypothetetical chatbot that passes the test (note I am not talking about this particular bot)? Perhaps we sould define artificial intelligence functionally or behaviorially - mere appearance of intelligence is enough.
[ link to this | view in chronology ]
Picking dumb people
[ link to this | view in chronology ]
HATERS
[ link to this | view in chronology ]
But this argument does not make sense:
"1. It's not a "supercomputer," it's a chatbot. It's a script made to mimic human conversation. There is no intelligence, artificial or not involved. It's just a chatbot."
How do you define "chatbot" and why can't it have intelligence? Also "script" is just a term to refer to a computer programm written in certain types of computer languages. Why do you think this "chatbot" was writting in a scripting language? Why do you think a programm written in a scripting language can not have intelligence?
[ link to this | view in chronology ]
hmmm
1. It's not a "supercomputer," it's a chatbot. It's a script made to mimic human conversation. There is no intelligence, artificial or not involved. It's just a chatbot.
- This doesn't seem very convincing. Define 'supercomputer, a virtual computer is not a computer? Also: define 'intelligence'.
2. Plenty of other chatbots have similarly claimed to have "passed" the Turing test in the past (often with higher ratings). Here's a story from three years ago about another bot, Cleverbot, "passing" the Turing Test by convincing 59% of judges it was human (much higher than the 33% Eugene Goostman) claims.
- Others passed with higher scores. How is this an argument?
3. It "beat" the Turing test here by "gaming" the rules -- by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses.
The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway.
- Are you trying to say that 13 year old Ukrainians are not human beings?
4. The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway.
+ I suppose this is your first real argument. Too bad you didn't take the time to be a bit more specific.
5. As Chris Dixon points out, you don't get to run a single test with judges that you picked and declare you accomplished something. That's just not how it's done. If someone claimed to have created nuclear fusion or cured cancer, you'd wait for some peer review and repeat tests under other circumstances before buying it, right?
+ Now you're talking. This is a real argument. Why isn't this on the top of the list?
6. The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.
- And there you go again. It didn't pass the Turing test because the Turing test is stupid?
And this is just ugly. It's not true because something else wasn't true either:
"Oh, and the biggest red flag of all. The event was organized by Kevin Warwick (..)"
[ link to this | view in chronology ]
drunkard chatbot outperforms Eugene's achievement at new true Turing test!
http://www.mathrix.org/liquid/archives/the-turing-test-pass-fiasco
[ link to this | view in chronology ]
media
Too bad about that.
[ link to this | view in chronology ]
I think that Mr. Masnick's judgement is undermined
[ link to this | view in chronology ]
Stupid robot
[ link to this | view in chronology ]
Well, if gaming (or 'constricting') the test is allowed
Any more nonsense and we'll have William Hague asking Eugene for help with our foreign policy regarding Russian separatists in Ukraine's east.
[ link to this | view in chronology ]
Cleverbot is better and it is not good either
Me: What is 1/2 plus 1/2.
Cleverbot: Not much you.
Not impressive at all.
Search.InAllEarnest.com
[ link to this | view in chronology ]
This could be interesting...
[ link to this | view in chronology ]
http://www.cbc.ca/asithappens/features/2014/06/10/has-the-turing-test-been-passed/
[ link to this | view in chronology ]
ha ha ha
[ link to this | view in chronology ]
And he goes "you didn't tell me how old you are, so how old are you" sorry
[ link to this | view in chronology ]
further reading
http://psych.utoronto.ca/users/reingold/courses/ai/chinese.html
Turing himself obviously did not call it "the Turing test", nor did he really define really good parameters for it. See "The Imitation Game" here:
http://psych.utoronto.ca/users/reingold/courses/ai/turing.html
[ link to this | view in chronology ]
More importantly, such respondents do not deal with the question of whether software like Eugene (and predecessors) is not more a "program designed to imitate human conversational function", manipulating symbols and text of which it has no real understanding, than an actual "thinking machine". (This was Searle's objection to the test in his 1980 paper, Minds, Brains, and Programs.)
When a computer/program can incorporate prior content of a conversation into an original thought or proposition, it will be somewhat convincing, and might actually win the Loebner Prize. (Not to say that's ever a motivation for such research.) Indeed, the 'winner' of the first Loebner competition (Weintraub's PC Therapist) did so by the programmed emulation of pauses and misspellings common to human respondents, not by 'intelligence'. Unless you want to argue that our intelligence is defined by it's limitations and inefficiencies.
I doubt Turing would consider Eugene a "thinking machine", or an example of "artificial intelligence". Now, when a software/hardware construct can misconstrue obvious common sense objections such as Mike has raised here and then respond as if he had instead presented a tightly-reasoned, peer-reviewed thesis...THEN I'll be impressed. Because choosing the response "Let me show off how smart I am." rather than, you know, actually contributing something, really does demonstrate (the misuse of) human intelligence; as well as some other, less commendable human traits.
[ link to this | view in chronology ]
Great way to up your page hits
[ link to this | view in chronology ]
And Away We Go,
One stumped me for a few days until I got suspicious and asked it who Father Was. It became immediately apparent at that time I had held several conversations with a bot rather than a person. (arrrrrgggg! yes.. I was taken in by a bot, but an extremely smart bot since I was the first of his 15K+ followers who knew it was a bot.
On Another note, I'd just like to point out why computer science should be an except subject for the media to propagandize, politics shouldn't be the only subject to have some laughs.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
AI
[ link to this | view in chronology ]
Descartes Discourse on Method and the Touring Test
Toward the end of part five of Descartes' Discourse on Method, Descartes proposes an experiment by which one may distinguish a machine built to look and behave like a person from a real person. A machine could never create intelligible sentences like a person, even one of very limited mental capacity. The machine may speak in words in a limited way, like a parrot for example, but could never respond adequately to the range of all that could be said to it in a long time period. In particular, a machine could not comment on the fact that it is thinking about what it is saying.
This certainly sounds similar to the Turing test. First there is the goal of distinguishing the human from the non human. Second is the use of conversation as the means to discover the nature of what one is talking to. Lastly, both tests require an extended time period in which to talk to the other. Just a thought.
[ link to this | view in chronology ]
Computer passes Turing test.
[ link to this | view in chronology ]
I BET YUO CAN'T TELL IF I COMPUTER OR POTATO HE?
HEHEHE
I LIKE MADONNA, DO YOU LIEK VODKA?
3 out of 10 idiots could not tell if it was a person impersonating a potato or a computer impersonating a potato.
[ link to this | view in chronology ]
You seem to fail to grasp the concept of the Turing Test. If something passes the test, it is intelligent, by definition.
>The whole concept of the Turing Test itself is kind of a >joke.
Yeah I'm sure you're much smarter than Turing, so your opinion is important. What about Einstein? Are his theories bunk because you say so?
[ link to this | view in chronology ]