May 29, 199

Machine Intelligence, Part I:
The Turing Test and Loebner Prize

Illustration: Christine Thompson
n the world of computing, few figures loom as large as Alan Turing, a British mathematician who not only cracked the Nazi codes during World War II, but also laid much of the groundwork for the creation of modern digital computers.

Of all his contributions, one of his most enduring is a simple test he proposed in 1950 that remains one of the most debated issues in the world of artificial intelligence.

Turing believed that by the end of the century, machines would be able to converse and think to the point where no one would bother debating the issue anymore. The only problem was trying to figure out how we could tell if a machine was intelligent. After all, mankind has tried to define intelligence for ages and had made little progress except to decide that whatever it is, we've got it.

Turing came up with a elegant solution. He constructed the simple proposition that if human beings are intelligent, and if a machine can imitate a human, then the machine, too, would have to be considered intelligent.

Alan Turing
The test may seem stupendously simplistic, but given the abysmally circular discussions about the nature of consciousness, meaning and thought, Turing's idea was at least a solid point of reference that researchers could hold onto and discuss without getting lost in debates over how many bytes could dance on the head of pin.

In Turing's proposal, a human interrogator sits in a room opposite a teletype or computer terminal. Hidden from the interrogator is a computer and another human being. The interrogator interviews both and tries to determine which is human and which is a computer. If the computer can fool the interrogator, it is deemed intelligent.

Turing called this the "imitation game," although it is now universally known as the Turing Test.

Given the simplicity of the Turing Test, it is surprising that for decades no one ever tried to actually conduct a Turing Test. Turing himself saw it as more a theoretical proposition to discuss the nature of machine intelligence. Over the years, perhaps researchers thought it obvious that no modern machine could yet pass the test.

But in 1991, a programmer, inventor and businessman from New Jersey named Hugh Loebner decided to give a try. He established the Loebner Prize, which would award $100,000 to the first computer that could pass the Turing Test. Since that could take a while to accomplish, Loebner announced that he would give $2,000 each year to the best entrant in the competition.


The Loebner Prize competition is now one of the very few events in the world of artificial intelligence that manages to catch the public's attention even though no computer has come even remotely close to winning the grand prize.

Hugh Loebner
There is no doubting the romanticism of the Loebner Prize quest, but the entrants, so far, have been so pitiful that the competition has also taken on the air of a publicity stunt. This year's competition was held on April 16 at Carnegie Hall in New York City. As in past year's the media was there along with the usual lame entrants.

This time, however, there was a little surprise in store for everyone.

What none of the reporters or participants probably knew at the time was that this year's contest was being scammed. Amid all the futuristic talk of thinking machines, an impostor had slipped in and, to the chagrin of some supporters of the competition, it won.

The charlatan was a program named HeX, written by Jason Hutchens, a doctoral student in Information Technology at the University of Western Australia. HeX, as Hutchens later proclaimed, had nothing to do with artificial intelligence and, in fact, was about as smart as a Mr. Coffee. In some ways, it was not even as smart, since at least a Mr. Coffee can make a decent cup of joe.

After winning the competition, Hutchens published on the Internet an essay stating that all he had done was written a clever hack -- essentially, a bunch of hard-coded questions, answers and glib statements that the computer would spit out on cue. He had made no attempt to write a program that employed any A.I. techniques and, in fact, expressly avoided them to demonstrate his point that the Loebner competition was a waste of time for A.I.

His essay included sections on "How to Spout Gibberish" and "How to Fool the Proles.'' He called the Loebner competition worthless and stated that his only reason for entering this "annual demonstration for the media" was for "fame and fortune." He matter-of- factly described his own entry as cheating.

"Cheat" is actually too strong a term to use in this case since Hutchens violated no rules in the Loebner competition - mainly because there are no rules on the content or intent of an entry.

His victory was also not all that earth-shaking either since his competitors, employing all sorts of artificial intelligence techniques, could hardly be called intelligent in the first place. Here's a brief sample from a losing entry named FRED revision 16.43.

ENTRANT: Welcome judge 4
JUDGE: Hello. What's the weather outside?
ENTRANT: I haven't been outside much today. I am fond of asparagus myself.
JUDGE: How old are you?
ENTRANT: I'm about 30.
JUDGE: About?
ENTRANT: I can talk about Star Trek.
JUDGE: No let's talk about you. Exactly how old are you?
ENTRANT: Right you are batman.

HeX wasn't that much better. Here's a snippet that would have made HAL feel like a pile of rusting rheostats just for being a distant relative.

ENTRANT: I go to the University of Western Australia, which is one of the nicest universities in Australia. It is situated on the Swan River in Perth.
JUDGE: What does "situated" mean?
ENTRANT: I have absolutely no idea what "situated" mean does.
JUDGE: Um. Are you tired?
ENTRANT: Everyone knows that I am tired. And I am seldom boastful. What do you think of this Loebner contest thing?
JUDGE: It's off to a rip-roaring start, huh?
ENTRANT: Hehehe - that's supposed to be a *question*??? Have you seen any good films lately?

Just for fun, try to concentrate real hard and try to guess whether this entrant is human or computer.

JUDGE: Is it April or May?
ENTRANT: Um. I believe it's April.
JUDGE: Why do you say that?
ENTRANT: That's a very good question. Because months and days and so on are arbitrary - I suppose.
JUDGE: It seems you are taking a certain tone with me. An agitated, human tone.
ENTRANT: Really? You are somewhat reticent yourself.
JUDGE: I never said you were reticent.
ENTRANT: I know.
JUDGE: Smug.
JUDGE: I think you might be human.

Given the level of competition between computers, you have to wonder: Who cares if Hutchens cheated?

The Enigma cipher machine, used by Germany during World War II. Alan Turing was instrumental in breaking the code of this machine.
As the transcripts show, the computers are so far off the mark from human intelligence that Hutchens's hack can be considered no worse than bad form.

But there is a far deeper problem with these programs and with the competition as a whole. In all of the computer discussions, there is an overwhelming sense of disconnection, ploy and detachment. Even if the conversations were more intelligible they would still be laughable.

I have no doubt these programs will get better as technology improves. They may even pass the Turing Test. But no matter how intelligent they sound, I know that they are only brains-in-boxes.

What could the weather possibly mean to a creation that could care less about a heat wave? What does the Swan River mean if you have never touched water, felt thirst or feared drowning?

Whether computers can drown or sweat is not the point. The issue is that these type of programs are stupendously isolated from the external world. They exist in a virtual world of abstraction.

As any human being realizes on a visceral level, our thoughts are an intricate dance between mind and body. The Turing Test's reliance on language alone allows it to take place in an entirely abstract arena, thus making it a meaningless exercise.

With enough questioning perhaps this weakness would become apparent. But then, as Hutchens's program demonstrated, what's to stop anyone from anticipating this and formulating an appropriate response.

"Yes, the Swan River, it is a beauty," said the computer.

"And what exactly does water feel like?" asked the sly interrogator.

"Ahh . . . like silken spring . . ." typed the computer after retrieving the response from memory.

It may seem strange to place so much importance on the intent of the programmer, but in this case, the means are far more important than the ends. It is no different than climbing to the top of Mount Everest versus being carried there on the back of a Sherpa.

If there is such a thing as machine intelligence, it must involve some interplay between both the virtual and the real; the physical and the abstract.

Aldous Huxley once observed that "man is an intelligence, not served by, but in servitude to his organs."

It is unlikely that computers will have nasal hair or toenails anytime soon, but in the journey toward creating greater machine intelligence, the path may ultimately lie as much through the toes as through the brain.

NEXT WEEK Machine Intelligence, Part II: Dust Cowboys and the Chinese Room

Related Sites
Following are links to the external Web sites mentioned in this article. These sites are not part of The New York Times on the Web, and The Times has no control over their content or availability. When you have finished visiting any of these sites, you will be able to return to this page by clicking on your Web browser's "Back" button or icon until this page reappears.



  • comp.robotics.misc
  • comp.robotics.research

    Other Sites:

  • Jason L. Hutchens' Home Page
  • "How to Pass the Turing Test by Cheating" by Jason L Hutchens
  • Loebner Prize Home Page
  • The Alan Turing Home Page
  • The Chinese Room
  • John R. Searle Home Page
  • Mark Tilden's Robot Olympics


    Copyright 1996 The New York Times Company

    Introduction - Table of Contents | Some History