The "Say It Sign It" system translates spoken words into sign language and then engages an avatar to communicate using gestures. The onscreen translator could work as a pop-up on a television, personal computer, mobile phone or auditorium screen, giving the hearing impaired wider access to television, radio and education.

"It was inspired by a vision that a deaf colleague on my team had of seeing, not words being brought up on his phone, but an animated character signing in British Sign Language," said Andy Stanford-Clark, master inventor at IBM Hursley in the U.K.

The colleague was Ben Fletcher who, at the time, was an intern for Extreme Blue program. Fletcher is deaf and British Sign Language is his first language.

"He was able to give feedback and keep us honest," said Stanford-Clark. After returning to school and completing his degree, Fletcher became a permanent member of the team, which also includes researchers from the Universities of Oxford, Durham, Glasgow and East Anglia.
The program has about three main components. First a voice recognition system takes words spoken into a microphone and converts those into a stream of text. The text is then sent through a translation program that looks for patterns in the words and applies different rules based on those patterns.

For example, if someone said "My name is Andy," a rule would transform that to the phrase in British Sign Language, which is "Name me Andy."

Once the word order is worked out, the program refers to a dictionary of gestures supplied by the Royal National Institute for Deaf People. These are used to direct the computer-animated character.

Tying it all together is a piece of software, or middleware, that allows different applications and computers to link together without being in the same building or even the same country.

Although the system is still in the prototype phase, the team envisions a couple of different scenarios for its use. The program could be hosted by a service Web site. A speaker would sign in on one end, while the recipient would sign in on the other end. As the person speaks, the words would be converted on a central server and then animated on the end-user’s screen.

The system could also work from a converter box atop a television, or it could be sold individually as a program that speakers and receivers would install on their own computers.

"The model is quite different from the things that I have seen in the past," said Guido Gybels, director of new technologies at the Royal National Institute for Deaf People in London. "It’s built in a way that, in principle, can be applied to other foreign languages."

But, said Gybels, "There are significant technical and scientific challenges, in addition to cultural ones, that need to be addressed before we can see this as an off-the-shelf product."

For starters, the speech recognition technology is still a long way off from capturing free-flowing unconstrained human dialogue.

Furthermore, while the English language (as well as other languages) has been studied in great depth, sign language has not, said Gybels.

If scientists are going to accurately convert speech into sign language they will need that understanding.

And lastly, he points out the notion of using virtual humans and animated characters doesn’t always fly with people you’re asking to use these kind of services. Not all humans are going to accept a replacement, especially one that may not have the natural, subtle motions — the finger, lip and movements — of the real thing.

Currently Say It Sign It translates spoken English into British Sign Language but Stanford-Clark said that it would not be a big leap to translate between other spoken languages and forms of sign language.

Via: Discovery Channel