NSF Award
9726363
9726363 Cole, Ronald Massaro, Dominic Oregon Graduate Institute of Science Challenges in CISE: Creating Conversational Agents for Language Training Technologies for the Next Generation of Interactive Systems The challenge addressed by this research is the creation of a realistic conversational agent for the domain of language training, and for spoken language training of the hearing impaired in particular. The conversational agent consists of four key technologies, two for output and two for input: (1) a 3D model of a talking face with accurate movement of the articulators (lips, tongue, etc.) and a variety of facial gestures and expression, (2) natural text-to-speech based on concatenative synthesis which can quickly learn new voices and is capable of hyperarticulation of the kind used to provide feedback in language training, (3) visual speech recognition using an unobtrusive desk-top camera to aid speech recognition and to provide information about the articulators of the student, and (4) auditory speech recognition so the system can understand what is said and also so the system can detect mispronunciations. Research advances are required in speech recognition and understanding, language generation and speech synthesis, recognition of facial cues and head movements, and generation of speech by artificial talking heads to improve the accuracy and to tailor the technology to conversational agents in general and language training in particular. Participatory design experiments will be conducted with schools in the Portland area. The Tucker-Maxon Oral School will use the conversational agents to teach hearing-impaired students to use and understanding auditory and visual speech. This realistic testbed will provide the research team with invaluable feedback on the usefulness of the system.
