NSF Award
2402878
This project builds and tests new technology to help people with severe speech and motor impairments better control computers and use them to communicate. Such impairments make it very hard to use voice, touch, keyboard, and other common ways of interacting with computers. One alternative that can help is "single-switch" interaction, where systems allow people to use the muscle control they do have to simulate clicking a single button. For instance, Stephen Hawking famously composed lectures and interview responses with his computer by twitching his cheek; other interfaces use blinking, or puffing air into a special sensor. However, these single-switch interaction methods, and the interfaces built based on them, are much slower and more error-prone than voice, touch, or keyboard interfaces. In this project, the research team collaborates with people with severe disabilities to develop technology to make it faster, easier, and more accurate for this set of people to communicate and use computers. The key general idea is to use additional information about a person's context such as where they are looking, what they have done recently, and other clues to their intentions to help computers guess, and suggest, complicated communication goals based on simple single-switch interactions.<br/><br/>To meet these goals, this project leverages machine learning and a novel user interface to synthesize four currently under-utilized information sources. First, users are likely to be looking at a target when they select it; unlike methods that use eye gaze as a pointer, such passive information requires no conscious eye effort from the user and may provide information even when a user's gaze is not precise enough for selection by itself. Second, modern large language models can often anticipate what a user is interested in writing and can likely learn to leverage contextual clues (e.g., who the user is writing to or topics suggested by a communication partner) with only a small number of training examples. Third, patterns of behavior across users and time can offer clues; for example, similar users may provide information about a new user, or a user might exhibit different behaviors at different times of the day. Fourth, sequences of noisy input actions can jointly inform a desired user action such as writing a word. The project combines these four sources of information with a user's single-switch input to improve speed and accuracy of text entry and computer interaction tasks. This project's innovations are designed and tested with users with disabilities in an extensive, multi-pronged set of participatory design activities and experiments. Participatory design with this set of users is particularly challenging due to their communication speed. This project therefore investigates a novel combination of participatory design methods in which stakeholders provide feedback via an asynchronous messaging platform, in co-design interviews, and via surveys designed to be easily completed with single-switch interaction techniques.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
