NSF Award
1816627
There is an increasingly large array of spoken language interfaces available, such as virtual assistants and telephone customer service interfaces. These technologies both (1) recognize the words spoken by a user and (2) extract actionable information, such as the topic of the user's query and the degree of match between the query and documents in a database. Such applications are typically treated as a pipeline of automatic speech transcription followed by text processing to extract the meaning. This project aims to develop technology that directly extracts meaning from speech, while using a variety of linguistic information along the way. This approach is intended to mitigate the effects of speech recognition errors, as well as to use all of the meaning-bearing information in speech, such as intonation. This work is expected to have long-term broad impact through technological advances, as well as immediate broad impact through the PI's involvement in local schools and mentoring for a diverse set of visiting students.<br/><br/>The technical goals of this work are (1) to do high-quality natural language processing directly on speech; (2) to seamlessly integrate domain knowledge into end-to-end speech models; (3) improve the performance-vs.-resources tradeoff; and (4) develop models for embedding arbitrary speech signals into meaning-bearing representations. The process of mapping from speech to meaning can be viewed as a hierarchy of tasks, from the most basic acoustic-phonetic tasks to the deepest semantic tasks. The experimental work will focus on two task hierarchies: a "retrieval" hierarchy including query-by-example search, keyword spotting, semantic speech search; and a "recognition" hierarchy including phonetic recognition, word recognition, parsing, and topic identification. The main technical approaches to be developed include hierarchical multitask learning methods for incorporating domain knowledge and mitigating low-data settings, as well as new models for acoustic-semantic speech embedding.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
