NSF Award
2312761
Augmented reality (AR) represents an exciting new frontier for mobile applications, merging the physical and virtual worlds to facilitate real-time interactions between users and their surrounding physical environments. Next-generation AR is centered on interactive and environmentally-aware applications, which sense the state of the real world via computationally heavy perception algorithms, with results augmented in real-time for multiple users. However, current environmentally-aware AR applications are well-recognized to lack sufficient resilience for long-term, large-scale, and safety-critical deployments. The goal of this project is to exploit the collaborative nature of next-generation AR applications to provide reliable, real-time environmental awareness that can adapt to the surrounding environment, user mobility, and computing resource availability. The core insight of this project is that instead of simply competing with one another for network and compute resources, AR devices can collaborate with one another to run and adapt AR applications to optimize overall utility. The project will involve modeling environmental awareness and mobility patterns in AR, incorporating these models into an emulator to facilitate experimentation and training, and designing reinforcement learning policies that capture environmental awareness and dynamic user relationships. These policies will be deployed in a set of new collaborative AR applications, such as AR-based physical rehabilitation, and evaluated via expansive studies with multiple groups of users in a wide range of conditions. <br/><br/>This project will lay the groundwork for resilient next-generation AR applications, which can benefit multiple domains, including retail, manufacturing, and healthcare. The project will develop an example next-generation AR-supported rehabilitation application, which will be showcased to clinicians across multiple hospital systems and will be integrated into the PIs’ K-12 and public outreach efforts. The multi-user AR emulator created as part of this project will be open-sourced to enable the development of a wide range of other applications and will be designed to be extensible to additional use cases, such as AR-aided human-robot collaboration. Moreover, the machine learning techniques for dynamic user collaboration developed as part of this project will be applicable to other cases in which multiple entities interact dynamically in a coordinated manner, such as supply chain management. The project will train several cohorts of undergraduate students, and results will be presented at multiple events and venues aimed at broadening participation in computing.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
