NSF Award
2402991
Three-dimensional, large-size, and long-duration immersive content captured from real scenes will have a significant impact in the foreseeable future. Playing a critical role in holographic communication, immersive content allows viewers to exercise 6-degree-of-freedom (6DoF) motion during playback. Most existing research on immersive content delivery focuses on single-viewer scenarios. This project proposes to enable, for the first time, a large number of co-located viewers over a millimeter wave (mmWave) network that is capable of providing high bandwidth, with a single access point and edge server. It suits numerous use cases such as massive interactive demonstration and immersive classroom education. This project aims demonstrable networking and systems research with a synergy among wireless networking & sensing, multimedia systems, machine learning, and computer vision. It will help bridge the digital divide by reducing the cost of multi-user holographic communication and telepresence. It will also provide a platform to conduct various outreach activities and community services. As streaming emerging multimedia content is playing a key role in the post-COVID world, the project will have a high impact on global societies and economies.<br/><br/>To overcome the challenge of supporting multiple users with limited network and compute resources, this project innovates in three key dimensions. First, it will develop an accurate motion prediction model that captures users' collective motion and their interactions, and study how to adapt to changes deviating from training data. Second, this project will leverage mmWave sensing based on FMCW (frequency-modulated continuous-wave) radar to directly incorporate environment reflection profiles into beamforming and mmWave throughput prediction. Assisted by 6DoF motion prediction, this will lead to proactive and fast beamforming, as well as an accurate forecast of mmWave performance that benefits upper layers. To realize environment profiling based on mmWave sensing, the project will design two techniques: collaboratively reconstructing indoor 3D reflectivity maps and building a neural representation of indoor mmWave reflections. Third, this project proposes two approaches to scale up at the application layer: hybrid streaming where certain viewers receive 3D content and others consume content live-transcoded by the edge, and allowing viewers to share a transcoded view. The team will integrate the above thrusts into a holistic framework, implement it on their mmWave testbed with heterogeneous client devices, and conduct extensive evaluations including field trials with real users.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
