NSF Award
2312839
Most videos streamed on the Internet are sequences of flat two-dimensional (2D) images captured by regular video cameras. A Point-Cloud Video (PCV) records the three-dimensional (3D) geometry and color information of a dynamic scene using a sequence of point-cloud frames, each of which is a discrete set of data points in space captured by a 3D scanner or a camera array. A captured PCV can be viewed by a viewer from any angle at any viewing distance to obtain a truly immersive visual experience. Deployed PCV will enable new opportunities in many domains, including education, business, healthcare and entertainment, etc. Meanwhile, streaming PCV over the Internet requires significantly higher bandwidth and lower latency than the traditional 2D video; processing PCV also incurs high computation loads on the source and receiver sides. The project addresses the communication and computation challenges of PCV, and will contribute towards the wide deployment of high quality and robust PCV streaming through the global Internet. <br/><br/>The project is developing object-centric, view-adaptive, progressive, and edge-aware PCV coding and streaming designs to deliver robust and high-quality viewer Quality-of-Experience (QoE) in the faces of network and viewer dynamics. It includes several research thrusts: 1) The project team is investigating object-based coding schemes that maximally explore the spatial and temporal coherences of points within the same object for PCV compression. A hierarchical slicing structure is being developed for representing dynamic octrees to enable rate and Field-of-View (FoV) adaptations during streaming. A viewer's FoV is predicted by considering the other viewers' FoVs and the movements of objects in a PCV; 2) The project team is studying progressive PCV streaming that gradually refines the spatial resolution of each region in the predicted FoV as its playback time approaches. The researchers are investigating novel hybrid-learning bases PCV streaming solutions and joint rate and playback speed adaptation for low-latency live streaming; 3) The project team is designing edge PCV caching algorithms that work seamlessly with edge-based PCV post-processing. They are also exploring the gains of multi-user delivery from edge-based multicast and cross-user FoV predictions; 4) A fully-functional PCV streaming testbed is being developed to conduct modern dance education experiments by streaming PCVs of professional dancers to dance students in on-demand and live fashions. The project will generate better tools for the review of human motion in three dimensions that can also benefit many other applications, including sports science/sports medicine, occupational and physical therapy, rehabilitation engineering, and media production. Valuable research opportunities are being created for graduate and undergraduate students, especially women and minority students. The project is also creating opportunities for dance students and practitioners to participate in STEM Research.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
