NSF Award
1924751
This project will provide algorithms for automatic, adaptive detection and description of changes in time-lapse imagery - a series of images obtained from the same scene over a long time frame. We wish to identify when there are "significant" changes in the scene, and provide a text description of those changes in natural English, where a human analyst provides feedback to determine what kinds of changes are important (e.g., a building being built, deforestation) or unimportant (e.g., seasonal changes). We will in particular focus on satellite or aerial imagery, for which data sets commonly used to train image recognition systems are inadequate. This fundamental research has the potential to transform many application domains, including surveillance, autonomous robotics, monitoring of civil infrastructure, high-throughput microscopy, and climate science, in all of which change is a common and significant occurrence. Our work on novel formulations of change description will also impact on core areas of computer vision and natural language processing, where many similar problems arise. The project will involve graduate students training and postdoctoral associate mentoring.<br/><br/>Detecting change is one of the fundamental abilities for an agent perceiving and interacting with the world. Describing changes in natural language is key to making human interaction with such an agent efficient, accurate and transparent. Our work will advance both the theoretical understanding of these goals and the practical methods for implementing them. Specifically, we will address the above challenges for developing novel mathematical frameworks for localizing gradual changes and describing those changes in natural language; we will develop theoretical and practical means to analyze and overcome corruption in observed imagery; and we will develop novel theory and methods for leveraging human feedback. This work will yield fundamental advances in the fields of change point detection and localization, image reconstruction using deep neural networks and limited training data, and multi-armed bandit methodology for adapting to human feedback.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
