Systems, methods, and computer readable media for predictive analytics and change detection from remotely sensed imagery

Information

  • Patent Grant
  • 12050994
  • Patent Number
    12,050,994
  • Date Filed
    Monday, November 15, 2021
    3 years ago
  • Date Issued
    Tuesday, July 30, 2024
    10 months ago
  • CPC
  • Field of Search
    • CPC
    • G06N3/08
    • G06N3/04
    • G06N3/044
    • G06N3/045
    • G06F18/217
    • G06F18/22
    • G06F18/2413
    • G06T7/74
    • G06T7/97
    • G06T2207/20084
    • G06T2207/10016
    • G06T2207/10032
    • G06T2207/20076
    • G06T2207/20081
    • G06T2207/30184
    • G06T7/00
    • G06V10/764
    • G06V10/82
    • G06V20/17
    • G06V20/176
    • G06V20/188
  • International Classifications
    • G06T7/00
    • G06F18/21
    • G06F18/22
    • G06N3/04
    • G06N3/08
    • G06T7/73
    • G06V10/764
    • G06V10/82
    • G06V20/10
    • G06V20/17
    • Disclaimer
      This patent is subject to a terminal disclaimer.
      Term Extension
      395
Abstract
Systems and methods are provided for automatically detecting a change in a feature. For example, a system includes a memory and a processor configured to analyze a change associated with a feature over a period of time using a plurality of remotely sensed time series images. Upon execution, the system would receive a plurality of remotely sensed time series images, extract a feature from the plurality of remotely sensed time series images, generate at least two time series feature vectors based on the feature, where the at least two time series feature vectors correspond to the feature at two different times, create a neural network model configured to predict a change in the feature at a specified time, and determine, using the neural network model, the change in the feature at a specified time based on a change between the at least two time series feature vectors.
Description
TECHNICAL FIELD

Embodiments of the present disclosure relate to the field of change detection, predictive analytics, and predictive maintenance from remotely sensed imagery.


BACKGROUND OF THE INVENTION

Real world objects change over time. For example, a roof of a house degrades over time due to natural events, such as rain, wind, and sunlight. As another example, a tree can grow or die over time. Information related to these changes can be useful. For example, an insurance company can assess a risk level for a house depending on the condition of the roof of the house and/or the condition of the surrounding trees. Although these changes can be detected and assessed manually, the manual process is costly, time-consuming, and error-prone.


Therefore, there is a need in the art for improved systems and methods for detecting and analyzing changes in real world objects.


SUMMARY

This invention includes systems and methods that allow automatic detection of changes of an object over time. Non-limiting examples of an object include a house, a driveway, a car, a farmland, a construction site, a neighborhood/community, etc. These changes may occur on the appearance of an object. For example, a displacement of a shingle on a roof. These changes could be dimensional changes such as a growth of a tree over time, or changes in the water level of a body of water.


Disclosed subject matter includes, in one aspect, a system for automatically detecting a change of an object feature over time. The system includes a memory that stores a computational module configured to analyze a change of an object feature over a period of time. The system also includes a processor that is coupled to the memory and configured to execute the stored computational module. Upon execution, the system would detect a change of an object feature over time using a neural network model.


In an exemplary scenario, upon execution, the system would receive a plurality of remotely sensed time series images over a period of time. In some instances, these images may be transmitted from a remote image capturing device such as a satellite camera. Subsequently, the system would extract at least one feature from the plurality of remotely sensed time series images. For example, shingles placement on a roof that is under construction. The system then generates a time series feature vector corresponding to the feature captured in an image at a particular time. To identify a change, the system would generate at least two time series feature vectors among the plurality of remotely sensed time series images. The system then creates a neural network model to predict a change in the feature at a specified time based on the feature vectors. The specified time can be now or at certain time in the future. To determine the change, the system would identify the change (or change pattern) associated with the time series feature vectors and use the change to output a prediction.


Disclosed subject matter includes, in another aspect, a process for detecting a change of a feature through a plurality of remotely sensed time series images. The process includes receiving a plurality of remotely sensed time series image. The process then proceeds to extract a feature from the plurality of remotely sensed time series images. The process then proceeds to generate at least two time series feature vectors associated with the feature extracted from the plurality of remotely sensed time series images at two different time period. The process then proceeds to create a neural network model configured to predict a change (or a change pattern) in the feature at a specific time. The process then proceeds to determine the change of the feature based on the change (or the change pattern) between the time series feature vectors.


Disclosed subject matter includes, in yet another aspect, a non-transitory computer readable medium having executable instructions operable to cause a system to receive a plurality of remotely sensed time series images. The instructions are further operable to cause the system to extract a feature from the plurality of remotely sensed time series images. The instructions are further operable to cause the system to generate at least two time series feature vectors based on the feature. The generated time series feature vectors correspond to the state of the feature at different times. The instructions are further operable to cause the system to create a neural network model configured to predict a change in the feature at a specified time. The instructions are further operable to cause the system to determine, using the neural network model, the change in the feature at a specified time based on a change between the at least two time series feature vectors.


Before explaining example embodiments consistent with the present disclosure in detail, it is to be understood that the disclosure is not limited in its application to the details of constructions and to the arrangements set forth in the following description or illustrated in the drawings. The disclosure is capable of embodiments in addition to those described and is capable of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein, as well as in the abstract, are for the purpose of description and should not be regarded as limiting.


These and other capabilities of embodiments of the disclosed subject matter will be more fully understood after a review of the following figures, detailed description, and claims.


It is to be understood that both the foregoing general description and the following detailed description are explanatory only and are not restrictive of the claimed subject matter.





BRIEF DESCRIPTION OF THE DRAWINGS

Various objects, features, and advantages of the disclosed subject matter can be more fully appreciated with reference to the following detailed description of the disclosed subject matter when considered in connection with the following drawings, in which like reference numerals identify like elements.



FIG. 1 illustrates a method for detecting and analyzing a change in an object feature through a plurality of remotely sensed time series images in accordance with some embodiments of the present disclosure.



FIG. 2 illustrates a process for determining a change in a feature at a specified time in accordance with some embodiments of the present disclosure.



FIG. 3 illustrates a system for detecting and analyzing a change in an object feature through a plurality of remotely sensed time series images in accordance with some embodiments of the present disclosure.



FIG. 4 illustrates a block diagram of a server in accordance with some embodiments of the present disclosure.





DETAILED DESCRIPTION

In the following description, numerous specific details are set forth regarding the systems and methods of the disclosed subject matter and the environment in which such systems and methods may operate, etc., in order to provide a thorough understanding of the disclosed subject matter. It will be apparent to one skilled in the art, however, that the disclosed subject matter may be practiced without such specific details, and that certain features, which are well known in the art, are not described in detail in order to avoid complication of the disclosed subject matter. In addition, it will be understood that the examples provided below are exemplary, and that it is contemplated that there are other systems and methods that are within the scope of the disclosed subject matter.


Disclosed systems, methods, and computer readable media can be used to detect changes in time series data, provide predictive analytics based on the detected changes, and perform predictive maintenance using time series data. For instance, the disclosed system can detect progressive material change over time to determine the desirable schedule for maintenance. In some embodiments, predictive maintenance means that a measure for means of preventing total breakdown of an item (for example the roof of a house) can be avoided by sending a craftsman/handyman to fix the gradual damage once the damage (which is over a predefined threshold) was assessed by a method incorporating predictive analytics. Hence the damage can be kept under more control, causes less cost than in the catastrophic case. Further, the chances to send a professional inspector to the site without a serious cause (hence causing unnecessary costs) can be kept at a minimum. In some embodiments, a predictive model means a state (for example a roof condition) that can be extrapolated several steps into the future and thus estimated how it will evolve over time by taking into account the state history. In some embodiments, the time series data can include remotely sensed imagery, which can be acquired by using an image acquisition device (e.g., the image acquisition device described in FIG. 3). The changes that can be detected can include changes in a built environment, vegetation (e.g., changes in landscape), topology (e.g. changes due to natural disasters, such as a landslide, tornado, volcano, earthquake, etc.), and any other suitable changes that can be detected. In some embodiments, the training set of the disclosed system can differentiate important and less important visual changes. For example, the disclosed system can determine that certain features contained within the image are due to seasonal changes while others are due to hazardous events such as a wildfire.


According to some embodiments, deep learning of time series data can be applied to infer a single state and/or a sequence of states of a scene or an object within the time series data. In some embodiments, deep learning of time series data can be applied to perform predictive analysis of the future evolution of such states. In some embodiments, a computer neural network model, such as a deep neural network (DNN), can be used for deep learning. In some embodiments, a scene can include one or more objects (e.g., an artificial object, such as a house, a building, a roof, a pool, a driveway, a shed, or any other type of artificial object, and/or a natural object, such as a tree, a body of water, vegetation, or any other type of natural object). For example, in remotely sensed imagery of property parcels, the images can include a house, a roof (which can be part of the house), a shed, a pool, trees, rocks, and/or any other similar items.


According to some embodiments, images of certain locations can be taken on a regular basis (e.g., every month, every six months, every year, or any other suitable regular interval) and/or on an irregular basis (e.g., images taken as needed). For example, in the case of remotely sensed imagery of property parcels, images of the property parcels can be taken on the first day of each month. These images can be used to create deep learning models, such as neural network models, to provide systems, methods, and computer readable media for change detection, predictive analytics, and predictive maintenance.


According to some embodiments, the use of deep learning can be categorized into at least three types: many-to-one inference; many-to-many inference; and many-to-many prediction. In the many-to-one inference type, the current state can be inferred from a sequence of time steps taken up until the present time. For example, given remotely sensed images of a house over the course of twelve months, where one image was acquired each month, one can infer on the current condition of the roof by presenting the sequence of images to the DNN. In the many-to-many inference type, the state at each time step up until the present time can be inferred. For example, given remotely sensed images of a house over the course of twelve months, where one image was acquired each month, one can infer on the condition of the roof of the house in each month. In the many-to-many prediction type, the evolution of a state or multiple states in the future can be predicted. For example, given remotely sensed images of a house over the course of twelve months, where one image was acquired each month, one can predict the state or states in the future months to come with increasing uncertainty. In some embodiment, the uncertainty is based on a confidence interval associated with one or more sampled instances. For example, in the case of roof condition, the confidence interval associated with a near term prediction using recently obtained images may be higher than the confidence interval associated with a distant future prediction using recently obtained images. In some embodiments, a recurrent neural network (RNN) model can receive, as an input, the sequence of images up until the present time. The model can then run without input from one or several time steps in order to generate a prediction for one or multiple time steps. For example, if a time series spans over 6 months, at the fourth month the prediction model can run without inputs from the first and second month.


According to some embodiments, instead of estimating a single time instance target prediction (following a many-to-one paradigm), an RNN model can be used for sequence-to-sequence (many-to-many) mapping. In these embodiments, the RNN can predict the development and/or behavior of an estimated target value over time. This can be achieved either by applying the RNN to a running window or to an indefinitely long sequence. When applying the RNN to the running window, in some embodiments, the RNN resets after a predefined amount of samples have been obtained. In some embodiments, RNN can also be trained on a stream of data (which, in some embodiments, can be infinitely in length) by not resetting after a predefined amount of samples.


According to some embodiments, predictive analytics refer to a machine learning domain in which models trained on time series of data can be used to predict the future behavior of a real-world system. In some embodiments, a time series can be generated using a convolutional neural network (CNN) in the context of remotely sensed imagery. The CNN can serve as a feature extractor For example, a CNN can be applied to a time series of successive remotely sensed images of a physical location (e.g., a house or a property parcel) to extract a feature from the physical location and generate a feature vector corresponding to the feature. For example, the CNN can extract the roof as a feature from each of the successive remotely sensed images of a house. In some embodiments, the resulting feature vectors and the corresponding time steps can be used to train an RNN. The RNN can be used to handle dynamic temporal behavior, where the target value for training can be taken from several steps into the future of the time series. For example, the RNN can receive time-series of feature vectors extracted from the image of the roof of a house over a period of time. Based on this time-series of feature vectors, the state of the roof at a particular time (e.g., past or present) can be inferred and/or the state of the roof in the future can be predicted. In some embodiments, the state of the roof can be assigned a score (e.g., a scale of 1 to 5, where 1 represents the state being very bad and 5 represents the state being excellent). For example, for a time-series [t0, t1, t2, t3, t4, t5, t6, t7, t8] (where the time t0 to t8 represent regular intervals at which the images were taken in the past), the feature vectors could correspond to scores [5, 5, 4, 1, 5, 5, 4, 4, 3]. In some embodiments, each time stamp is associated with a score. In some embodiments, each time stamp is associated with other suitable indicators, classes, and/or distributions. Based on these scores, the state of the roof of the house can be inferred for each of the time instances in the time-series. It can also be inferred that at time ta, the roof likely experienced a sudden damage possibly due to a disaster (e.g., a fire or a falling tree) and that at time t5, a new roof likely had been installed. Moreover, the state of the roof at the present (which is time t8 in this example) can be predicted to be 3 based on the trend. The future state(s) of the roof can be predicted similarly.


According to some embodiments, the resulting estimation or prediction can be interpreted as the probability of a certain state occurring N steps in the future. In some embodiments, the CNN and the RNN can be trained disjointly in two successive steps. In some embodiments, weights of CNN models that have been pre-trained on a certain problem domain can be used. For example, weights of CNN models that have been pre-trained on remotely sensed imagery of house roofs can be used. The weights are the parameter that models the connection between the neurons of the CNN. In some embodiments, the RNN can be built using one to several successive layers. In some instances, a deep RNN can span over several recurrent layers in sequence.



FIG. 1 illustrates a method 100 for detecting and analyzing a change in a feature in remotely sensed time series images according to certain embodiments. In some embodiments, the method utilizes a DNN model for predictive analytics. In some embodiments, the DNN model can include two parts: a CNN 110 and an RNN 130. In some embodiments, the CNN 110 is trained independently from the RNN 130.


According to certain embodiments, the CNN contains a Deep Learning algorithm which can take in an image, assign importance (learnable weights and biases) to various aspects/objects in the image and be able to differentiate one form the other. The CNN is trained to learn the parameters and transformation values associated with an image. In an iterative process, CNN can be trained to identify and map spatial parameters to the space of parcel boundaries, and thus identify the parcel. A training example for the CNN is disclosed in U.S. patent application Ser. No. 15/253,488, which is incorporated by reference in its entirety.


According to some embodiments, the RNN includes multiple copies of the same network, each passing a message to a successor. Like the CNN, the RNN can accept an input vector x, and gives an output vector y. However, unlike a CNN, the output vector's contents are influenced not only by the input provided by the user, but also by the entire history of inputs that the user has provided in the past. According to some embodiments, the RNN has an internal state that gets updated every time a new input is received.


According to some embodiments, the CNN 110 serves as a feature extractor. The CNN 110 can receive a set of time series images 101_1, 101_2, . . . , 101_n that respectively correspond to the time series [t0, t1, . . . , tn] (where n can be equal to or greater than 1, and when n is equal to 1, the time series set would include only two elements: t0 and t1). In some embodiments, the time series images 101_1, 101_2, . . . , 101_n can be remotely sensed images (e.g., aerial images).


According to some embodiments, for each of the time-series images 101_1, 101_2, . . . , 101_n, the CNN 110 can extract a feature vector x, resulting in a time series of feature vectors x0 121_1, x1 121_2, . . . xn 121_n, which respectively correspond to the time-series images 101_1, 101_2, . . . , 101_n. In some embodiments, the CNN can identify an object based on the object's primitive features such as edges, shapes, and/or any combination thereof. For example, the CNN may recognize a house's roof based on features such as the roof's edges and shapes.


According to some embodiments, the RNN 130 can receive the time series of feature vectors x0 121_1, x1 121_2, . . . xn 121_n as inputs from CNN 110. In some embodiments, the RNN 130 can make inference on a single prediction y 150 and/or a time-series of predictions [y0 . . . yk] (not shown in the figure) (where k can be equal to or greater than 1) that can depend on the problem domain.



FIG. 2 illustrates a process for determining a change in a feature at a specified time in accordance with some embodiments of the present disclosure. In some embodiments, the process 200 can be modified by, for example, having steps combined, divided, rearranged, changed, added, and/or removed.


According to some embodiments, at step 202, a server receives a plurality of remotely sensed time series images. In some embodiments, the plurality of remotely sensed time series images are captured at regular intervals. In some embodiments, the plurality of remotely sensed time series images are captured at irregular intervals. The process 200 then proceeds to step 204.


At step 204, the server extracts one or more features from the plurality of remotely sensed time series images received at step 202. In some embodiments, the feature can be extracted by a CNN. The feature can be a permanent feature, a temporary feature, or a changing feature. For example, in a roofing context, the permanent feature is a roof's shingles; the temporary feature is observable changes due to sun light or weather; and the changing feature is shingle conditions, missing shingles, streaks, and/or spots. In some embodiments, the feature is at least one of a property condition, a neighborhood condition, a built environment, a vegetation state, a topology, a roof, a building, a tree, or a body of water. In some embodiments, the feature is defined by an aggregated state derived from the sub-states of partial features (or sub-components). For example, the feature can be an overall neighborhood state, which can be an aggregate state of a plurality of sub-states corresponding to sub-components (e.g., the houses within the neighborhood, the roads within the neighborhood, the bodies of water within the neighborhood, and any other sub-features that can part of the neighborhood). The process 200 then proceeds to step 206.


At step 206, the server generates one or more time series feature vectors based on the feature, each of which corresponds to the feature from the each of the plurality of time series images extracted at step 204. In some embodiments, the plurality of time series feature vectors can be generated by the CNN. In some embodiments, the server further determines a plurality of time series scores, each of which corresponds to the feature from the each of the plurality of time series images. The process 200 then proceeds to step 208.


At step 208, the server creates a neural network model configured to predict a change in the feature based on the plurality of time series feature vectors generated in step 206 In some embodiments, insignificant changes can be discarded when the neural network model is created. Insignificant changes may include changes that are not important to the state of the feature. For example, a shadow on a roof is not significant to the state of the roof. The neural network model can be trained to identify these insignificant changes. For example, the training dataset can contain two images of a roof, one with shadow and one without shadow. The system then applies the same ground-truth mapping value is to both images irrespective of the shadow effect, hence, forcing the neural network to become invariant to shadows when trained with this data. In some embodiments, the neural network model can be composed of a neural network that is at least one of an RNN or a CNN. The process 200 then proceeds to step 210.


At step 210, the server determines, using the neural network model, a change in the feature at a specified time based on a change between/among the time series feature vectors. In some embodiments, the specified time is the present. In some embodiments, the specified time is in the future. In some embodiments, the change in the feature can be predicted based on a plurality of time series scores.



FIG. 3 illustrates a system 300 for detecting and analyzing a change in a feature in remotely sensed time series images. The system 300 can include a server 304, at least one client device 301 (e.g., client devices 301-1, 301-2, . . . , 301-N), an imagery acquisition device 303, a local storage medium 305, and a remote storage medium 306. All components in the system 300 can be coupled directly or indirectly to a communication network 302. The components described in the system 300 can be further broken down into more than one component and/or combined together in any suitable arrangement. Further, one or more components can be rearranged, changed, added, and/or removed. For example, in some embodiments, the system 300 can obtain the data from third-party vendors. In other embodiments, the system 300 can directly acquire data through the imagery acquisition device 303.


Each client device 301 can communicate with the server 304 to send data to, and receive data from, the imagery analysis server 304 via the communication network 302. Each client device 301 can be directly or indirectly coupled to the server 304. Additionally, each client device 301 can be connected to the server 304 via any other suitable device(s), communication network, or a combination thereof. A client device 301 can include, for example, a desktop computer, a mobile computer, a tablet computer, a cellular device, a smartphone, a television, or any computing system that is capable of performing the computation processes described above.


The server 304 is configured to receive imagery data from the imagery acquisition device 303. The imagery analysis server 304 can extract, analyze, and/or label structural and/or geospatial information of the received imagery data based on the techniques disclosed in this present disclosure. In some embodiments, a classifier can be trained and/or maintained in the server 304. The server 304 is shown as a single server; however, the server 304 can include more than one server. For example, in some embodiments, the server 304 can include multiple modular and scalable servers and/or other suitable computing resources. The server 304 can support elastic computing, which can dynamically adapt computing resources based on demand. The server 304 can be deployed locally and/or remotely in a third-party cloud-based computing environment. In some embodiments, within the server 304 or any other suitable component of system 300, a device or a tool—including those described in the present disclosure—can be implemented as software and/or hardware.


The imagery acquisition device 303 is configured to provide the server 304 with imagery data. In some embodiments, the imagery acquisition device 303 can acquire satellite imagery, aerial imagery, radar, sonar, LIDAR, seismography, or any other suitable mode or combination of modes of sensory information. In some embodiments, the system 300 does not include the imagery acquisition device 303 and can obtain imagery data from third-party vendors. In some embodiments, the imagery acquisition device can be part of a GIS. The system 300 includes two storage media: the local storage medium 305 and the remote storage medium 306.


The local storage medium 305 can be located in the same physical location as the server 304, and the remote storage medium 306 can be located at a remote location or any other suitable location or combination of locations. In some embodiments, the system 300 includes more than one local storage medium, more than one remote storage medium, and/or any suitable combination thereof. In some embodiments, the system 300 may only include the local storage medium 305 or only include the remote storage medium 306.


The system 300 can also include one or more relational databases, which include scalable read replicas to support dynamic usage. The one or more relational databases can be located in the local storage medium 305, the remote storage medium 306, the server 304, and/or any other suitable components, combinations of components, or locations of the system 300.


The communication network 302 can include the Internet, a cellular network, a telephone network, a computer network, a packet switching network, a line switching network, a local area network (LAN), a wide area network (WAN), a global area network, or any number of private networks currently referred to as an Intranet, and/or any other network or combination of networks that can accommodate data communication. Such networks may be implemented with any number of hardware and software components, transmission media and/or network protocols. In some embodiments, the communication network 302 can be an encrypted network. While the system 300 shows the communication network 302 as a single network, the communication network 302 can also include multiple interconnected networks described above.



FIG. 4 illustrates a block diagram of the server 304 in accordance with some embodiments of the present disclosure. The server 304 includes a processor 402, a memory 404, and a module 406. The server 304 may include additional modules, fewer modules, or any other suitable combination of modules that perform any suitable operation or combination of operations.


The processor 402 is configured to implement the functionality described herein using computer executable instructions stored in temporary and/or permanent non-transitory memory. The processor can be a general purpose processor and/or can also be implemented using an application specific integrated circuit (ASIC), programmable logic array (PLA), field programmable gate array (FPGA), and/or any other integrated circuit.


The processor 402 can execute an operating system that can be any suitable operating system (OS), including a typical operating system such as Windows, Windows XP, Windows 7, Windows 8, Windows Mobile, Windows Phone, Windows RT, Mac OS X, Linux, VXWorks, Android, Blackberry OS, iOS, Symbian, or other OS.


The module 406 and/or other modules in the server 304 can be configured to cause the processor 402 or the server 304 to perform various functions described herein. These functions can include the functions related to the CNN 110 and RNN 130, as described in connection with FIG. 1 above. In some embodiments, within the module 406 or any other suitable component of the server 304, a device or a tool can be implemented as software and/or hardware.


In some embodiments, the module 406 can be implemented in software using the memory 404. The memory 404 can be a non-transitory computer readable medium, flash memory, a magnetic disk drive, an optical drive, a programmable read-only memory (PROM), a read-only memory (ROM), or any other memory or combination of memories.


It is contemplated that systems, devices, methods, and processes of the disclosure invention encompass variations and adaptations developed using information from the embodiments described herein. Adaptation and/or modification of the systems, devices, methods, and processes described herein may be performed by those of ordinary skill in the relevant art.


Throughout the description, where articles, devices, and systems are described as having, including, or comprising specific components, or where processes and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are articles, devices, and systems of the present disclosure that consist essentially of, or consist of, the recited components, and that there are processes and methods according to the present disclosure that consist essentially of, or consist of, the recited processing steps.


It should be understood that the order of steps or order for performing certain action is immaterial so long as the disclosure remains operable. Moreover, two or more steps or actions may be conducted simultaneously. The mention herein of any publication, for example, in the Background section, is not an admission that the publication serves as prior art with respect to any of the claims presented herein. The Background section is presented for purposes of clarity and is not meant as a description of prior art with respect to any claim.


It is to be understood that the disclosed subject matter is not limited in its application to the details of construction and to the arrangements of the components set forth above or illustrated in the drawings. The disclosed subject matter is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, may readily be utilized as a basis for the designing of other structures, methods, and systems for carrying out the several purposes of the disclosed subject matter.

Claims
  • 1. A method for feature change detection comprising: receiving a set of time-series images depicting a common feature;generating at least two feature vectors for the common feature based on the set of time-series images, wherein the at least two feature vectors are associated with the common feature at different timestamps; anddetermining a change in the common feature based on the at least two feature vectors, using a neural network model, wherein the neural network model receives as input the at least two feature vectors.
  • 2. The method of claim 1, wherein the images of the set of time-series images are captured at regular intervals.
  • 3. The method of claim 1, wherein the set of time-series images is remotely sensed.
  • 4. The method of claim 1, wherein the common feature is at least one of a property condition, a neighborhood condition, a built environment, a vegetation state, a topology, a roof, a building, a tree, or a body of water.
  • 5. The method of claim 1, wherein the common feature is associated with an aggregated state based on one or more sub-states of one or more partial features.
  • 6. The method of claim 1, wherein the common feature is extracted by a convolutional neural network, wherein the at least two feature vectors are generated by the convolutional neural network.
  • 7. The method of claim 1, wherein the neural network model comprises at least one of a recurrent neural network or a convolutional neural network.
  • 8. The method of claim 1, further comprising extracting the common feature from each image in the set of time-series images, wherein the at least two feature vectors are generated based on the extracted common feature.
  • 9. The method of claim 1, further comprising determining at least two time-series scores, wherein the at least two time-series scores are associated with the common feature at different timestamps.
  • 10. The method of claim 9, wherein the change in the common feature is determined based on the at least two time-series scores.
  • 11. The method of claim 1, wherein the change is determined for a specified timestamp.
  • 12. A system for feature change detection, comprising a computer processor configured to: receive a set of time-series images depicting a shared feature;generate at least two feature vectors for the shared feature based on the set of time-series images, wherein the at least two feature vectors are associated with the shared feature at different timestamps; anddetermine a change in the shared feature based on the at least two feature vectors, using a neural network model.
  • 13. The system of claim 12, wherein the set of time-series images are captured at non-regular intervals.
  • 14. The system of claim 12, wherein the shared feature comprises a building, wherein the feature vectors are representative of a visual appearance of the building at different timestamps.
  • 15. The system of claim 12, wherein the change is determined for a specified timestamp.
  • 16. The system of claim 15, wherein the specified timestamp is a past timestamp.
  • 17. The system of claim 12, wherein the at least two time-series feature vectors are generated by a convolutional neural network.
  • 18. The system of claim 12, wherein the neural network model comprises at least one of a recurrent neural network or a convolutional neural network.
  • 19. The system of claim 12, wherein the computer processor is further configured to extract the shared feature from each image in the set of time-series images, wherein the at least two feature vectors are generated based on the extracted shared feature.
  • 20. The system of claim 12, wherein the set of time-series images comprise remotely sensed images.
RELATED APPLICATION

This application is a continuation of U.S. application Ser. No. 16/682,615 filed Nov. 13, 2019, which claims the benefit of U.S. Provisional Application No. 62/767,257, filed Nov. 14, 2018, which is incorporated by reference herein in its entirety. This application is also related to the following application, which is incorporated by reference in its entirety: U.S. patent application Ser. No. 15/253,488, titled “Systems and methods for analyzing remote sensing imagery,” which was filed on Aug. 31, 2016.

US Referenced Citations (189)
Number Name Date Kind
5633946 Lachinski et al. May 1997 A
6044171 Polyakov et al. Mar 2000 A
6249616 Hashimoto Jun 2001 B1
7327902 Ritt et al. Feb 2008 B2
7873238 Schultz et al. Jan 2011 B2
7945117 Hermosillo et al. May 2011 B2
8078436 Pershing et al. Dec 2011 B2
8081841 Schultz et al. Dec 2011 B2
8145578 Pershing et al. Mar 2012 B2
8170840 Pershing May 2012 B2
8207964 Meadow et al. Jun 2012 B1
8209152 Pershing Jun 2012 B2
8396256 Lu et al. Mar 2013 B2
8401222 Thornberry et al. Mar 2013 B2
8452125 Schultz et al. May 2013 B2
8515125 Thornberry et al. Aug 2013 B2
8531472 Freund et al. Sep 2013 B2
8542880 Thornberry et al. Sep 2013 B2
8548248 Mitchell et al. Oct 2013 B2
8655070 Yang et al. Feb 2014 B1
8660382 Schultz et al. Feb 2014 B2
8670961 Pershing et al. Mar 2014 B2
8731234 Ciarcia et al. May 2014 B1
8760285 Billman et al. Jun 2014 B2
8774525 Pershing Jul 2014 B2
8818770 Pershing Aug 2014 B2
8823732 Adams et al. Sep 2014 B2
8825454 Pershing Sep 2014 B2
8848983 Brewington et al. Sep 2014 B1
8965812 Linville Feb 2015 B2
8977520 Stephens et al. Mar 2015 B2
8995757 Ciarcia et al. Mar 2015 B1
9117310 Coene et al. Aug 2015 B2
9129376 Pershing Sep 2015 B2
9135737 Pershing Sep 2015 B2
9141880 Ciarcia Sep 2015 B2
9141925 Jung et al. Sep 2015 B2
9147287 Ciarcia Sep 2015 B2
9159130 Kneepkens Oct 2015 B2
9159164 Ciarcia Oct 2015 B2
9183538 Thornberry et al. Nov 2015 B2
9244589 Thornberry et al. Jan 2016 B2
9275496 Freund et al. Mar 2016 B2
9292913 Schultz et al. Mar 2016 B2
9329749 Thornberry et al. May 2016 B2
9389084 Chen et al. Jul 2016 B1
9437029 Schultz et al. Sep 2016 B2
9514568 Pershing et al. Dec 2016 B2
9520000 Freund et al. Dec 2016 B2
9536148 Gross Jan 2017 B2
9542738 Schultz et al. Jan 2017 B2
9576201 Wu et al. Feb 2017 B2
9599466 Pershing Mar 2017 B2
9743046 Giuffrida et al. Aug 2017 B2
9753950 Schultz et al. Sep 2017 B2
9805489 Schultz et al. Oct 2017 B2
9836882 Freund et al. Dec 2017 B2
9911042 Cardona et al. Mar 2018 B1
9911228 Pershing et al. Mar 2018 B2
9922412 Freeman et al. Mar 2018 B1
9933254 Thornberry et al. Apr 2018 B2
9933257 Pershing Apr 2018 B2
9959581 Pershing May 2018 B2
9959653 Schultz et al. May 2018 B2
9972126 Freund et al. May 2018 B2
10002415 Shen et al. Jun 2018 B2
10169680 Sachdeva et al. Jan 2019 B1
10197391 Thornberry et al. Feb 2019 B2
10229532 Freund et al. Mar 2019 B2
10311302 Kottenstette et al. Jun 2019 B2
10338222 Schultz et al. Jul 2019 B2
10346935 Thornberry et al. Jul 2019 B2
10402676 Wang et al. Sep 2019 B2
10430902 Rawat et al. Oct 2019 B1
10453147 Davis Oct 2019 B1
10455197 Giuffrida et al. Oct 2019 B2
10489953 Schultz et al. Nov 2019 B2
10503843 Keane Dec 2019 B2
10515414 Pershing Dec 2019 B2
10528960 Pershing et al. Jan 2020 B2
10529029 Okazaki Jan 2020 B2
10571575 Schultz et al. Feb 2020 B2
10573069 Freund et al. Feb 2020 B2
10648800 Thornberry et al. May 2020 B2
10650285 Okazaki May 2020 B1
10663294 Pershing et al. May 2020 B2
10685149 Pershing Jun 2020 B2
10796189 Wang et al. Oct 2020 B2
10839469 Pershing Nov 2020 B2
10937178 Srinivasan Mar 2021 B1
10943464 Hayward et al. Mar 2021 B1
11037255 Ganev et al. Jun 2021 B1
11164257 Devereaux Nov 2021 B1
11210552 Kossyk et al. Dec 2021 B2
11430069 Pedersen et al. Aug 2022 B1
11430076 Martin et al. Aug 2022 B1
11488255 Mast et al. Nov 2022 B1
11599706 Mark et al. Mar 2023 B1
11631235 Vianello et al. Apr 2023 B2
11651552 Valladolid et al. May 2023 B2
11676298 Portail et al. Jun 2023 B1
11861843 Portail et al. Jan 2024 B2
20030146913 Shen et al. Aug 2003 A1
20040153330 Miller et al. Aug 2004 A1
20050084178 Lure et al. Apr 2005 A1
20050154657 Kim et al. Jul 2005 A1
20060165267 Wyman et al. Jul 2006 A1
20080133319 Adiga et al. Jun 2008 A1
20080255862 Bailey et al. Oct 2008 A1
20100014781 Liu et al. Jan 2010 A1
20100111428 Yu et al. May 2010 A1
20100217566 Wayne et al. Aug 2010 A1
20120005109 Stinson Jan 2012 A1
20120120069 Kodaira et al. May 2012 A1
20120141014 Lepikhin et al. Jun 2012 A1
20120265675 Lawrence Oct 2012 A1
20130006676 Helitzer et al. Jan 2013 A1
20130054476 Whelan Feb 2013 A1
20130121571 Gokturk et al. May 2013 A1
20140015924 Pryor Jan 2014 A1
20140139515 Kim et al. May 2014 A1
20140180989 Krizhevsky et al. Jun 2014 A1
20150029176 Baxter et al. Jan 2015 A1
20150088886 Brouwer et al. Mar 2015 A1
20150131852 Sweetser et al. May 2015 A1
20150186953 Gross Jul 2015 A1
20150187015 Adams et al. Jul 2015 A1
20150242747 Packes et al. Aug 2015 A1
20150269598 Terrazas et al. Sep 2015 A1
20150286786 El-Baz et al. Oct 2015 A1
20150302529 Jagannathan Oct 2015 A1
20150347872 Taylor et al. Dec 2015 A1
20160026900 Ando Jan 2016 A1
20160027051 Gross Jan 2016 A1
20160048934 Gross Feb 2016 A1
20160063516 Terrazas et al. Mar 2016 A1
20160092959 Gross Mar 2016 A1
20160162986 Ghosh Jun 2016 A1
20160292800 Smith Oct 2016 A1
20170032481 Dsouza et al. Feb 2017 A1
20170236024 Wang Aug 2017 A1
20170270650 Howe et al. Sep 2017 A1
20170352099 Howe et al. Dec 2017 A1
20170352100 Shreve Dec 2017 A1
20170357984 Takamatsu et al. Dec 2017 A1
20180018553 Bach et al. Jan 2018 A1
20180096420 Sun et al. Apr 2018 A1
20180101917 Fujita Apr 2018 A1
20190057465 Spath Feb 2019 A1
20190188516 Porter et al. Jun 2019 A1
20190213412 Kottenstette et al. Jul 2019 A1
20190213438 Jones et al. Jul 2019 A1
20190294647 Brouwer et al. Sep 2019 A1
20190304026 Lyman et al. Oct 2019 A1
20190311044 Xu et al. Oct 2019 A1
20190354772 Tasli et al. Nov 2019 A1
20200034861 Lundgren et al. Jan 2020 A1
20200082168 Fathi et al. Mar 2020 A1
20200134733 Maddox et al. Apr 2020 A1
20200134753 Vickers Apr 2020 A1
20200151500 Kossyk et al. May 2020 A1
20200160030 Lavi May 2020 A1
20200226373 Kottenstette et al. Jul 2020 A1
20200279653 Hayward Sep 2020 A1
20200348132 Du et al. Nov 2020 A1
20210065340 El-Khamy et al. Mar 2021 A1
20210089811 Strong Mar 2021 A1
20210110439 Jayne et al. Apr 2021 A1
20210118165 Strong Apr 2021 A1
20210124850 Messervy et al. Apr 2021 A1
20210133936 Chandra et al. May 2021 A1
20210151195 Hayward May 2021 A1
20210182529 Singh et al. Jun 2021 A1
20210188312 Shikari et al. Jun 2021 A1
20210199446 Marschner et al. Jul 2021 A1
20210312710 Fathi et al. Oct 2021 A1
20220012518 Sutherland Jan 2022 A1
20220012918 Kozikowski et al. Jan 2022 A1
20220036486 Dhandapani et al. Feb 2022 A1
20220036537 Johnson et al. Feb 2022 A1
20220099855 Li et al. Mar 2022 A1
20220292650 Amirghodsi et al. Sep 2022 A1
20220405856 Hedges et al. Dec 2022 A1
20230011777 Brown et al. Jan 2023 A1
20230119132 Cebulski et al. Apr 2023 A1
20230143198 Vianello May 2023 A1
20230230220 Portail et al. Jul 2023 A1
20230230250 Vianello Jul 2023 A1
20230237775 Portail Jul 2023 A1
Foreign Referenced Citations (9)
Number Date Country
101549596 Aug 2012 CN
105917335 Aug 2016 CN
108197583 Jun 2018 CN
108629287 Oct 2018 CN
3340106 Jun 2018 EP
4033426 Jul 2022 EP
2018055287 Apr 2018 JP
2012115594 Aug 2012 WO
2018119035 Jun 2018 WO
Non-Patent Literature Citations (49)
Entry
Zheng, Chuanxia , et al., “Pluralistic Image Completion”, arXiv:1903.04227v2. Apr. 5, 2019.
Zhu, Hengshu , et al., “Days on Market: Measuring Liquidity in Real Estate Markets”, KDD '16, Aug. 13-17, 2016, San Francisco, CA, USA.
“Cape analytics now detects yard debris from Geospatial imagery”, Dec. 18, 2019, pp. 1-4 (Year: 2019).
“CLIP: Connecting Text and Images”, https://openai.com/blog/clip/, Jan. 5, 2021.
“Gdal_viewshed”, © 1998-2022 Frank Warmerdam, Even Rouault, and others, downloaded Aug. 3, 2021, https://gdal.org/programs/gdal_viewshed.html.
“Geospatial Property Analytics: Supercharging AVMs for SFR Investing”, Cape Analytics, downloaded Oct. 3, 2022, https://capeanalytics.com/blog/geospatial-property-analytics-avm-sfr-investing/.
“Hail Risk Assessment Hail Risk Score, Hail Damage Score”, Verrisk Analytics, downloaded Oct. 15, 2021, https://www.verisk.com/insurance/products/hail-risk-assessment/.
“HOA Lookup City of Goodyear GIS”, https://www.arcgis.com/apps/webappviewer/index.html?id=816ddfbd119e49d4a74bb30cb4592c22, first downloaded May 31, 2023.
“Industry-Leading Wildfire Risk Insights”, zesty.ai, downloaded Oct. 15, 2021, https://www.zesty.ai/wildfire-risk.
“Manage wildfire risk at the address level”, Verisk, downloaded Oct. 15, 2021, https://www.verisk.com/siteassets/media/downloads/underwriting/location/location-fireline.pdf?1 =.
“The Real Value of Change Detection: A ChangeFinderTM Story Map”, https://www.eagleview.com/newsroom/2018/02/real-value-change-detection/, EagleView US, Feb. 28, 2018.
“Unique Building Identifier (UBID)”, Office of Energy Efficiency & Renewable Energy, https://www.energy.gov/eere/buildings/unique-building-identifier-ubid#:˜:text=UBID, first downloaded Dec. 22, 2022.
“ValPro+—Cape Analytics”, Cape Analytics, https://web.archive.org/web/20210301122341/https://capeanalytics.com/valpro/, downloaded Oct. 3, 2022.
“Weiss Analytics Launches ValPro+, First AVM Powered by Cape Analytics' Instant Property Condition Assessments”, Cape Analytics, Feb. 19, 2021, https://capeanalytics.com/resources/weiss-analytics-valpro-instant-property-condition-assessments/.
Ali, Imtiaz , “Unsupervised video analysis for counting of wood in river during floods”, Springer-Verlag, 2009, pp. 578-587 (year: 2009).
Ambrosch, Philipp , et al., “System and Method for Property Group Analysis”, U.S. Appl. No. 18/333,803, filed Jun. 13, 2023.
Arrowsmith, Ellie , et al., “Wildfire Fuel Management and Risk Mitigation”, Insurance Institute for Business & Home Safety, zesty.ai, Apr. 2021.
Baranzini, Andrea , et al., “A Sight for Sore Eyes Assessing the value of view and landscape use on the housing market”, Publisher:CRAG—Haute école de gestion de Genève, Genève, 2007.
Carlberg, Matthew , et al., “Classifying Urban Landscape in Aerial Lidar Using 3D Shape Analysis”, IEEE, 978-1-4244-5654-3, 2009, pp. 1701-1704 (Year: 2009).
Chen, Yixing , et al., “development of city—modeling”, Energy and Buildings, 2019, pp. 252-265 (Year 2019).
Corbin, Matthew , et al., “System and Method for Property Data Management”, U.S. Appl. No. 18/104,969, filed Feb. 2, 2023.
Dai, Xieer , et al., “Viewshed Effects and House Prices: Estimating a Spatial Hedonic Model”, https://alrov.tau.ac.il/wp-content/uploads/2021/08/%D7%93%D7%A0%D7%99%D7%90%D7%9C-%D7%A4%D7%9C%D7%96%D7%A0%D7%A9%D7%98%D7%99%D7%99%D7%9F-2019.pdf, downloaded Aug. 3, 2021.
Eigen, David , et al., “Depth Map Prediction from a Single Image using a Multi-Scale Deep Network”, Proceedings of Advances in Neural Information Processing Systems (NIPS), Jun. 2014.
Ermolin, S. V., et al., “Predicting Days-on-Market for Residential Real Estate Sales”, (Year: 2016).
Friedland, Carol , et al., “Integrated Aerial-Based and ground Based- Scales”, ResearchGate, Feb. 9, 2015, pp. 1-7 (Year: 2015).
Hamaguchi, Ryuhei , et al., “Rare Event Detection using Disentangled Representation Learning”, arXiv:1812.01285, submitted Dec. 4, 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Long Beach, CA, USA, Jun. 15-20, 2019.
Jiang, Shasha , et al., “Automatic urban debris zone extraction from post-hurricane very high-resolution satellite and aerial imagery”, Geomatics, Natural Hazards and Risk, 7:3, 933-952, https://www.tandfonline.com/doi/full/10.1080/19475705.2014.1003417, 2016.
Johnson , et al., “Valuing curb appeal”, The Journal of Real Estate Finance and Economics 60 (2020): 111-133. (Year: 2020).
Kalinicheva, Ekaterina , et al., “Unsupervised Change Detection Analysis in Satellite Image Time Series using Deep Learning Combined with Graph-Based Approaches”, https://hal.archives-ouvertes.fr/hal-02749683, Jun. 3, 2020.
Kamara, Amadu Fullah , et al., “A hybrid neural network for predicting Days on Market a measure of liquidity in real estate industry”, ScienceDirect, Knowledge-Based Systems, 208 (2020) 106417.
Kotova, Nadia , et al., “Liquidity in Residential Real Estate Markets”, Apr. 2021, https://anthonyleezhang.github.io/pdfs/sfipd.pdf.
Lee, Jun-Tae , et al., “Image Aesthetic Assessment Based on Pairwise Comparison—A Unified Approach to Score Regression, Binary Classification, and Personalization”, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 27, 2019-Nov. 2, 2019.
Lee, Xiang , et al., “System and Method for Property Analysis”, U.S. Appl. No. 18/303,990, filed Apr. 20, 2023.
Liu , et al., “Learning multi-instance deep ranking and regression network for visual house appraisal”, IEEE Transactions on Knowledge and Data Engineering 30.8 (2018): 1496-1506. (Year: 2018).
Maire, Frederic , et al., “A Convolutional Neural Network for Automatic Analysis of Aerial Imagery”, 2014 International Conference on Digital Image Computing: Techniques and Applications (DICTA), 8 pages (2014).
Maloy, Mark A., et al., “An Accuracy Assessment of Various GIS-Based Viewshed Delineation Techniques”, 2016.
Mansouri, Babak , “Earthquake building damage detection using VHR Satellite data”, Case study; Two villages near Sarpol-e Zahab) , JSEE, Jul. 29, 2018, pp. 45-55 (Year: 2018).
Moussa, A. , et al., “Man-made objects classification from satellite/aerial imagery using neural networks”, International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences—ISPRS Archives, 4 pages (2010).
Muhs, Sebastian , “Automatic delineation—maps”, Elsevier, 2016, p. 71-84 (Year: 2016).
Neptune, Nathalie , et al., “Automatic Annotation of Change Detection Images”, Sensors, published Feb. 5, 2021.
Plank, Simon , “Rapid damage—Sentinel”, Remote Sensing, May 28, 2014, pp. 4870-4906 (Year 2014).
Poursaeed, Omid , et al., “Vision-based Real Estate Price Estimation”, arXiv:1707.05489 [cs.CV], https://doi.org/10.48550/arXiv.1707.05489, Machine Vision and Applications, 29(4), 667-676, 2018.
Radford, Alec , et al., “Learning Transferable Visual Models From Natural Language Supervision”, arXiv:2103.00020v1 [cs.CV] Feb. 26, 2021.
Tran, Kieu , “Applying Segmentation and Neural Networks to Detect and Quantify Marine Debris From Aerial w Images Captured by an Unmanned Aerial System and Mobile Device”, Aug. 2018, pp. 1-70 (Year 2018).
Urbis, Arvydas , et al., “GIS-Based Aesthetic Appraisal of Short-Range Viewsheds of Coastal Dune and Forest Landscapes”, Forests 2021, 12, 1534. https://doi.org/10.3390/f12111534.
Van Leer, Kevin , “Responding to the Alarm Bell: Enabling a Better Understanding of Defensible Space and Wildfire Risk for Individual Structures”, CAPE Analytics, Apr. 11, 2019, https://capeanalytics.com/resources/responding-to-the-alarm-bell-enabling-a-better-understanding-of-defensible-space-and-wildfire-risk-for-individual-structures/.
Wang, Jianjun , et al., “Generating Viewsheds without Using Sightlines”, Photogrammetric Engineering & Remote Sensing, vol. 66 No. 1, Jan. 2000, pp. 87-90.
Wang, Xuenting , et al., “Predicting the Attractiveness of Real-Estate Images by Pairwise Comparison using Deep Learning”, 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Shanghai, China, 2019, pp. 84-89, doi: 10.1109/ICMEW.2019.0-106.
Yu, Bailang , et al., “Automated derivation—method”, Landscape and urban planning, 2020, pp. 210-219 (Year: 2010).
Related Publications (1)
Number Date Country
20220076055 A1 Mar 2022 US
Provisional Applications (1)
Number Date Country
62767257 Nov 2018 US
Continuations (1)
Number Date Country
Parent 16682615 Nov 2019 US
Child 17526769 US