Patent Grant
12249151
Automated security and monitoring systems typically rely on imaging devices, such as video cameras, with computer vision capabilities for performing object recognition. Objection recognition may be used to detect anomalies in a recorded scene.
Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments, examples, aspects, and features.
The apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments, examples, aspects, and features described and illustrated so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
The identification of anomalies in the numbers of objects (e.g., people) in a recorded scene is dependent on the context of that scene. A number of people that is anomalous for one scene, or camera view, may be normal for another camera view. Similarly, the number of people and/or the dynamics in the number of people that is normal in a camera view during a first period of time, such as during daytime, may be anomalous during a second period of time, such as during nighttime. Conventional anomaly detection methods require users to define many specific criteria that strictly define what is considered an anomaly, such as a specific crowd size or a specific crowd growth rate.
Additionally, conventional methods of reporting anomalies based on collected statistics of numbers of people in a recorded scene are unreliable because the data relating to the number of detected people is highly correlated. As an example, a computer may determine that, for a period of two minutes, a group of people detected in a region of interest includes twenty people. In the particular region of interest, twenty people may be an anomalous group size that occurs infrequently (e.g., only once per month). However, if the computer detects the number of people in each frame of the video data, and the video data includes ten frames per second (“fps”), after two minutes the computer will calculate that the anomaly of twenty people actually occurs 1,200 times per month. In such instances, the computer may falsely determine that a group of ten people is not anomalous.
As another example, a computer may determine that a group of people detected in the region of interest includes eleven people, and the average number of people in that region of interest is ten people. Even though it may not be anomalous for eleven people to be in a region of interest where ten people is normal, if the computer has never detected eleven people in that region before, then the computer will determine that detecting eleven people has zero probability of occurring. In such instances, the computer would determine that a group of eleven people is anomalous.
Therefore, there is need for a crowd anomaly detection method that takes into account crowd statistics for a region of interest. One example provides a video security system including an image sensor configured to capture video data, and an electronic processor. The electronic processor is configured to detect an anomaly by: obtaining the video data captured using the image sensor; determining a number of people detected in each of a plurality of frames of the video data; for each moment t, determining an average number of people n over a time interval (t−Δt1, t), where Δt1 is greater than or equal to the moment t; for each non-overlapping time interval Δt1, updating a histogram based on the average number of people for the time interval Δt1; and, at each moment t, determining a probability ρ associated with the average number of people n over the time interval (t−Δt1, t) using the histogram. In response to determining that the probability ρ of the average number of people n over the time interval (t−Δt1, t) is less than a threshold probability p*, the electronic processor generates an alert indicating that an anomalous crowd size is detected.
In some aspects, the electronic processor is further configured to fit the histogram to a decay function.
In some aspects, each moment t is approximately 1 second.
In some aspects, Δt1 is a user-defined value.
In some aspects, Δt1 is between approximately 0.3 minutes and approximately 2 minutes.
In some aspects, the electronic processor is configured to provide the alert to a user interface.
In some aspects, the electronic processor is further configured to preprocess the video data by smoothing the number of people detected in each of the plurality of frames of the video data over each moment t.
Another example provides a video security system including an image sensor configured to capture video data, and an electronic processor. The electronic processor is configured to detect an anomaly by: obtaining the video data captured using the image sensor; determining a number of people detected within each of a plurality of frames of the video data; for each moment t, determining an average number of people n1(t) over a time interval (t−Δt1, t), where Δt1 is greater than or equal to the moment t; for each moment t, determining an average number of people n2(t) over a time interval (t−Δt2, t), where Δt2 is greater than or equal to Δt1; for each moment t, determining a change Δn in an average number of people based on at least one selected from the group consisting of the average number of people n1(t) and the average number of people n2(t); for each moment t, selecting one of a plurality of first histograms based on the average number of people n2(t); for each moment t, determining a probability ρ of the change Δn in the average number of people based on the selected one of the plurality of first histograms; and determining whether the probability ρ is less than a threshold probability ρ*. In response to determining that the probability ρ is less than a threshold probability ρ*, the electronic processor generates an alert indicating that an anomalous crowd growth is detected.
In some aspects, the electronic processor is further configured to, for each non-overlapping time interval Δt1, update one of a plurality of second histograms based on the average number of people n2(t) over the time interval (t−Δt2, t) and the change Δn by: assigning the change Δn to one of a plurality of bins based on the average number of people n2(t) over the time interval (t−Δt2, t), wherein each of the plurality of bins is associated with a non-overlapping range of values of the average number of people n2(t) over a plurality of time intervals Δt2; updating an equi-depth histogram with the average number of people n2(t) over the time interval (t−Δt2, t), where the equi-depth histogram includes the plurality of bins; selecting one of the plurality of second histograms based on the assigned one of the plurality of bins, wherein each of the plurality of second histograms is associated with one of the plurality of bins; and updating the selected one of the plurality of second histograms with the change Δn in the average number of people, wherein each of the plurality of first histograms is an aggregation second histograms included in one of a plurality of non-overlapping clusters of second histograms. Histograms included in the plurality of second histograms are a different type of histogram than histograms included in the plurality of first histograms.
In some aspects, the electronic processor is further configured to, for each non-overlapping time interval Δt3 where Δt3 is greater than or equal to Δt1, rearrange boundaries of the plurality of first histograms by: updating the equi-depth histogram by modifying the non-overlapping ranges of values of the average number of people n2(t) associated with the plurality of bins such that each bin includes an equal amount of data; updating the plurality of second histograms according to modifications to the non-overlapping ranges of values of the average number of people n2(t) associated with the plurality of bins such that each of the plurality of second histograms is associated with one of the plurality of bins; updating the plurality of non-overlapping clusters of second histograms based on changes to the plurality of second histograms; and updating the plurality of first histograms based on changes to the plurality of non-overlapping clusters of second histograms.
In some aspects, each of the plurality of second histograms is included in one of the plurality of non-overlapping clusters of second histograms.
In some aspects, each of the plurality of first histograms is a sum of second histograms included in one of a plurality of non-overlapping clusters of second histograms.
In some aspects, a number of the plurality of second histograms is predetermined.
In some aspects, the electronic processor is configured to, for each moment t, determine the change Δn in an average number of people by subtracting the average number of people n2(t) over the time interval (t−Δt2, t) from the average number of people n1(t) over the time interval (t−Δt1, t).
In some aspects, the electronic processor is configured to, for each moment t, determine the change Δn in an average number of people by subtracting the average number of people n1(t−Δt1) calculated over a prior time interval Δt1 from the average number of people n1(t) calculated over a more recent time interval Δt1.
In some aspects, the electronic processor is further configured to extrapolate the selected one of the plurality of first histograms by fitting a Skellam probability distribution to the selected one of the plurality of first histograms.
In some aspects, the electronic processor is configured to provide the alert to a user interface.
In some aspects, Δt2 is a user-defined value.
In some aspects, Δt1 is between approximately 0.3 minutes and approximately 2 minutes.
Another example provides a method for detecting an anomaly in a region of interest of a camera. The method includes capturing video data using an image sensor; and detecting an anomaly by: obtaining the video data captured using the image sensor; determining a number of people detected within each of a plurality of frames of the video data; for each moment t, determining an average number of people n1(t) over a time interval (t−Δt1, t), where Δt1 is greater than or equal to the moment t; for each moment t, determining an average number of people n2(t) over a time interval (t−Δt2, t), where Δt2 is greater than or equal to Δt1; for each moment t, determining a change Δn in an average number of people based on at least one selected from the group consisting of the average number of people n1(t) and the average number of people n2(t); for each moment t, selecting one of a plurality of first histograms based on the average number of people n2(t); for each moment t, determining a probability ρ of the change Δn in the average number of people based on the selected one of the plurality of first histograms; determining whether the probability ρ is less than a threshold probability ρ*; and in response to determining that the probability ρ is less than a threshold probability ρ*, generating an alert indicating that an anomalous crowd growth is detected.
Another example provides a method for detecting an anomaly in a region of interest of a camera. The method includes capturing video data using an image sensor; and detecting an anomaly by: obtaining the video data captured using the image sensor; determining a number of people detected within each of a plurality of frames of the video data; for each moment t, determining an average number of people n1(t) over a time interval (t−Δt1, t), where Δt1 is greater than or equal to the moment t; for each moment t, determining an average number of people n2(t) over a time interval (t−Δt2, t), where Δt2 is greater than or equal to Δt1; for each moment t, determining a change Δn in an average number of people based on at least one selected from the group consisting of the average number of people n1(t) and the average number of people n2(t); for each moment t, selecting one of a plurality of first histograms based on the average number of people n2(t); for each moment t, determining a probability ρ of the change Δn in the average number of people based on the selected one of the plurality of first histograms; determining a frequency f of the change Δn in the average number of people based on the probability ρ; determining whether the frequency f is less than or equal to a threshold frequency f_th; in response to determining that the frequency f is less than or equal to a threshold frequency f_th, generating an alert indicating that an anomalous crowd size change is detected.
In some aspects, the threshold frequency f_th is a user-selected frequency threshold.
In some aspects, when Δn is positive, the frequency f is defined as f=(1−F(Δn))T/Δt1, when Δn is negative, the frequency f is defined as f=(F(Δn))T/Δt1, where F(Δn) is a cumulative distribution function corresponding to a probability density distribution function obtained from the selected one of the plurality of first histograms evaluated at Δn, and T is a time interval defining a measurement unit for the frequency f.
Another example provides a video security system including an image sensor configured to capture video data; and an electronic processor configured to detect an anomaly by: obtaining the video data captured using the image sensor; determining a number of people detected in each of a plurality of frames of the video data; for each moment t, determining an average number of people n over a time interval (t−Δt1, t), where Δt1 is greater than or equal to the moment t; for each non-overlapping time interval Δt1, updating a histogram based on the average number of people for the time interval Δt1; at each moment t, determining a probability ρ associated with the average number of people n over the time interval (t−Δt1, t) using the histogram; and determining a frequency f of the average number of people n based on the probability ρ; determining whether the frequency f is less than or equal to a threshold frequency f_th; in response to determining that the frequency f is less than or equal to a threshold frequency f_th, generating an alert indicating that an anomalous crowd size is detected.
For ease of description, some or all of the example systems presented herein are illustrated with a single exemplar of each of its component parts. Some examples may not describe or illustrate all components of the systems. Other examples may include more or fewer of each of the illustrated components, may combine some components, or may include additional or alternative components.
The user device 32 may be any suitable device for receiving a communication from the camera 24. For example, the user device 32 may be a mobile computing device (e.g., a cell phone, a smart phone, a tablet computer, a laptop computer, a personal digital assistant (PDA), etc.), a desktop computing device, a server computing device, or other networked computing device.
The camera 24 also includes an electronic processor 38 (for example, a microprocessor or other electronic device). The electronic processor 38 is electrically or communicatively coupled to the image sensor 34, a memory 42, and a communication interface 56. The electronic processor 38 includes suitable processing circuitry for performing the methods described herein or any combination of suitable processing circuitry. For example, the electronic processor 38 may include a digital signal processor (DSP), a graphics processing unit (GPU) embedded processor, a vision processing unit, etc. One or more circuit units included in the electronic processor 38 may operate independently or in parallel with one another.
In the example shown, the memory 42 stores data and computer program instructions for performing, among other things, the methods described herein. For example, the memory 42 includes image storage 46 for storing images captured by the image sensor 34, an anomaly detection program 50 executable by the electronic processor 38 for identifying anomalies in a region of interest recorded by the camera 24, and an anomaly detection configuration 54. The anomaly detection configuration 54 includes, for example, a user-defined frequency of anomaly detection. For example, using the user device 32, a user may set the anomaly detection configuration 54 of the camera 24 such that an anomaly is defined as an event that occurs no more than once a week, once a month, once a year, or the like. The memory 42 may include read only memory (ROM), random access memory (RAM), other non-transitory computer-readable media, or a combination thereof.
The camera 24 (e.g., the electronic processor 38) is configured to send and receive data, such as video data, video analytics data, alerts, and/or commands to one or more external devices (e.g., the user device 32) using the communication interface 56. In some instances, the anomaly detection program 50 and the anomaly detection configuration 54 are stored external to the camera 24, and the camera 24 transmits video data to an external device for anomaly detection. Accordingly, although methods relating to the anomaly detection program 50 are described herein with respect to the components of the camera 24, it should be understood that such methods may be performed using components external to the camera 24. The camera 24 may include additional components other than those described with respect to
In some instances, the electronic processor 38 preprocesses the number of people detected in each of the plurality of frames to reduce noise in the detection of the number of people in each frame (at block 112). For example, the electronic processor 38 may smooth the number of people detected in each frame over each moment t. Each moment of time t may be a period of time including a predetermined number of frames (e.g., five frames, ten frames, etc.) In some instances, each moment t is approximately one second. The electronic processor 38 smooths the number of people detected in each frame over each moment t by, for example, determining a median number of people detected in each frame over the moment t. However, the electronic processor 38 may smooth the number of people detected in each frame over each moment t using other methods, such as, for example, averaging the number of people detected in each frame over the moment t.
As described above, reporting anomalies only based on the probability distribution of the number of people detected in each frame of video data results in significant data correlation. Therefore, the electronic processor 38 performs block averaging to decorrelate the data related to the number of people detected in the video data. The electronic processor 38 determines, for each of a plurality of overlapping time intervals Δt1 within the plurality of frames, an average number of detected people over the time interval Δt1 (at block 116). The time interval Δt1 may be a user-defined value included in the anomaly detection configuration 54. The time interval Δt1 may be, for example, a value between approximately 0.3 minutes and approximately 2 minutes. However, other values for the time interval Δt1 are contemplated. The time interval Δt1 may otherwise be referred to as a decorrelation time interval Δt1. For example, as illustrated in
Referring again to
As described above, if the electronic processor 38 has never detected, for example, eleven people in the region of interest before, then, using only the data recorded in the histogram, the electronic processor 38 would determine that detecting eleven people has zero probability of occurring. The determination that detecting eleven people has zero probability would be caused by the limited amount of video data processed by the electronic processor 38 by that point in time. Therefore, as illustrated in
Referring again to
The electronic processor 38 compares the determined probability ρ with a threshold probability ρ*, and determines whether the probability ρ is less than the threshold probability ρ* (at block 128). The threshold probability ρ* may be a user-defined probability threshold included, for example, in the anomaly detection configuration 54. The threshold probability ρ* indicates a frequency of an anomaly occurrence (e.g., a crowd size anomaly occurrence) for the region of interest recorded by the camera 24. For example, a user may select a frequency of anomaly occurrence of no more than once per week, once per month, or another period of time. For example, when the electronic processor 38 determines, using the histogram, that the average number of detected people n corresponds to a value n in the histogram having a sufficiently low probability ρ of occurring (e.g., below the threshold probability ρ*), the electronic processor 38 determines that a crowd size anomaly has occurred.
In response to detecting an anomalous crowd size (YES at block 128), the electronic processor 38 generates an alert indicating that a crowd size anomaly has occurred (at block 132). The electronic processor 38 provides the alert to, for example, the user device 32 over the network 28 and continues performing the anomaly detection method 100 by, for example, repeating the steps of the method 100. When an anomalous crowd size is not detected (NO at block 128), the electronic processor 38 continues the anomaly detection method 100 by, for example, repeating the steps of the method 100.
In some instances, as an alternative to or in addition to detecting a crowd size anomaly, the electronic processor 38 is configured to detect, using histogram data, a crowd growth anomaly.
The method 200 includes obtaining video data captured using the image sensor 34 (at block 204). The video data includes a plurality of frames, and the electronic processor 38 determines a number of specific objects, such as people, detected in each of the plurality of frames (at block 208). The electronic processor 38 detects the number of people in each frame using known computer vision and object detection techniques.
In some instances, the electronic processor 38 preprocesses the number of people detected in each of the plurality of frames to reduce noise in the detection of the number of people in each frame (at block 212). For example, the electronic processor 38 may smooth the number of people detected in each frame over each moment t. Each moment of time t may be a period of time including a predetermined number of frames (e.g., five frames, ten frames, etc.) In some instances, each moment t is incremented by approximately one second. The electronic processor 38 smooths the number of people detected in each frame over each moment t by, for example, determining a median number of people detected in each frame over the moment t. However, the electronic processor 38 may smooth the number of people detected in each frame over each moment t using other methods, such as, for example, averaging or removing outliers in the number of people detected in each frame over the moment t.
For each moment t, the electronic processor 38 determines an average number of people n1(t) over a time interval (t−Δt1, t) (at block 216) within the plurality of frames. In some embodiments, the time interval Δt1 is a user-defined value included in the anomaly detection configuration 54 of the camera 24. The time interval Δt1 may be, for example, a value between approximately 0.3 minutes and approximately 2 minutes. However, other values for the time interval Δt1 are contemplated.
The electronic processor 38 additionally determines, for each moment t, an average number of people n2(t) over a time interval (t−Δt2, t) (at block 220) within the plurality of frames. For a given moment t, the time interval Δt2 begins prior to the most recent time interval Δt1, and the time interval Δt2 is greater than the time interval Δt1. For example, the time interval Δt2 may be approximately 10 minutes, approximately 30 minutes, approximately 60 minutes, or the like. In some instances, the time interval Δt2 is a user-defined time interval included as part of the anomaly detection configuration 54.
For each moment t, the electronic processor 38 determines a change Δn in the number of people based on the average number of people n1(t) over the time interval (t−Δt1, t) and/or the average number of people n2(t) over the time interval (t−Δt2, t) (at block 224). In some instances, the electronic processor 38 determines the change Δn for each moment t by subtracting the average number of people n2(t) over the time interval (t−Δt2, t) from the average number of people n1(t) over the time interval (t−Δt1, t). As described above, the time interval Δt2 may be longer than the time interval Δt1. Therefore, in other words, at a moment in time t, the electronic processor 38 may determine the change Δn in the number of people by subtracting the average number of people over the longer period of time from the average number of people detected at that moment t. For example, when, at a moment t, the electronic processor 38 determines that the average number of people n2(t) over the time interval (t−Δt2, t) is 10 people, and the average number of people n1(t) over the time interval (t−Δt1, t) is 8 people, then the electronic processor 38 determines that the change Δn in the number of people is minus 2 (−2) people.
However, in some instances, the electronic processor 38 determines the change Δn for each moment t by subtracting the average number of people n1(t−Δt1) calculated over a prior time interval Δt1 from the average number of people n1(t) calculated over a more recent time interval Δt1. For example, when the electronic processor 38 determines that the average number of people n1(t−Δt1), calculated over the time interval (t−(2*Δt1)), t−Δt1), is 10 people, and the average number of people n1(t), calculated over the time interval (t−Δt1, t), is 8 people, then the electronic processor 38 determines that the change Δn in the number of people is minus 2 (−2) people.
Anomaly analysis for the change Δn in the number of people varies depending on the average number of people n2(t) detected for that moment t over the time interval Δt2. For example, a change Δn of six people over a time interval Δt2 (e.g., thirty minutes) may be an anomaly if the average number of detected people n2(t) over the time interval Δt2 is zero, but may not be an anomaly if the average number of detected people n2(t) over that same time interval Δt2 is one hundred people. Therefore, the electronic processor 38 generates a plurality of aggregated histograms based on the change Δn in the number of people rather than a single histogram, and, once per non-overlapping time interval Δt1, updates one of the plurality of aggregated histograms with the change Δn (at block 228).
For a respective range of n2, the corresponding histogram describes changes Δn in the number of people and the probability ρ corresponding to each change Δn. To account for values of change Δn that may not have been recorded in the respective histogram before, the electronic processor 38 may approximate the probability distribution of each histogram using, for example a Skellam distribution. For example, the electronic processor 38 may approximate the probability distribution according to the equation:
where μ1 and μ2 are parameters of the distribution, and IΔn(2√{square root over (μ1μ2)}) denotes the modified Bessel function of the first kind. However, other methods of approximating the probability distribution of the histograms are contemplated.
Each bin of the equi-depth histogram corresponds to one of a plurality of primitive histograms generated by the electronic processor 38, such as the plurality of primitive histograms illustrated in
As illustrated in
After updating the equi-depth histogram (at block 304), the electronic processor 38 identifies and selects one of the plurality of primitive histograms corresponding to the average number of people n2(t) for the change Δn (at block 308). The electronic processor 38 then updates the selected one of the plurality of primitive histograms with the change Δn, which, in turn, causes an update to the aggregated histogram for which the selected primitive histogram is included in a cluster (at block 312). In other words, the electronic processor 38 adds a calculated change Δn to the appropriate primitive histogram (one histogram) based on the number of detected people associated with the change Δn to build aggregated histograms that take into account not just changes in the number of detected people but also an initial number of people.
As described above with respect to
Referring again to
However, crowd statistics may change over time, and the electronic processor 38 determines the boundaries of each bin (e.g., the ranges of n2) such that each bin includes an equal amount of data. Therefore, as illustrated in
After modifying or updating the boundaries of the bins, the electronic processor 38 updates the plurality of primitive histograms according to the changes in bin boundaries of the equi-depth histogram (at block 408). The modification of the boundary between two bins may result in a change of shape of one or more primitive histograms. For example, as illustrated in
Based on the updated primitive histograms, the electronic processor 38 updates the primitive histogram clusters (at block 412). The electronic processor 38 compares the plurality of primitive histograms to identify a plurality of clusters, where each of the plurality of clusters includes one or more of the plurality of primitive histograms. The electronic processor 38 determines the primitive histogram clusters based on a similarity of the primitive histograms such that primitive histograms that are similar to one another (e.g., in overall shape or other characteristic) are placed in the same cluster. For example, as illustrated in
As illustrated in
After updating the primitive histogram clusters, the electronic processor 38 recalculates the plurality aggregated histograms (at block 416). As described above, each aggregated histogram corresponds to a primitive histogram cluster, and is an aggregation (e.g., a sum) of the primitive histograms included in the corresponding cluster. As illustrated in
Referring again to
The electronic processor 38 compares the determined probability ρ with a threshold probability ρ* and determines whether the probability ρ is less than the threshold probability ρ* (at block 240). Similar to the threshold probability described above with respect to
In response to detecting an anomalous crowd growth (YES at block 240), the electronic processor 38 generates an alert indicating that a crowd growth anomaly has occurred (at block 244). The electronic processor 38 provides the alert to, for example, the user device 32 over the network 28, and continues performing the anomaly detection method 200 by, for example, repeating the steps of the method 200. When an anomalous crowd size is not detected (NO at block 240), the electronic processor 38 continues the anomaly detection method 200 by, for example, repeating the steps of the method 200.
As described above, crowd size and change in crowd size algorithms may be used to determine a probability value for each moment of time (e.g., each moment of time t) for each measurement of a number of people (crowd size) or each measurement of a change in the number of people (crowd size change). These histograms, including tail portions, may be normalized such that the probability, determined using the function fitted to the histogram and/or the tail portion of the histogram has the meaning of a probability density function. Such a probability may be the value of a histogram bin corresponding to the measurement or corresponding to a value of the function that fits the histogram (crowd size change) or a tail of the histogram (crowd size).
This probability value by itself, however, does not characterize a degree of abnormality of the measurement, and a user who wants to get anomalies reported to him or her may not be interested in the probability of any given event. For example, for a uniform probability distribution, all events may have a low probability, but only events at the edges of the distribution are anomalous. Thus, instead of a probability itself, the user is interested in how close an event is to the “extremely large” measurements or how far it is from the “normal” measurements. At the same time, the user assumes that anomalies are events that happen rarely, and the frequency of the anomalous event characterizes a level of the event's abnormality. That frequency can be used as a parameter that a user can define according to the user's specific needs. For example, a user may want to get events reported to them as anomalies (on average, in long time intervals, and for stationary situations) once in a day, once in a week, once in a month, or the like and different users may want to receive alerts at different frequencies.
Accordingly, some embodiments described herein may define an anomaly as an event that belongs to the tail of the histogram bounded by a predefined value. This value, considered as a threshold for the measurement number of people (crowd size) or the measurement of the change in the number of people (crowd size change), defines a set of events whose total frequency can be calculated using a cumulative distribution function. In such embodiments, the threshold frequency does not define the frequency of any given event, but rather defines the frequency of all anomalous events reported to a particular user. Accordingly, a probability determined according to the methods described above (crowd size or crowd size change) can be converted or mapped to a frequency (a cumulative frequency calculated using a cumulative distribution function) and, as part of controlling alerts reported to a particular user, compared to the user-specific threshold frequency to determine whether an alert for the event should be reported to the user or whether the event should be dropped or filtered from the user's user interface (dashboard) or other type of anomaly report or messaging.
For example, referring now to
The electronic processor 38 obtains video data of a scene (at block 508) and determines crowd events in the video data for each moment t (at block 512). Crowd events are, for example, measurements of average number of people (e.g., calculated according to blocks 108-116 of the method 100) and/or measurements of change in average number of people (e.g., calculated according to blocks 208-224 of the method 200). The electronic processor 38 determines a probability ρ associated with the crowd event (at block 520). The electronic processor 38 may determine the probability ρ in a manner substantially similar to that of block 124 of the method 100 with respect to crowd size anomaly detection and/or block 236 of the method 200 with respect to change in crowd size anomaly detection.
The electronic processor 38 determines a frequency f of the crowd event by, for example, mapping the probability ρ of the crowd event to a frequency of the set of events defined by the measured crowd event (e.g., the average number of people n and/or the change Δn in average number of people) (at block 520). With respect to a crowd event that is a crowd size detection of n_1 people for a given moment t, the electronic processor 38 may determine the frequency as f=(1−F(n_1)) T/Δt, where F(n_1)) is a cumulative distribution function evaluated at the determined average number of people n_1, and Δt is the time interval between each measurement of average number of people (e.g., each moment t). The cumulative distribution function F is the integral of the probability density function ρ by (or obtained from) the normalized crowd size histogram. Frequency describes, for example, an expected frequency of an event defined by detecting a number of people greater than n_1.
With respect to a crowd event that is a positive change in crowd size (crowd growth) detection of Δn_g people for a given moment t, the electronic processor 38 may determine the frequency as f=(1−F(Δn_g)) T/At, where F(Δn_g) is a cumulative distribution function, evaluated at the determined change Δn_g in average number of people, and Δt is the time interval between each measurement of n (e.g., each moment t). The cumulative distribution function F is the integral of the probability density function ρ by (or obtained from) the normalized histogram of crowd size change. Frequency describes, for example, an expected frequency of an event defined by detecting a number of people greater than n_g.
With respect to a crowd event that is a negative change in crowd size (crowd evacuation or a negative crowd growth) detection of Δn_s people for a given moment t, the electronic processor 38 may determine the frequency as f=(F(Δn_s))T/Δt, where F(Δn_s) is a cumulative distribution function, evaluated at the determined change Δn_s in average number of people, and Δt is the time interval between each measurement of n (e.g., each moment t). The cumulative distribution function F is the integral of the probability density function ρ by (or obtained from) the normalized histogram of crowd size change. Frequency describes, for example, an expected frequency of an event defined by detecting of decrease in the number of people greater than by absolute value or equal to n_s.
In response to determining the frequency f of the crowd event based on the probability ρ, the electronic processor 38 determines whether the determined frequency f is equal to or less than the threshold frequency f_th (at block 524). When the determined frequency f is less than or equal to the threshold frequency f_th, a crowd anomaly alert (i.e., an anomalous crowd size alert and/or an anomalous change in crowd size alert) for the crowd event is provided to the user (at block 528).
In contrast, when the determined frequency f is greater the threshold frequency f_th, the electronic processor 38 filters the crowd event from reporting or anomaly alerting for the user (at block 532). Such events that are filtered from user reporting are still used by the electronic processor 38 to, for example, update crowd size histograms (e.g., according to block 120 of the method 100) or update change in crowd size histograms (e.g., according to blocks 228-232 of the method 200). In some embodiments, the electronic processor 38 may report all events and the user-specified frequency threshold may be applied at a user-facing or edge component of the system to appropriately filter what events reported by the electronic processor 38 are reported to the user (i.e., what events the user is alerted to). As noted above, different users may have different thresholds to allow users to establish different anomaly thresholds and specify such thresholds as a frequency rather than a probability, which may be difficult for a user to define.
Accordingly, in some instances, determining whether a probability ρ exceeds a probability threshold ρ* (e.g., according to block 128 of the method 100 and/or block 240 of the method 200) includes mapping the probability ρ to a frequency f (at block 520) and determining whether the frequency f is less than a threshold frequency f_th (at block 524). However, in other embodiments, the frequency determination described herein may be used in combination with a probability threshold. For example, in some embodiments, a probability threshold may be used to control what events are reported by the system (e.g., as an initial filter to control reporting and bandwidth requirements), wherein the user-specific frequency thresholds may be used to further filter what events are reported or otherwise provided to a particular user. In yet other embodiments, no probability thresholds are used but rather two frequency thresholds are used. In these embodiments, a first frequency threshold may control what events are reported by the system (e.g., events reported immediately as an alarm), and a second frequency threshold may be a user-specific threshold that controls what events are reported or otherwise provided to a particular user.
In the foregoing specification, specific examples have been described. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the claimed subject matter. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of present teachings.
The benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential features or elements of any or all the claims.
Moreover, in this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” “has,” “having,” “includes,” “including,” “contains,” “containing,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “comprises . . . a,” “has . . . a,” “includes . . . a,” or “contains . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, contains the element. The terms “a” and “an” are defined as one or more unless explicitly stated otherwise herein. The terms “substantially,” “essentially,” “approximately,” “about,” or any other version thereof, are defined as being close to as understood by one of ordinary skill in the art, and in one non-limiting example the term is defined to be within 10%, in another example within 5%, in another example within 1% and in another example within 0.5%. The term “coupled” as used herein is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is “configured” in a certain way is configured in at least that way but may also be configured in ways that are not listed.
It will be appreciated that some examples may be comprised of one or more generic or specialized processors (or “processing devices”) such as microprocessors, digital signal processors, customized processors and field programmable gate arrays (FPGAs) and unique stored program instructions (including both software and firmware) that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of the method and/or apparatus described herein. Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic. Of course, a combination of the two approaches could be used.
Moreover, an example can be implemented as a computer-readable storage medium having computer readable code stored thereon for programming a computer (e.g., comprising a processor) to perform a method as described and claimed herein. Examples of such computer-readable storage mediums include, but are not limited to, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (Read Only Memory), a PROM (Programmable Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read Only Memory) and a Flash memory. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and ICs with minimal experimentation.
The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.
Additionally, unless the context of their usage unambiguously indicates otherwise, the articles “a,” “an,” and “the” should not be interpreted as meaning “one” or “only one.” Rather these articles should be interpreted as meaning “at least one” or “one or more.” Likewise, when the terms “the” or “said” are used to refer to a noun previously introduced by the indefinite article “a” or “an,” “the” and “said” mean “at least one” or “one or more” unless the usage unambiguously indicates otherwise.
It should also be understood that although certain drawings illustrate hardware and software located within particular devices, these depictions are for illustrative purposes only. In some embodiments, the illustrated components may be combined or divided into separate software, firmware, and/or hardware. For example, instead of being located within and performed by a single electronic processor, logic and processing may be distributed among multiple electronic processors. Regardless of how they are combined or divided, hardware and software components may be located on the same computing device or may be distributed among different computing devices connected by one or more networks or other suitable communication links.
Thus, in the claims, if an apparatus or system is claimed, for example, as including an electronic processor or other element configured in a certain manner, for example, to make multiple determinations, the claim or claim element should be interpreted as meaning one or more electronic processors (or other element) where any one of the one or more electronic processors (or other element) is configured as claimed, for example, to make some or all of the multiple determinations, for example, collectively. To reiterate, those electronic processors and processing may be distributed.
This application is a continuation-in-part of U.S. application Ser. No. 18/325,650, filed May 30, 2023, the entire content of which is hereby incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9183512 | Lehmann et al. | Nov 2015 | B2 |
| 10726956 | Jiang | Jul 2020 | B2 |
| 12100216 | Kudo | Sep 2024 | B2 |
| 20090222388 | Hua | Sep 2009 | A1 |
| 20100316257 | Xu | Dec 2010 | A1 |
| 20100322516 | Xu | Dec 2010 | A1 |
| 20110115920 | Wang | May 2011 | A1 |
| 20140161315 | Ostrovsky-Berman | Jun 2014 | A1 |
| 20150313530 | Kodra | Nov 2015 | A1 |
| 20160140399 | Yano | May 2016 | A1 |
| 20180046778 | Jiang | Feb 2018 | A1 |
| 20210110145 | Alcock et al. | Apr 2021 | A1 |
| 20220138475 | Chowdhury | May 2022 | A1 |
| 20220254162 | Felemban | Aug 2022 | A1 |
| 20220262121 | Iqbal | Aug 2022 | A1 |
| 20230101369 | Hirata | Mar 2023 | A1 |
| 20230386216 | Kudo | Nov 2023 | A1 |
| 20240087328 | Kajiki | Mar 2024 | A1 |
| 20240144490 | Chan | May 2024 | A1 |
| 20240161496 | Latapie | May 2024 | A1 |
| Number | Date | Country |
|---|---|---|
| 3135393 | Oct 2020 | CA |
| 103839065 | Jun 2014 | CN |
| 109241845 | Jan 2019 | CN |
| Entry |
|---|
| Bhuiyan, et al., “Video analytics using deep learning for crowd analysis: a review,” Multimedia Tools and Applications, Published Mar. 29, 2022, (28 pages). |
| Sinha, et al., “A Survey and analysis of Crowd Anomaly Detection Techniques,” IEEE, 2021 Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks, (5 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20240404290 A1 | Dec 2024 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18325650 | May 2023 | US |
| Child | 18512904 | US |
