Patent Grant
11862000
Detecting motion and counting moving objects may be useful for intruder detection, monitoring rental properties, judging an effectiveness of marketing campaigns, building design and layout, and/or the like.
The following detailed description of example implementations refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.
A traditional motion detection system may utilize multiple electronic devices to measure a quantity of vehicles, drones, people, and/or the like traversing a certain passage or entrance. The electronic devices may include video cameras, smart-flooring sensors, infrared beams, thermal imaging systems, and/or the like. However, such electronic devices may be intrusive and may require additional hardware and/or software to effectively operate. Furthermore, current standards fail to define methods to count a quantity of people in a zone since detecting simultaneous motions and segregating the motions into unique objects is challenging. Thus, current motion detection systems consume computing resources (e.g., processing resources, memory resources, communication resources, and/or the like), networking resources, and/or other resources associated with installing intrusive electronic monitoring devices, maintaining the electronic monitoring devices, maintaining software associated with the electronic monitoring devices, and/or the like.
Some implementations described herein provide a device (e.g., a network device, such as a router, a set-top box (STB), a cloud-based device, and/or the like) that detects people and movement in a zone. For example, the device may receive radio frequency (RF) transmissions from access points provided in the zone, and may calculate channel state information (CSI) for the access points based on the RF transmissions. The device may identify CSI phases that satisfy a phase threshold to eliminate surrounding movement in the zone and to focus on an entry location of the zone, and may perform a short-time Fourier transform of the CSI phases to generate a frequency versus time graph. The device may perform a spectrogram analysis of the frequency versus time graph or may process the frequency versus time graph, with the assistance of a model, to determine a quantity of people in the zone and start and stop times associated with entries and exits of the people to and from the zone. The device may perform actions based on the quantity of people and the start and stop times.
In this way, the device detects people and movement in a zone. For example, the device may collect CSI associated with devices communicating with the device via communication links in the zone. CSI is information that estimates a channel by representing channel properties of a communication link. CSI describes how a signal propagates from a transmitting device to a receiving device and reveals a combined effect of disturbances (e.g., scattering, fading, and power decay) with distance. The device may utilize the CSI to calculate a quantity of people in the zone and/or movement of people in the zone. Thus, the device may conserve computing resources, networking resources, and/or other resources that would have otherwise been consumed by installing intrusive electronic monitoring devices, maintaining the electronic monitoring devices, maintaining software associated with the electronic monitoring devices, and/or the like.
As shown in
As further shown in
The NLOS RF transmissions may be generated based on an orthogonal frequency division multiplexing (OFDM) scheme. OFDM is a bandwidth-efficient digital multicarrier modulation scheme for wideband wireless communications. OFDM is a form of signal modulation that divides a high data rate modulating stream (e.g., an NLOS RF transmission) into multiple streams and places the streams onto many slowly modulated narrowband close-spaced subcarriers. In this way, the multiple streams are less sensitive to frequency-selective fading. In OFDM, an overall spectrum band may be divided into many small and partially overlapped signal-carrying frequency bands called subcarriers.
As further shown in
In some implementations, the network device 105 may include a container (e.g., a Wi-Fi sensing container or a Linux container) that calculates the CSI for the access points 110 based on the NLOS RF transmissions, and/or processes the CSI. The container may standardize the CSI format into a data structure, and may be utilized for processing the CSI. This may limit operation of other containers on the network device 105. In some implementations, the network device 105 may utilize the access point 110, the connected device 115, and/or the processing system 120 to process the CSI.
In some implementations, the network device 105 may provide the NLOS RF transmissions to the connected device 115 and the connected device 115 may calculate the CSI for the access points 110 based on the NLOS RF transmissions. The network device 105 may utilize a subchannel (e.g., of the CSI) with a greatest variance for processing. A phase difference between the first access point 110-1 and the second access point 110-2 may be valuable in determining a dynamically changing phase, by eliminating a static phase offset.
As shown in
As further shown in
As further shown in
As shown in
In some implementations, the network device 105 may store the bi-dimensional locations of the access points 110 in a data structure (e.g., a table, a database, a list, and/or the like) associated with the network device 105. The data structure may also include unique identifiers for the access points 110, manufacturer information associated with the access points 110, device names associated with the access points 110, and/or the like. The bi-dimensional locations of the access points 110 may be utilized for determining entry and exit of people to and from the zone, relative motion in the zone, mapping zones in an area, and/or the like. In case of an emergency, the bi-dimensional locations of the access points 110 may be utilized to retrieve exact locations of people and motion in the zone.
As shown in
In some implementations, when determining whether the person is entering or exiting the zone over the time period, the network device 105 may determine, at a first time (t1), a first phase (Q1) associated with the first access point 110-1, and may determine, at the first time (t1), a second phase (Q2) associated with the second access point 110-2. The network device 105 may calculate a first phase difference (e.g., Q2-Q1) based on the first phase (Q1) and the second phase (Q2), and may determine, at a second time (t2), a third phase (Q3) associated with the first access point 110-1. The network device 105 may determine, at the second time (t2), a fourth phase (Q4) associated with the second access point 110-2, and may calculate a second phase difference (e.g., Q4-Q3) based on the third phase (Q3) and the fourth phase (Q4). The network device 105 may determine whether the person is entering or exiting the zone (e.g., during the time period t2-t1) based on the first phase difference and the second phase difference.
In some implementations, motion direction may be detected by utilizing the phase difference between the access points 110 relative to the network device 105. Adding phase thresholds based on triangulation may enable noise from people already in the zone to be disregarded. Only motion at the entry location is detected to prevent false entries and exits from being detected.
As shown in
As shown in
As shown in
Y[n]=a*x[n]+(1−a)*Y[n−1],
where Y[n] corresponds to a current output (e.g., the exponential moving average), Y[n−1] corresponds to a previous output, x[n] corresponds to a current input, and a corresponds to a step value (e.g., a modifiable value, such as 0.1).
If the exponential moving average exceeds a noise threshold, the network device 105 may determine that entry or exit of people to or from the zone has started. If the exponential moving average decreases, the network device 105 may determine that motion of people to or from the zone has ceased. The network device 105 may utilize the exponential moving average to determine velocities of people entering or exiting the zone by correlating the exponential moving average with a phase difference method of determining a direction of traversal of the people. The network device 105 may determine that more people are moving in the zone when a motion energy increases. The network device 105 may determine the motion energy as follows:
Energy=Σi=1windowlength/2magnitude2,
where magnitude values may be normalized Fast Fourier Transform (FFT) coefficients calculated over the time window length. However, the motion energy may increase when a person runs into the zone. With enough data collected, the network device 105 may generate a lookup table based on the average velocity, the motion energy, and the quantity of people entering the zone, where the quantity of people=function (motion energy, velocity).
As further shown in
A collection of frequency versus time graphs in different environments (e.g., homes, malls, offices, and/or the like) may be utilized for training the machine learning model. The machine learning model may not require velocities of the people in the zone since the machine learning model may inherently process the velocities in a more accurate way than complex signal processing. The CNN model may require lesser samples for training, but may only classify a quantity of people entering or exiting the zone. The deep learning single shot detector model may require more samples for training, but may be more accurate and may be utilized to detect the velocities and the start and stop times associated with entries and exits of the people to and from the zone.
As shown in
In some implementations, performing the one or more actions includes the network device 105 determining that an intruder has entered the zone and contacting a law enforcement agency. For example, the network device 105 may detect a person entering the zone when no one should be in the zone (e.g., when owners of the zone are not home). The network device 105 may determine that the detected person is an intruder and may contact a law enforcement agency to respond to a potential crime. In this way, the network device 105 conserves computing resources, networking resources, and/or other resources that would have otherwise been consumed by installing intrusive electronic monitoring devices, maintaining software associated with the electronic monitoring devices, and/or the like.
In some implementations, performing the one or more actions includes the network device 105 determining that the quantity satisfies a rental threshold quantity and generating additional charges for rental of the zone. For example, the zone may be a rental home that limits a quantity of people in the rental home (e.g., via the rental threshold quantity). The rental home may charge extra fees for people entering the zone over the rental threshold quantity. If the network device 105 determines that the quantity of people in the zone satisfies (e.g., exceeds) the rental threshold quantity, the network device 105 may generate additional charges for the rental of the zone. In this way, the network device 105 conserves computing resources, networking resources, and/or other resources that would have otherwise been consumed by maintaining the electronic monitoring devices, maintaining software associated with the electronic monitoring devices, and/or the like.
In some implementations, performing the one or more actions includes the network device 105 determining that the quantity satisfies a capacity threshold and causing additional people to be prevented from entering the zone. For example, the zone may include a capacity threshold for safety purposes (e.g., fire safety purposes, building code purposes, and/or the like). If the network device 105 determines that the quantity of people in the zone satisfies the capacity threshold, the network device 105 may alert an entity in charge of the zone to prevent additional people from entering the zone. In this way, the network device 105 conserves computing resources, networking resources, and/or other resources that would have otherwise been consumed by installing intrusive electronic monitoring devices, maintaining the electronic monitoring devices, and/or the like.
In some implementations, performing the one or more actions includes the network device 105 causing crowd control or foot traffic control to be implemented in the zone based on the quantity and the start and stop times. For example, the zone may be a sports arena associated with a sporting event. When the sporting event ends, the network device 105 may determine that the zone is becoming overcrowded based on the quantity and the start and stop times. The network device 105 may alert an entity in charge of the sports arena to implement crowd control or foot traffic control in the zone based on the zone becoming overcrowded. In this way, the network device 105 conserves computing resources, networking resources, and/or other resources that would have otherwise been consumed by maintaining the electronic monitoring devices, maintaining software associated with the electronic monitoring devices, and/or the like.
In some implementations, performing the one or more actions includes the network device 105 causing retail displays in the zone to be modified based on the quantity and the start and stop times. For example, the zone may be in a store selling merchandise and network device 105 may determine that a larger quantity of people are in the zone during certain times of the day. Based on this determination, the network device 105 may alert an entity in charge of the store to modify retail displays (e.g., to display more merchandise) in the zone during the certain times of the day (e.g., to generate more sales). In this way, the network device 105 conserves computing resources, networking resources, and/or other resources that would have otherwise been consumed by installing intrusive electronic monitoring devices, maintaining software associated with the electronic monitoring devices, and/or the like.
In some implementations, performing the one or more actions includes the network device 105 retraining the machine learning model based on the quantity and the start and stop times. For example, the network device 105 may utilize the quantity and the start and stop times as additional training data for retraining the machine learning model, thereby increasing the quantity of training data available for training the machine learning model. Accordingly, the network device 105 may conserve computing resources associated with identifying, obtaining, and/or generating historical data for training the machine learning model relative to other systems for identifying, obtaining, and/or generating historical data for training machine learning models.
Implementations described herein may utilize phase analysis between subcarrier frequencies to determine entry, exit, and walking motion in the zone. Hence, a variance of subcarrier frequencies may be determined by the phase difference in consecutive packets (via CSI). The variance of the subcarrier frequencies may be calculated as a passive process and a weighted average may be maintained, with a latest reading being assigned a maximum weight. The long-term variance calculation may create a gradual adaptive correction for changes in the zone while ensuring that the long-term variance calculation is not impacted by one-offs. An alternative method may include accounting for both amplitude and phase variance.
Implementations described herein may utilize a phase-weighted variance calculation, as follows:
New Subcarrier Variance =(Phase Difference between Consecutive Packets /Average Phase Variance for all subcarrier frequencies)*w1 +(Old Subcarrier Variance)*w2.
Implementations described herein may utilize an amplitude and phase-weighted variance calculation, as follows:
New Subcarrier Variance=(w3*Phase Difference between Consecutive Packets/Average Phase difference+w4*Amplitude difference between Consecutive Packets/Average Amplitude difference)*w1+(Old Subcarrier Variance)*w2.
In some implementations, an optimal first weight (w1) may be 0.1, an optimal second weight (w2) may be 0.9, an optimal third weight (w3) may be 0.7, and an optimal fourth weight (w4) may be 0.3, although other values are contemplated for the weights.
In this way, the network device 105 detects people and movement in a zone. For example, the network device 105 may collect CSI associated with the access points 110 communicating with the network device 105 via communication links in the zone. The network device 105 may utilize the CSI to calculate a quantity of people in the zone and/or movement of people in the zone. Thus, the network device 105 may conserve computing resources, networking resources, and/or other resources that would have otherwise been consumed by installing intrusive electronic monitoring devices, maintaining the electronic monitoring devices, maintaining software associated with the electronic monitoring devices, and/or the like.
As indicated above,
The network device 105 includes one or more devices capable of receiving, processing, storing, routing, and/or providing traffic (e.g., a packet and/or other information or metadata) in a manner described herein. For example, the network device 105 may include a router, such as a label switching router (LSR), a label edge router (LER), an ingress router, an egress router, a provider router (e.g., a provider edge router or a provider core router), a virtual router, or another type of router. Additionally, or alternatively, the network device 105 may include a gateway, a switch, a firewall, a hub, a bridge, a reverse proxy, a server (e.g., a proxy server, a cloud server, or a data center server), a load balancer, and/or a similar device. In some implementations, the network device 105 may be a physical device implemented within a housing, such as a chassis. In some implementations, the network device 105 may be a virtual device implemented by one or more computing devices of a cloud computing environment or a data center. In some implementations, a group of network devices 105 may be a group of data center nodes that are used to route traffic flow through a network.
The access point 110 includes one or more devices capable of receiving, generating, storing, processing, and/or providing information, as described elsewhere herein. The access point 110 may include a communication device and/or a computing device. For example, the access point 110 may include a wireless communication device, a wireless access point (WAP), an STB, a desktop computer, a smart speaker, a smart display device, a smart television, a motion detector, a camera, or a similar type of device.
The connected device 115 includes one or more devices capable of receiving, generating, storing, processing, and/or providing information, as described elsewhere herein. The connected device 115 may include a communication device and/or a computing device. For example, the connected device 115 may include a wireless communication device, an STB, a desktop computer, a smart speaker, a smart display device, a smart television, or a similar type of device.
The cloud computing system 202 includes computing hardware 203, a resource management component 204, a host operating system 205, and/or one or more virtual computing systems 206. The cloud computing system 202 may execute on, for example, an Amazon Web Services platform, a Microsoft Azure platform, or a Snowflake platform. The resource management component 204 may perform virtualization (e.g., abstraction) of the computing hardware 203 to create the one or more virtual computing systems 206. Using virtualization, the resource management component 204 enables a single computing device (e.g., a computer or a server) to operate like multiple computing devices, such as by creating multiple isolated virtual computing systems 206 from the computing hardware 203 of the single computing device. In this way, the computing hardware 203 can operate more efficiently, with lower power consumption, higher reliability, higher availability, higher utilization, greater flexibility, and lower cost than using separate computing devices.
The computing hardware 203 includes hardware and corresponding resources from one or more computing devices. For example, the computing hardware 203 may include hardware from a single computing device (e.g., a single server) or from multiple computing devices (e.g., multiple servers), such as multiple computing devices in one or more data centers. As shown, the computing hardware 203 may include one or more processors 207, one or more memories 208, one or more storage components 209, and/or one or more networking components 210. Examples of a processor, a memory, a storage component, and a networking component (e.g., a communication component) are described elsewhere herein.
The resource management component 204 includes a virtualization application (e.g., executing on hardware, such as the computing hardware 203) capable of virtualizing computing hardware 203 to start, stop, and/or manage one or more virtual computing systems 206. For example, the resource management component 204 may include a hypervisor (e.g., a bare-metal or Type 1 hypervisor, a hosted or Type 2 hypervisor, or another type of hypervisor) or a virtual machine monitor, such as when the virtual computing systems 206 are virtual machines 211. Additionally, or alternatively, the resource management component 204 may include a container manager, such as when the virtual computing systems 206 are containers 212. In some implementations, the resource management component 204 executes within and/or in coordination with a host operating system 205.
A virtual computing system 206 includes a virtual environment that enables cloud-based execution of operations and/or processes described herein using the computing hardware 203. As shown, the virtual computing system 206 may include a virtual machine 211, a container 212, or a hybrid environment 213 that includes a virtual machine and a container, among other examples. The virtual computing system 206 may execute one or more applications using a file system that includes binary files, software libraries, and/or other resources required to execute applications on a guest operating system (e.g., within the virtual computing system 206) or the host operating system 205.
Although the processing system 120 may include one or more elements 203-213 of the cloud computing system 202, may execute within the cloud computing system 202, and/or may be hosted within the cloud computing system 202, in some implementations, the processing system 120 may not be cloud-based (e.g., may be implemented outside of a cloud computing system) or may be partially cloud-based. For example, the processing system 120 may include one or more devices that are not part of the cloud computing system 202, such as the device 300 of
The network 220 includes one or more wired and/or wireless networks. For example, the network 220 may include a cellular network, a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a private network, the Internet, and/or a combination of these or other types of networks. The network 220 enables communication among the devices of the environment 200.
The number and arrangement of devices and networks shown in
The bus 310 includes one or more components that enable wired and/or wireless communication among the components of the device 300. The bus 310 may couple together two or more components of
The memory 330 includes volatile and/or nonvolatile memory. For example, the memory 330 may include random access memory (RAM), read only memory (ROM), a hard disk drive, and/or another type of memory (e.g., a flash memory, a magnetic memory, and/or an optical memory). The memory 330 may include internal memory (e.g., RAM, ROM, or a hard disk drive) and/or removable memory (e.g., removable via a universal serial bus connection). The memory 330 may be a non-transitory computer-readable medium. Memory 330 stores information, instructions, and/or software (e.g., one or more software applications) related to the operation of the device 300. In some implementations, the memory 330 includes one or more memories that are coupled to one or more processors (e.g., the processor 320), such as via the bus 310.
The input component 340 enables the device 300 to receive input, such as user input and/or sensed input. For example, the input component 340 may include a touch screen, a keyboard, a keypad, a mouse, a button, a microphone, a switch, a sensor, a global positioning system sensor, an accelerometer, a gyroscope, and/or an actuator. The output component 350 enables the device 300 to provide output, such as via a display, a speaker, and/or a light-emitting diode. The communication component 360 enables the device 300 to communicate with other devices via a wired connection and/or a wireless connection. For example, the communication component 360 may include a receiver, a transmitter, a transceiver, a modem, a network interface card, and/or an antenna.
The device 300 may perform one or more operations or processes described herein. For example, a non-transitory computer-readable medium (e.g., the memory 330) may store a set of instructions (e.g., one or more instructions or code) for execution by the processor 320. The processor 320 may execute the set of instructions to perform one or more operations or processes described herein. In some implementations, execution of the set of instructions, by one or more processors 320, causes the one or more processors 320 and/or the device 300 to perform one or more operations or processes described herein. In some implementations, hardwired circuitry may be used instead of or in combination with the instructions to perform one or more operations or processes described herein. Additionally, or alternatively, the processor 320 may be configured to perform one or more operations or processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.
The number and arrangement of components shown in
As shown in
As further shown in
As further shown in
As further shown in
As further shown in
In some implementations, performing the spectrogram analysis of the frequency versus time graph to determine the quantity of people in the zone and the start and stop times includes calculating an exponential moving average based on the frequency versus time graph; determining that people are entering or exiting the zone based on the exponential moving average satisfying a noise threshold; calculating the start and stop times and velocities of the people based on the exponential moving average; calculating motion energies of the people based on normalized fast Fourier transform coefficients; and determining the quantity of the people based on the velocities of the people and the motion energies.
In some implementations, the machine learning model includes one of a convolutional neural network model or a deep learning single shot detector model.
As further shown in
In some implementations, performing the one or more actions includes determining that an intruder has entered the zone, and contacting a law enforcement agency about the intruder. In some implementations, performing the one or more actions includes determining that the quantity satisfies a rental threshold quantity, and generating additional charges for rental of the zone based on determining that the quantity satisfies the rental threshold quantity.
In some implementations, process 400 includes calculating locations of the one or more access points in the zone based on the channel state information, and identifying the channel state information phases includes identifying the channel state information phases based on the locations of the one or more access points. In some implementations, calculating the locations of the one or more access points includes determining times of flight of the radio frequency transmissions based on the channel state information, determining angles of arrival of the radio frequency transmissions based on the channel state information, and calculating the locations of the one or more access points based on the times of flight and the angles of arrival.
In some implementations, process 400 includes determining whether a person is entering or exiting the zone over a time period based on phase differences included in the channel state information. In some implementations, determining whether the person is entering or exiting the zone over the time period includes determining, at a first time, a first phase associated with a first access point of the one or more access points; determining, at the first time, a second phase associated with a second access point of the one or more access points; calculating a first phase difference based on the first phase and the second phase; determining, at a second time, a third phase associated with the first access point; determining, at the second time, a fourth phase associated with the second access point; calculating a second phase difference based on the third phase and the fourth phase; and determining whether the person is entering or exiting the zone based on the first phase difference and the second phase difference.
In some implementations, process 400 includes training the machine learning model with a plurality of frequency versus time graphs associated with different types of zones, prior to processing the frequency versus time graph with the machine learning model.
Although
As used herein, the term “component” is intended to be broadly construed as hardware, firmware, or a combination of hardware and software. It will be apparent that systems and/or methods described herein may be implemented in different forms of hardware, firmware, and/or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods are described herein without reference to specific software code—it being understood that software and hardware can be used to implement the systems and/or methods based on the description herein.
As used herein, satisfying a threshold may, depending on the context, refer to a value being greater than the threshold, greater than or equal to the threshold, less than the threshold, less than or equal to the threshold, equal to the threshold, not equal to the threshold, or the like.
To the extent the aforementioned implementations collect, store, or employ personal information of individuals, it should be understood that such information shall be used in accordance with all applicable laws concerning protection of personal information. Additionally, the collection, storage, and use of such information can be subject to consent of the individual to such activity, for example, through well known “opt-in” or “opt-out” processes as can be appropriate for the situation and type of information. Storage and use of personal information can be in an appropriately secure manner reflective of the type of information, for example, through various encryption and anonymization techniques for particularly sensitive information.
Even though particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of various implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of various implementations includes each dependent claim in combination with every other claim in the claim set. As used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiple of the same item.
No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Further, as used herein, the article “the” is intended to include one or more items referenced in connection with the article “the” and may be used interchangeably with “the one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, or a combination of related and unrelated items), and may be used interchangeably with “one or more.” Where only one item is intended, the phrase “only one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise. Also, as used herein, the term “or” is intended to be inclusive when used in a series and may be used interchangeably with “and/or,” unless explicitly stated otherwise (e.g., if used in combination with “either” or “only one of”).
In the preceding specification, various example embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the broader scope of the invention as set forth in the claims that follow. The specification and drawings are accordingly to be regarded in an illustrative rather than restrictive sense.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20150301167 | Sentelle | Oct 2015 | A1 |
| 20160334498 | Jamieson | Nov 2016 | A1 |
| 20200302187 | Wang | Sep 2020 | A1 |
| 20210247426 | Liu | Aug 2021 | A1 |
| 20220026519 | Wu | Jan 2022 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20230245539 A1 | Aug 2023 | US |
