Patent Grant
10948581
The present disclosure relates to detecting presence of an object, and more particularly, to methods and apparatus for detecting presence of an object in an environment.
Detecting presence of an object in an environment can be applied to various applications, such as smart home devices and systems, and home security and surveillance. Motion sensors and proximity sensors can be used to detect the presence of the object. For example, passive infrared (PIR) sensors can be used to detect whether a human has moved in or out of a sensor's range. However, motion sensors and proximity sensors may not be able to provide accurate and/or prompt presence detection of a variety of objects, such as an inanimate object, a human, or an animal. In addition, motion sensors and proximity sensors may not be applicable to a variety of environments, such as a large room, an open space office, a public space, or an outdoor environment.
Embodiments of the present application provide improved methods and apparatus for detecting presence of an object in an environment.
These embodiments also include a method for detecting presence of an object in an environment. The method includes receiving a Doppler signal during a frame in frequency domain; separating the Doppler signal in the frequency domain into a plurality of sub-band signals; determining a plurality of sub-band signal energies corresponding to the plurality of sub-band signals; determining whether motion of the object is detected in accordance with one of the plurality of sub-band signal energies and a baseline energy; and responsive to a determination that motion of the object is detected, setting a flag of object presence.
These embodiments further include a method for detecting presence of an object in an environment. The method includes receiving a Doppler signal in time domain; determining whether motion of the object is detected in accordance with the Doppler signal in the time domain and a first baseline energy; responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the time domain and the first baseline energy, setting a first flag of object motion; transforming the Doppler signal in the time domain to a Doppler signal in frequency domain; determining whether motion of the object is detected in accordance with the Doppler signal in the frequency domain and a second baseline energy; responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the frequency domain and the second baseline energy, setting a second flag of object motion; and setting a flag of object presence based on the first flag of object motion and the second flag of object motion.
These embodiments further include an apparatus for detecting presence of an object in an environment. The apparatus includes a memory storing instructions and a processor configured to execute the instructions to cause the apparatus to: receive a Doppler signal during a frame in frequency domain; separate the Doppler signal in the frequency domain into a plurality of sub-band signals; determine a plurality of sub-band signal energies corresponding to the plurality of sub-band signals; determine whether motion of the object is detected in accordance with one of the plurality of sub-band signal energies and a baseline energy; and responsive to a determination that motion of the object is detected, set a flag of object presence.
These embodiments also include an apparatus for detecting presence of an object in an environment. The apparatus includes a memory storing instructions and a processor configured to execute the instructions to cause the apparatus to: receive a Doppler signal during a frame in time domain; determine whether motion of the object is detected in accordance with the Doppler signal in the time domain and a first baseline energy; responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the time domain and the first baseline energy, set a first flag of object motion; transform the Doppler signal in the time domain to a Doppler signal in frequency domain; determine whether motion of the object is detected in accordance with the Doppler signal in the frequency domain and a second baseline energy; responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the frequency domain and the second baseline energy, set a second flag of object motion; and set a flag of object presence based on the first flag of object motion and the second flag of object motion.
These embodiments further include a non-transitory processor-readable medium storing instructions that are executable by one or more processors of an apparatus to perform a method for detecting presence of an object in an environment. The method includes receiving a Doppler signal in frequency domain; separating the Doppler signal in the frequency domain into a plurality of sub-band signals; determining a plurality of sub-band signal energies corresponding to the plurality of sub-band signals; determining whether motion of the object is detected in accordance with one of the plurality of sub-band signal energies and a baseline energy; and responsive to a determination that motion of the object is detected, setting a flag of object presence.
These embodiments also include a non-transitory processor-readable medium storing instructions that are executable by one or more processors of an apparatus to perform a method for detecting presence of an object in an environment. The method includes receiving a Doppler signal during a frame in time domain; determining whether motion of the object is detected in accordance with the Doppler signal in the time domain and a first baseline energy; responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the time domain and the first baseline energy, setting a first flag of object motion; transforming the Doppler signal in the time domain to a Doppler signal in frequency domain; determining whether motion of the object is detected in accordance with the Doppler signal in the frequency domain and a second baseline energy; responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the frequency domain and the second baseline energy, setting the second flag of object motion; and setting a flag of object presence based on the first flag of object motion and a second flag of object motion.
It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the invention, as claimed.
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise represented. The implementations set forth in the following description of exemplary embodiments do not represent all implementations consistent with the invention. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the invention as recited in the appended claims.
External motion of person 110 may cause waves or signals reflected from moving person 110 to experience a frequency shift that is related to a radial component of velocity of person 110. External motion of person 110 is defined as any change in a pose or position of person 110's body center of mass (BCOM). This includes all external motion due to walking. For example, when person 110 walks in the environment in
Alternatively, object presence detector 100 can detect presence of an inanimate object or an animal, such as a moving vehicle or a cat walking in a room, by receiving signals, reflected from the vehicle or the cat, that experience a Doppler frequency shift because of the vehicle moving or cat walking. The received signals from the moving vehicle or the walking cat may be different from that from person 110. Object presence detector 100 can adjust to different sampling rates to detect presence of different objects.
System 200 may be implemented in object presence detector 100 or apparatus for detecting object presence. For example, as shown in
In some embodiments, system 200 may be implemented by one or more application-specific integrated circuits (ASICs). For example, all or a part of band-pass filter 220, environmental noise estimator 240, baseline-noise database 250, and motion detector 260 can be implemented by one or more integrated circuits, registers, and/or a memory circuit. Other part of system 200 can be implemented by programs executed by one or more processors including, e.g., processor 160, interacting with the one or more ASICs.
As an exemplary scenario for system 200, when person 110 walks in an environment, system 200 receives a Doppler signal, xk(t), reflected from person 110, where k is an index of frames. The Doppler signal is defined as a signal that experiences a Doppler frequency shift due to movement of person 110. The received Doppler signal xk(t) can be analog or digital. When the received Doppler signal xk(t) is analog, system 200 can sample the received Doppler signal at an appropriate sampling rate fs before filtering the Doppler signal xk(t) through band-pass filter 220.
System 200 is configured to check each frame to detect for the presence of an object. Each frame includes, for example, 200 samples of the received Doppler signal. The sampling rate can be at least, for example, a double rate of the Doppler signal's frequency. For example, when system 200 is used to detect a Doppler signal at 200 Hz, 250 Hz, or 300 Hz. the sampling rate, fs, is at least 400, 500, or 600 samples per second, respectively. System 200 may adjust the sampling rate to detect Doppler signals at different frequencies in accordance with a target to be detected.
As shown in
When the received Doppler signal xk(t) is analog, band-pass filter 220 includes an analog filter configured to filter the received Doppler signal in a frequency band as illustrated above and system 200 can sample a filtered Doppler signal to obtain the Doppler signal xk(n).
Environmental noise estimator 240 is configured to estimate environmental noise in the time domain in the environment. Before system 200 detects motion of an object, system 200 is configured to obtain an environmental noise in the environment. For example, environmental noise estimator 240 is configured to accumulate signal energies of xk(n) over current and previous twenty frames and average out an environmental noise energy. The environmental noise energy is defined as a baseline energy Nk in the environment and is used to detect the presence of an object at the k+1th frame.
When system 200 detects the presence of an object in a frame, system 200 is configured to maintain a baseline energy at a value equal to that in a previous frame. For example, Nk=Nk−1 if person 110 is detected. On the other hand, when system 200 does not detect the presence of an object in a frame, system 200 is configured to update the baseline energy in accordance with a received signal energy (that is considered an environmental noise energy) in this frame and the baseline energy. For example, environmental noise estimator 240 is configured to update the baseline energy by a weighted moving average: Nk=Σi=0L−2wiNk−i−1+wL−1E{xk2(n)}, where Σi=0L−1wi=1, wi is a weight of environmental noise energy of the ith frame in calculation of the updated baseline energy, and E{xk(n)}=Σn=0Ns−1xk2(n). Accordingly, system 200 is configured to detect the presence of an object in the k+1th frame in accordance with the updated baseline energy Nk.
In some embodiments, when system 200 does not detect the presence of an object in a frame, system 200 is configured to maintain a baseline energy at a value equal to that in a previous frame. In other words, system 200 is configured not to update the baseline energy.
After environmental noise estimator 240 estimates a baseline energy, system 200 is configured to store the estimated baseline energy in baseline-noise database 250. For example, after environmental noise estimator 240 accumulates signal energies of xk(n) over twenty frames and averages out a baseline energy Nk, system 200 is configured to store the baseline energy Nk in baseline-noise database 250. As another example, when system 200 does not detect the presence of an object in a frame, environmental noise estimator 240 updates the baseline energy as described above and system 200 is configured to store the updated baseline energy in baseline-noise database 250.
Motion detector 260 is configured to detect motion of an object in a frame. For example, as shown in
MFkTD=1 if sk>βNk−1, and
MFkTD=0 if sk<βNk−1,
When the Doppler signal energy sk is greater than the threshold βNk−1, motion detector 260 determines that motion of the object is detected and sets the flag of object motion, e.g., MFkTD=1. When the Doppler signal energy sk is less than the threshold βNk−1, motion detector 260 determines that motion of the object is not detected and does not set the flag of object motion, e.g., MFkTD=0. Environmental noise estimator 240 updates the baseline energy by the above-described weighted moving average: Nk=Σi=0L−2wiNk−i−1+wL−1E{xk2(n)}.
After motion detector 260 determines whether motion of the object is detected and accordingly, sets or does not set the flag of object motion MFkTD, motion detector 260 is also configured to set a flag of object presence, OFkTD in accordance with the flag of object motion MFkTD. For example, motion detector 260 sets: OFkTD=MFkTD.
In some embodiments, when system 200 does not detect motion of an object in accordance with the Doppler signal energy and the baseline energy in the time domain, system 200 is also configured to determine whether motion of the object is detected in accordance with a Doppler signal and a baseline energy in the frequency domain. When system 200 also determines that motion of the object is not detected in accordance with the Doppler signal and the baseline energy in the frequency domain, system 200 determines that motion of the object is not detected in accordance with the Doppler signal and updates the baseline energy in accordance with the signal energy in the time domain. For example, environmental noise estimator 240 updates the baseline energy by the above-described weighted moving average: Nk=Σi=0L−2wiNk−i−1+wL−1E{xk2(n)}.
Similar to implementation alternatives of system 200, system 300 may be implemented in object presence detector 100 or apparatus for detecting object presence, as illustrated above for system 200. System 300 can also be partly or fully implemented by one or more ASICs and/or programs, as illustrated and described above for system 200.
As an exemplary scenario for system 300, when person 110 walks in an environment, system 300 receives a Doppler signal xk(t) in the time domain, where k is an index of frames. System 300 then band-pass filters the Doppler signal in a frequency band, for example, from 0 to 250 Hz and obtains a Doppler signal xk(n), where k is an index of frames, n=0, 1, 2, . . . , Ns−1, and Ns is a number of samples in a frame.
As shown in
Sub-band decomposer 320 is configured to separate the Doppler signal in the frequency domain into a plurality of sub-band signals. For example, as shown in
Environmental noise estimator 340 is configured to estimate environmental noise in the frequency domain in the environment. Before system 300 detects motion of an object, system 300 is configured to obtain an environmental noise in the environment. For example, environmental noise estimator 340 is configured to accumulate signal energies of Xk(f) over current and previous twenty frames and average out an environmental noise energy in the frequency domain. The environmental noise energy in the kth frame in the frequency domain is defined as a baseline energy NkFD in the frequency domain in the environment and is used to detect the presence of an object in the k+1th frame.
When system 300 detects the presence of an object in a frame, system 300 is configured to maintain a baseline energy at a value equal to that in a previous frame. For example, NkFD=Nk−1FD if person 110 is detected. On the other hand, when system 300 does not detect the presence of an object in a frame, system 300 is configured to update a baseline energy in accordance with a received signal energy (that is considered as an environmental noise energy) in this frame and the baseline energy. For example, environmental noise estimator 340 is configured to update the baseline energy by a weighted moving average: NkFD=Σi=0L−2wiNk−i−1FD+wL−1E{Xk2(f)}, where Σi=0L−1wi=1, wi is a weight of environmental noise energy of the ith frame in calculation of an updated baseline energy, and E{Xk2(f)}=Σf=0Ns−1Xk2(f). Accordingly, system 300 is configured to detect presence of an object in the k+1th frame in accordance with the updated baseline energy NkFD.
In some embodiments, when system 300 does not detect the presence of an object in a frame, system 300 is configured to maintain a baseline energy at a value equal to that in a previous frame. In other words, system 300 is configured not to update the baseline energy.
After environmental noise estimator 340 estimates a baseline energy, system 300 is configured to store the estimated baseline energy in baseline-noise database 350. For example, after environmental noise estimator 340 accumulates signal energies of Xk(f) over twenty frames and averages out a baseline energy NkFD in the frequency domain, system 300 is configured to store the baseline energy NkFD in baseline-noise database 350. As another example, when system 300 does not detect the presence of an object in a frame, environmental noise estimator 340 updates the baseline energy as described above and system 300 is configured to store the updated baseline energy in baseline-noise database 350.
In some embodiments, environmental noise estimator 340 is configured to estimate environmental noises in respective sub-bands in the environment. Thus, sub-band decomposer 320 also sends sub-band signals Xk,m(f) to environmental noise estimator 340. Before system 300 detects motion of an object, system 300 is configured to obtain environmental sub-band baseline energies in the environment. For example, environmental noise estimator 340 is configured to accumulate sub-band signal energies of Xk,m(f) in five sub-bands over current and previous twenty frames and average out environmental sub-band noise energies. The environmental sub-band noise energies in the kth frame are defined as sub-band baseline energies Nk,mFD for the m sub-bands in the environment and are used to detect the presence of an object at the k+1th frame.
When system 300 detects presence of an object in a frame, system 300 is configured to maintain sub-band baseline energies at values equal to those in a previous frame. For example, Nk,mFD=Nk−1,mFD if person 110 is detected. On the other hand, when system 300 does not detect the presence of an object in a frame, system 300 is configured to update sub-band baseline energies in accordance with received sub-band signal energies (that are considered as environmental sub-band noise energies) in this frame and the sub-band baseline energies. For example, environmental noise estimator 340 is configured to update the sub-band baseline energies by a weighted moving average: Nk,mFD=Σi=0L−2wiNk−i−1,mFD+wL−1E{Xk,m2(f)}, where Σi=0L−1wi=1, wi is a weight of environmental noise energy of the ith frame in calculation of updated sub-band baseline energies, and E{Xk,m2(f)}=Σf=0Ns−1Xk,m2(f). Accordingly, system 300 is configured to detect the presence of an object in the k+1th frame in accordance with the updated sub-band baseline energies Nk,mFD.
In some embodiments, when system 300 does not detect the presence of an object in a frame, system 300 is configured to maintain sub-band baseline energies at values equal to those in a previous frame. In other words, system 300 is configured not to update the sub-band baseline energies.
After environmental noise estimator 340 estimates a baseline energy, system 300 is configured to store the estimated sub-band baseline energies in baseline-noise database 350. For example, after environmental noise estimator 340 accumulates sub-band signal energies of Xk,m(f) over twenty frames and averages out sub-band baseline energy Nk,mFD, system 300 is configured to store the sub-band baseline energies Nk,mFD in baseline-noise database 350. As another example, when system 300 does not detect presence of an object in a frame, environmental noise estimator 340 updates the sub-band baseline energies as described above and system 300 is configured to store the updated sub-band baseline energies in baseline-noise database 350.
Each component of system 300 can read baseline energy and/or sub-band baseline energies from baseline-noise database 350 whenever the component of system 300 needs the baseline energy and/or sub-band baseline energies. For example, when sub-band motion detector 360 determines motion of an object in sub-bands, sub-band motion detector 360 reads sub-band baseline energies from baseline-noise database 350.
Sub-band motion detector 360 is configured to detect motion of an object in sub-bands in a frame in the frequency domain. For example, as shown in
Sub-band motion detector 360 then determines whether motion of the object is detected in accordance with one of sub-band signal energies and a baseline energy. For example, sub-band motion detector 360 determines whether motion of person 110 is detected in accordance with sub-band signal energies Sk,m and a baseline energy Nk−1FD in the kth frame as follows:
MFk,mFD=1 if Sk,m≥2βNk−1FD, and
MFk,mFD=0 if Sk,m<2βNk−1FD,
When a sub-band signal energy Sk,m is equal to or greater than the threshold 2βNk−1FD, sub-band motion detector 360 determines that motion of the object is detected and sets the flag of sub-band object motion, e.g., MFk,mFD=1. For example, when sub-band motion detector 360 determines that sub-band signal energy Sk,2 is greater than the threshold 2βNk−1FD, sub-band motion detector 360 determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,2FD=1. When sub-band signal energy Sk,m is less than the threshold 2βNk−1FD, sub-band motion detector 360 determines that motion of the object is not detected in the sub-band and does not set the flag of sub-band object motion, e.g., MFk,mFD=0, m=0, 1, 3, and 4.
As another example, sub-band motion detector 360 determines whether motion of person 110 is detected in accordance with sub-band signal energies Sk,m and a baseline energy Nk−1FD in the kth frame as follows:
MFk,mFD=1 if Sk,m≥2βNk−1FD, and
MFk,mFD=0 if Sk,m<2βNk−1FD,
When a sub-band signal energies Sk,m is equal to or greater than the threshold βNk−1FD, sub-band motion detector 360 determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,mFD=1. For example, when sub-band motion detector 360 determines that sub-band signal energy Sk,1 is greater than the threshold βNk−1FD, sub-band motion detector 360 determines that motion of the object in the sub-band is detected and sets the flag of sub-band object motion, e.g., MFk,1FD=1. When sub-band motion detector 360 determines that sub-band signal energy Sk,2 is greater than the threshold βNk−1FD, sub-band motion detector 360 also determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,2FD=1 When the remaining sub-band signal energies Sk,m are less than the threshold βNk−1FD, sub-band motion detector 360 determines that motion of the object is not detected in the remaining sub-bands and does not set the flags of sub-band object motion, e.g., MFk,mFD=0, m=0, m=0, 3, and 4.
In some embodiments, the baseline energy includes a plurality of sub-band baseline energies corresponding to the plurality of sub-band signals. For example, environmental noise estimator 340 estimates environmental noises in respective sub-bands in the environment and averages out sub-band baseline energies Nk,mFD, where m=0, 1, 2, 3, and 4. Sub-band motion detector 360 is configured to detect motion of an object in sub-bands in a frame in accordance with respective sub-band baseline energies. For example, sub-band motion detector 360 determines whether motion of person 110 is detected in accordance with sub-band signal energies Sk,m and a sub-band baseline energy Nk−1,mFD in the kth frame as follows:
MFk,mFD=1 if Sk,m≥2βNk−1FD, and
MFk,mFD=0 if Sk,m<2βNk−1FD,
When a sub-band signal energy Sk,m is equal to or greater than the threshold 2βNk−1,mFD, sub-band motion detector 360 determines that motion of the object is detected and sets the flag of sub-band object motion, e.g., MFk,mFD==1. For example, when sub-band motion detector 360 determines that sub-band signal energy Sk,2 is greater than the threshold 2βNk−1,2FD, sub-band motion detector 360 determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,2FD=1. When a sub-band signal energy Sk,m is less than the threshold 2βNk−1,mFD, sub-band motion detector 360 determines that motion of the object is not detected in the sub-band and does not set the flag of sub-band object motion, e.g., MFk,mFD=0, m=0, 1, 3, and 4.
As another example, sub-band motion detector 360 determines whether motion of person 110 is detected in accordance with sub-band signal energies Sk,m and a baseline energy Nk−1,mFD in the kth frame as follows:
MFk,mFD=1 if Sk,m≥βNk−1,mFD, and
MFk,mFD=0 if Sk,m<βNk−1,mFD,
When a sub-band signal energy Sk,m is equal to or greater than the threshold βNk−1,mFD, sub-band motion detector 360 determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,mFD==1. For example, when sub-band motion detector 360 determines that sub-band signal energy Sk,1 is greater than the threshold βNk−1,1FD, sub-band motion detector 360 determines that motion of the object in the sub-band is detected and sets the flag of sub-band object motion, e.g., MFk,1FD==1. When sub-band motion detector 360 determines that sub-band signal energy Sk,2 is greater than the threshold βNk−1,2FD, sub-band motion detector 360 also determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,2FD=1. When the remaining sub-band signal energies Sk,m are less than the threshold βNk−1,mFD, sub-band motion detector 360 determines that motion of the object is not detected in the remaining sub-bands and does not set the flags of sub-band object motion, e.g., MFk,mFD=0, m=0, 3, and 4.
After sub-band motion detector 360 detects motion of an object in sub-sub-bands in a frame, sub-band motion detector 360 sends all flags of sub-band object motion to object motion detector 380. For example, as shown in
Object motion detector 380 is configured to determine whether motion of an object is detected in accordance with the flags of sub-band object motion from sub-band motion detector 360. For example, when sub-band motion detector 360 determines whether motion of the object is detected in sub-bands in accordance with the threshold 2βNk−1FD or 2βNk−1,mFD and criteria thereof, object motion detector 380 determines that motion of the object is detected if one of five flags of sub-band objection motion is set, e.g., flag of object motion MFkFD=1 if MFk,mFD=1, where m=0, 1, 2, 3, or 4. For example, when MFk,2FD=1, object motion detector 380 determines that motion of the object is detected and sets MFkFD=1. When none of five flags of sub-band objection motion is set, object motion detector 380 determines that motion of the object is not detected and does not set the flag of object motion MFkFD.
As another example, when sub-band motion detector 360 determines object motion in sub-bands in accordance with the threshold βNk−1FD or βNk−1,mFD and criteria thereof, object motion detector 380 determines that motion of the object is detected if at least two of five flags of sub-band objection motion are set, e.g., MFkFD=1 if MFk,iFD=1 and MFk,jFD=1, where i=0, 1, 2, 3, or 4, j=0, 1, 2, 3, or 4, and i≠j. For example, when MFk,1FD=1 and MFk,2FD=1, object motion detector 380 determines that motion of the object is detected and sets MFkFD=1. Alternatively, object motion detector 380 can determine that motion of the object is detected if two of five flags of sub-band objection motion in two adjacent sub-bands are set, e.g., MFkFD=1 if MFk,iFD=1 and MFk,i+1FD=1, where i=0, 1,2, or 3. When only one or none of five flags of sub-band objection motion is set, object motion detector 380 determines that motion of the object is not detected and does not set MFkFD.
In some embodiments, sub-band motion detector 380 determines whether motion of the object is detected in accordance with both thresholds 2βNk−1FD and βNk−1FD or 2βNk−1,mFD and βNk−1,mFD and criteria thereof as illustrated above in sub-band motion detector 360. For example, object motion detector 380 and sub-band motion detector 360 can jointly determine whether motion of person 110 is detected in accordance with sub-band signal energies Sk,m and a sub-band baseline energy Nk−1,mFD in the kth frame as follows:
MFkFD=1 if Sk,m≥2βNk−1,mFD, m=0,1,2,3, or 4,
MFkFD=1 if Sk,i>βNk−1,iFD and Sk,j>βNk−1,jFD, i=0,1,2,3, or 4, j=0,1,2,3, or 4, i≠j, and
MFkFD=0 otherwise,
When a sub-band signal energy Sk,m is equal to or greater than the threshold 2βNk−1,mFD object motion detector 380 and sub-band motion detector 360 can jointly determine that motion of the object is detected and sets the flag of object presence, e.g., MFkFD=1. When none of sub-band signal energies Sk,m is greater than the threshold 2βNk−1,mFD, and at least two sub-band signal energies Sk,i and Sk,j are greater than the thresholds βNk−1,iFD and βNk−1,jFD, respectively, object motion detector 380 and sub-band motion detector 360 can jointly determine that motion of the object is detected and set the flag of object motion, e.g., MFkFD=1. Otherwise, object motion detector 380 and sub-band motion detector 360 can jointly determine that motion of the object is not detected and not set the flag of object motion, e.g., MFkFD=0.
When object motion detector 380 detects motion of the object, environmental noise estimator 340 does not update environmental sub-band baseline energy in accordance with sub-band signal energies in the frame. When object motion detector 380 does not detect motion of the object, environmental noise estimator 340 can update environmental sub-band baseline energy in accordance with sub-band signal energies in the frame. For example, environmental noise estimator 340 can update a sub-band baseline energy by a weighted moving average: Nk,mFD=Σi=0L−2wiNk−1,mFD+wL−1E{Xk,m2(f)}, where Σi=0L−1wi=1. In some embodiments, when object motion detector 380 does not detect motion of the object, environmental noise estimator 340 does not update the baseline energy in accordance with sub-band signal energies in the frame.
After object motion detector 380 determines whether motion of the object is detected and accordingly, sets or does not set the flag of object motion MFkFD, object motion detector 380 is also configured to set a flag of object presence OFkFD in accordance with the flag of object motion MFkFD. For example, object motion detector 380 sets: OFkFD=MFkFD.
Similar to implementation alternatives of system 200, system 400 may be implemented in object presence detector 100 or apparatus for detecting object presence, as illustrated above for system 200. System 400 can also be partly or fully implemented by one or more ASICs and/or programs, as illustrated above for system 200.
As an exemplary scenario for system 400, when person 110 walks in an environment, system 400 receives a Doppler signal xk(t) in the time domain, where k is an index of frames. System 400 then band-pass filters the Doppler signal in a frequency band, for example, from 0 to 250 Hz and obtains a Doppler signal, xk(n), where k is an index of frames, n=0, 1, 2, . . . , Ns−1, and Ns is a number of samples in a frame.
Time-domain object motion detector 420 is configured to detect motion of an object in the time domain. For example, time-domain object motion detector 420 receives the Doppler signal xk(n) in the time domain, determines whether motion of the object is detected in accordance with the Doppler signal xk(n) in the time domain and a baseline energy Nk−1, and sets a time-domain flag of object motion MFkTD=1 in response to a determination that motion of the object is detected. Time-domain object motion detector 420 can be configured to perform operations of system 200 described above with respect to the methods described above and shown in
Frequency-domain object motion detector 430 is configured to detect motion of the object in the frequency domain. For example, frequency-domain object motion detector 430 is configured to transform the Doppler signal, xk(n), in the time domain to a Doppler signal Xk(f) in the frequency domain, determine whether motion of the object is detected in accordance with the Doppler signal Xk(f) in the frequency domain and a baseline energy Nk−1FD or a sub-band baseline energy Nk−1,mFD, and set a frequency-domain flag of object motion MFkFD=1 in response to a determination that motion of the object is detected. Frequency-domain object motion detector 430 can be configured to perform operations of system 300 described above with respect to the methods shown in
Object presence detector 440 is configured to detect presence of the object in accordance with flags of object motion in the time and frequency domains. For example, as show in
In some embodiments, frequency-domain object motion detector 430 is configured to detect motion of the object in the frequency domain when a criterion is met. For example, frequency-domain object motion detector 430 transforms the Doppler signal xk(n) in the time domain to a Doppler signal Xk(f) in the frequency domain and proceeds with frequency-domain object motion detection when one of the following criteria is met:
In accordance with the criterion (a), frequency-domain object motion detector 430 transforms the Doppler signal xk(n) in the time domain to a Doppler signal Xk(f) in frequency domain and proceeds with frequency-domain object motion detection when time-domain object motion detector 420 does not detect motion of the object in the time domain.
In accordance with the criterion (b), frequency-domain object motion detector 430 transforms the Doppler signal xk(n) in the time domain to a Doppler signal Xk(f) in the frequency domain and proceeds with frequency-domain object motion detection when time-domain object motion detector 420 detects motion of the object in the time domain. Thus, frequency-domain object motion detector 430 detects motion of the object only when time-domain object motion detector 420 detects motion of the object. Thus, time-domain object motion detector 420 and frequency-domain object motion detector 430 both detect motion of the object and send time-domain and frequency-domain flags of object motion to object presence detector 440.
In accordance with the criterion (c), frequency-domain object motion detector 430 transforms the Doppler signal xk(n) in the time domain to a Doppler signal Xk(f) in the frequency domain and proceeds with frequency-domain object motion detection when time-domain object motion detector 420 detects motion of the object in the time domain, but the signal energy sk of the Doppler signal in the time domain is less than a threshold, e.g., sk<1.2βNk−1. Thus, frequency-domain object motion detector 430 detects motion of the object when time-domain object motion detector 420 detects motion of the object, but the signal energy of the Doppler signal is not strong enough.
Step 510 includes receiving a Doppler signal during a frame. For example, as shown in
Step 520 includes filtering the Doppler signal in a frequency band. For example, as illustrated system 200 in
Step 530 includes determining a signal energy of the filtered Doppler signal in the time domain. For example, as shown in
Step 540 includes determining whether motion of the object is detected in accordance with the Doppler signal and a baseline energy. For example, as illustrated system 200 in
MFkTD=1 if sk≥βNk−1, and
MFkTD=0 if sk<βNk−1,
When the Doppler signal energy sk is equal to or greater than the threshold βNk−1, motion detector 260 determines that motion of the object is detected and sets the flag of object motion, e.g., MFkTD=1. When the Doppler signal energy sk is less than the threshold βNk−1, motion detector 260 determines that motion of the object is not detected and does not set the flag of object motion, e.g., MFkTD=0. The threshold βNk−1 is obtained in accordance with a baseline energy Nk−1 (i.e., the environmental noise energy) in the environment and β=c×SNRthr, where c is a constant and SNRthr is a signal-to-noise ratio threshold.
Step 550 includes, responsive to a determination that motion of the object is detected in accordance with the signal energy and the baseline energy, setting a flag of object presence. For example, as illustrated in system 200 in
Step 560 includes, responsive to a determination that motion of the object is not detected in accordance with the Doppler signal, updating the baseline energy in accordance with the signal energy. For example, as illustrated in system 200 in
In some embodiments, step 540 further includes determining that motion of the object is not detected in accordance with the Doppler signal in the time domain and the Doppler signal in the frequency domain. For example, when system 200 does not detect motion of an object in accordance with the Doppler signal energy and the baseline energy in the time domain, system 200 can determine whether motion of the object is detected in accordance with a Doppler signal and a baseline energy in the frequency domain, as illustrated by methods in
In some embodiments, the threshold in step 540 is determined based on a baseline energy and another threshold. For example, the threshold βNk−1 is obtained in accordance with a baseline energy Nk−1 (i.e., the environmental noise energy) in the environment and a β=c×SNRthr, where c is a constant and SNRthr is a signal-to-noise ratio threshold. Thus, the threshold in step 540 (e.g., βk−1) is determined based on a baseline energy (e.g., a baseline energy Nk−1) and another threshold (e.g., SNRthr).
In some embodiments, step 560 can include, responsive to a determination that motion of the object is not detected in accordance with the Doppler signal, updating the baseline energy in accordance with a weighted average of the signal energy and the baseline energy. For example, as illustrated by system 200 in
In some embodiments, step 560 can include, responsive to a determination that motion of the object is not detected in accordance with the Doppler signal, updating the baseline energy in accordance with the signal energy in the frame and another signal energy in a previous frame. For example, as illustrated by system 200 in
The k−i−1th frame is a previous frame for the kth frame. The baseline energy in the k−i−1th frame, Nk−i−1, may include signal energy in a previous frame. For example, when motion detector 260 determines that motion of the object is not detected in the k−1th frame and does not set the flag of object motion, e.g., MFk−1TD=0, environmental noise estimator 240 updates the baseline energy Nk−1 by a similar weighted moving average: Nk−1=Σi=0L−2wiNk−i−2+wL−1E{xk−12(n)}, where E{xk−12(n)} is the signal energy in the k−1th frame (that is considered as an environmental noise energy), which is a previous frame for the kth frame. In such an update, the baseline energy Nk is determined based on the weighted moving average of the signal energies at least in the k−1th and kth frames when motion of the object is not present in the frames.
In some embodiments, the baseline energy in step 540 is a first baseline energy and method 500 further includes, responsive to a determination that motion of the object is not detected, determining whether motion of the object is detected in accordance with the signal energy and a second baseline energy, and responsive to a determination that motion of the object is detected in accordance with the signal energy and the second baseline energy, setting the flag of object presence.
For example, as illustrated by system 200 in
MFkTD=1 if sk≥βNk−1st, and
MFkTD=0 if sk<βNk−1st, and
When the Doppler signal energy sk is equal to or greater than the threshold αk−1st, motion detector 260 determines that motion of the object is detected and sets the flag of object motion, e.g., MFkTD=1. When the Doppler signal energy sk is less than the threshold αNk−1st, motion detector 260 determines that motion of the object is not detected and does not set the flag of object motion, e.g., MFkTD=0. The threshold αNk−1st is obtained in accordance with a baseline energy Nk−1st (i.e., an environmental noise energy) in the environment and α=cst×SNRthrst, where cst is a constant and SNRthrst is a signal-to-noise ratio threshold.
In some embodiments, when the Doppler signal energy Sk is less than the threshold βNk−1 but equal to or larger than 0.7 βNk−1, method 500 further includes operations described above in accordance with the second baseline energy Nk−1st.
In some embodiments, step 560 further includes, responsive to a determination that motion of the object is not detected in accordance with the signal energy and the second baseline energy, updating the second baseline energy in accordance with the signal energy. For example, when the Doppler signal energy sk is less than the threshold αNk−1st and motion detector 260 determines that motion of the object is not detected, motion detector 260 does not set the flag of object motion, e.g., MFkTD=0. Environmental noise estimator 240 updates the second baseline energy Nk−1st by the weighted moving average: Nkst=Σi=0L−2wiNk−i−1st+wL−1E{xk2(n)}, where E{xk2(n)} is the signal energy in the kth frame (that is considered as an environmental noise energy) and Nk−i−1st is the second baseline energy of environmental noise in the k−i−1th frame.
In some embodiments, the second baseline energy is determined based on a weighted moving average of a plurality of signal energies when motion of the object is not present, wherein the plurality of signal energies are energies of a plurality of Doppler signals received over a longer period than that of the first baseline energy. For example, when motion detector 260 determines that motion of the object is not detected and does not set the flag of object motion, e.g., MFkTD=0, environmental noise estimator 240 updates the second baseline energy Nk−1st by the weighted moving average: Nkst=Σi=0L−2wiNk−i−1st+wL−1E{xk2(n)}, where E{xk2(n)} is the signal energy in the kth frame (that is considered as an environmental noise energy) and Nk−i−1st is the second baseline energy of environmental noise in the k−i−1th frame.
The k−i−1th frame is a previous frame for the kth frame. The second baseline energy Nk−i−1st of environmental noise in the k−i−1th frame, may include signal energy in a previous frame. For example, when motion detector 260 determines that motion of the object is not detected in the k−1th frame and does not set the flag of object motion, e.g., MFk−1TD=0, environmental noise estimator 240 updates the second baseline energy Nk−1st by a similar weighted moving average: Nk−1st=Σi=0L−2wiNk−i−2st+wL−1E{xk−12(n)}, where E{k−12(n)} is the signal energy in the k−1th frame (that is considered as an environmental noise energy), which is a previous frame for the kth frame. In such an update, the second baseline energy Nkst is determined based on the weighted moving average of the signal energies at least in the k−1th and kth frames when motion of the object are not present in the frames.
In some embodiments, the second baseline energy Nkst is determined based on a plurality of Doppler signals received over a longer period than that of the first baseline energy Nk. For example, the second baseline energy Nkst can be determined based on a plurality of Doppler signals received over forty frames (those are considered as environmental noise energies), and the first baseline energy Nk is averaged out over twenty frames.
Alternatively, the second baseline energy Nkst can be a stationary baseline energy that is average out over all of those frames in which motion detector 260 does not detect motion of person 110. The first baseline energy Nk is a non-stationary baseline energy that is averaged out over the latest twenty of those frames in which motion detector 260 does not detect motion of person 110.
In some embodiments, method 500 is configured to detect presence of an inanimate object, a human, or an animal. Accordingly, step 510 includes receiving a Doppler signal from motion of an inanimate object, a human, or an animal during a frame, such as a moving vehicle, working person 110, or a cat walking in a room.
Step 610 includes receiving a Doppler signal during a frame in the frequency domain. For example, as shown in
Step 620 includes separating the Doppler signal in the frequency domain into a plurality of sub-band signals. For example, as system 300 illustrated in
Step 630 includes determining a plurality of sub-band signal energies corresponding to the plurality of sub-band signals. For example, as system 300 illustrated in
Step 640 includes determining whether motion of the object is detected in accordance with one of the plurality of sub-band signal energies and a baseline energy. For example, as system 300 illustrated in
MFk,mFD=1 if Sk,m≥2βNk−1FD, and
MFk,mFD=0 if Sk,m<2βNk−1FD,
When a sub-band signal energies Sk,m is equal to or greater than the threshold 2βNk−1FD, sub-band motion detector 360 determines that motion of the object is detected and sets the flag of sub-band object motion, e.g., MFk,mFD=1. For example, when sub-band motion detector 360 determines that sub-band signal energy Sk,2 is greater than the threshold 2βNk−1FD, sub-band motion detector 360 determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,2FD=1.
Step 650 includes, responsive to a determination that motion of the object is detected, setting a flag of object presence. For example, as system 300 illustrated in
After object motion detector 380 determines whether motion of the object is detected and accordingly, sets or does not set the flag of object motion MFkFD, object motion detector 380 sets a flag of object presence OFkFD in accordance with the flag of object motion MFkFD. For example, object motion detector 380 sets: OFkFD=MFkFD.
In some embodiments, step 640 includes determining that motion of the object is detected when at least two of the plurality of sub-band signal energies are equal to or greater than a threshold, and determining that motion of the object is not detected when less than two of the plurality of sub-band signal energies is equal to or greater than the threshold, wherein the threshold is determined based on the baseline energy.
For example, as system 300 illustrated in
MFk,mFD=1 if Sk,m≥βNk−1FD, and
MFk,mFD=0 if Sk,m<βNk−1FD,
When a sub-band signal energy Sk,m is equal to or greater than the threshold βNk−1FD, sub-band motion detector 360 determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,mFD=1. For example, when sub-band motion detector 360 determines that sub-band signal energy Sk,1 is greater than the threshold βNk−1FD, sub-band motion detector 360 determines that motion of the object in the sub-band is detected and sets the flag of sub-band object motion, e.g., MFk,1FD=1. When sub-band motion detector 360 determines that sub-band signal energy Sk,2 is greater than the threshold βNk−1FD, sub-band motion detector 360 also determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,2FD=1. When the remaining sub-band signal energies Sk,m are less than the threshold βNk−1FD, sub-band motion detector 360 determines that motion of the object is not detected in the remaining sub-bands and does not set the flags of sub-band object motion, e.g., MFk,mFD=0, m=0, 3, and 4.
Object motion detector 380 determines that motion of the object is detected if at least two of five flags of sub-band objection motion are set, e.g., MFkFD=1 if MFk,iFD=1 and MFk,jFD=1, where i=0, 1, 2, 3, or 4, j=0, 1, 2, 3, or 4, and i≠j. For example, when MFk,1FD1 and MFk,2FD2=1, object motion detector 380 determines that motion of the object is detected and sets MFkFD=1. Alternatively, object motion detector 380 can determine that motion of the object is detected if two of five flags of sub-band objection motion in two adjacent sub-bands are set, e.g., MFkFD=1 if MFk,iFD=1 and MFk,i+1FD=1, where i=0, 1, 2, or 3. When only one or none of five flags of sub-band objection motion is set, object motion detector 380 determines that motion of the object is not detected and does not set MFkFD.
In some embodiments, step 610 includes receiving a Doppler signal in the time domain and transforming the Doppler signal in the time domain to the Doppler signal in the frequency domain using a short-time Fourier transform (STFT) or a filter bank. For example, as system 300 illustrated in
In some embodiments, the baseline energy of environmental noise includes a plurality of sub-band baseline energies corresponding to the plurality of sub-band signals, and step 640 includes determining whether motion of the object is detected in accordance with the plurality of sub-band signal energies and the plurality of sub-band baseline energies.
For example, as illustrated by system 300 in
MFk,mFD=1 if Sk,m≥2Nk−1,mFD, and
MFk,mFD=0 if Sk,m<2βNk−1,mFD,
When a sub-band signal energy Sk,m is equal to or greater than the threshold 2βNk−1,mFD, sub-band motion detector 360 determines that motion of the object is detected and sets the flag of sub-band object motion, e.g., MFk,mFD=1. For example, when sub-band motion detector 360 determines that sub-band signal energy Sk,2 is greater than the threshold 2βNk−1,2FD, sub-band motion detector 360 determines that motion of the object is detected in the sub-band and sets the flag of sub-band object motion, e.g., MFk,2FD=1. When a sub-band signal energy Sk,m is less than the threshold 2βNk−1,mFD, sub-band motion detector 360 determines that motion of the object is not detected in the sub-band and does not set the flag of sub-band object motion, e.g., MFk,mFD=0, m=0, 1, 3, and 4.
In some embodiments, responsive to a determination that motion of the object is not detected, method 600 further includes updating the baseline energy in accordance with the one of the plurality of sub-band signal energies. For example, when system 300 does not detect presence of an object in a frame, system 300 is configured to update sub-band baseline energies in accordance with received sub-band signal energies (that are considered as environmental sub-band noise energies) in this frame and the sub-band baseline energies. For example, environmental noise estimator 340 is configured to update the sub-band baseline energies by the weighted moving average: Nk,m=Σi=0L−2wiNk−i−1,mFD+wL−1E{Xk,m2(f)}, where Σi=0L−1wi=1, wi is a weight of environmental noise energy of the ith frame in calculation of updated sub-band baseline energies, and E{Xk,m2f)} are sub-band signal energies for m=0, 1, 2, 3, and 4.
In some embodiments, method 600 is configured to detect presence of an inanimate object, a human, or an animal. Accordingly, step 610 includes receiving a Doppler signal in the frequency domain that is from motion of an inanimate object, a human, or an animal during a frame, such as a moving vehicle, walking person 110, or a cat walking in a room.
Step 710 includes receiving a Doppler signal in time domain. For example, as illustrated by system 400 in
Step 720 includes determining whether motion of the object is detected in accordance with the Doppler signal in the time domain and a first baseline energy. For example, as system 400 illustrated in
Step 730 includes, responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the time domain and the first baseline energy, setting a first flag of object motion. For example, as system 400 illustrated in
Step 740 includes transforming the Doppler signal in the time domain to a Doppler signal in the frequency domain. For example, as system 400 illustrated in
Step 750 includes determining whether motion of the object is detected in accordance with the Doppler signal in the frequency domain and a second baseline energy. For example, as system 400 illustrated in
Step 760 includes, responsive to a determination that motion of the object is detected in accordance with the Doppler signal in the frequency domain and the second baseline energy, setting a second flag of object motion. For example, as system 400 illustrated in
Step 770 includes setting a flag of object presence based on the first flag of object motion and the second flag of object motion. For example, as system 400 illustrated in
In some embodiments, step 740 includes transforming the Doppler signal in the time domain to a Doppler signal in the frequency domain when a criterion is met. For example, as system 400 illustrated in
In accordance with the criterion (a), frequency-domain object motion detector 430 transforms the Doppler signal xk(n) in the time domain to a Doppler signal Xk(f) in the frequency domain and proceeds with frequency-domain object motion detection when time-domain object motion detector 420 does not detect motion of the object in time domain.
In accordance with the criterion (b), frequency-domain object motion detector 430 transforms the Doppler signal xk(n) in the time domain to a Doppler signal Xk(f) in the frequency domain and proceeds with frequency-domain object motion detection when time-domain object motion detector 420 detects motion of the object in the time domain. Thus, frequency-domain object motion detector 430 detects motion of the object only when time-domain object motion detector 420 detects motion of the object. Thus, time-domain object motion detector 420 and frequency-domain object motion detector 430 both detect motion of the object and send time-domain and frequency-domain flags of object motion to object presence detector 440.
In accordance with the criterion (c), frequency-domain object motion detector 430 transforms the Doppler signal xk(n) in the time domain to a Doppler signal Xk(f) in frequency domain and proceeds with frequency-domain object motion detection when time-domain object motion detector 420 detects motion of the object in the time domain, but the signal energy sk of the Doppler signal in the time domain is less than a threshold, e.g., sk<1.2βNk−1. Thus, frequency-domain object motion detector 430 detects motion of the object when time-domain object motion detector 420 detects motion of the object, but the signal energy of the Doppler signal is not strong enough.
In some embodiments, method 700 is configured to detect presence of an inanimate object, a human, or an animal. Accordingly, step 710 includes receiving a Doppler signal from motion of an inanimate object, a human, or an animal during a frame, such as a moving vehicle, walking person 110, or a cat walking in a room.
Processor 860 includes any appropriate type of general-purpose or special-purpose microprocessor, digital signal processor, or microcontroller. Processor 860 can be representative of one or more processors in object presence detector 100.
Memory 880 may include any appropriate type of mass storage provided to store any type of information that processor 860 may need to operate. Memory 880 may be a volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other type of storage device or tangible (i.e., non-transitory) computer-readable medium including, but not limited to, a read-only memory (ROM), a flash memory, a dynamic random-access memory (RAM), and a static RAM. Memory 610 may be configured to store one or more programs for execution by processor 860 for detecting presence of an object in an environment, as disclosed herein.
Memory 880 may be further configured to store information and data used by processor 860. For instance, memory 880 may be configured to store received baseline energies of environmental noise therein for apparatus 800.
I/O interface 870 may be configured to facilitate communication between apparatus 800 and other apparatuses. For example, I/O interface 870 may receive a signal from another apparatus (e.g., a computer) including system configuration information for apparatus 800. I/O interface 870 may also output data of presence detection results to other apparatuses.
Communication unit 850 may include one or more cellular communication modules, including, for example, a 5G radio access system, a Long-Term Evolution (LTE), a High Speed Packet Access (HSPA), a Wideband Code-Division Multiple Access (WCDMA), a Global System for Mobile communication (GSM), a Bluetooth, and/or an Institute of Electrical and Electronics Engineers (IEEE) 802.11 or Wi-Fi communication module.
Antenna 820 may include one or more conductors (elements), electrically connected to receiver 840. During reception, oscillating electric and magnetic fields of an incoming radio wave exert force on the electrons in the antenna elements, causing them to move back and forth, creating oscillating currents in the antenna. Accordingly, antenna 820 can be configured to receive waves or signals in an environment.
Receiver 840 is configured to process the received waves or signals from antenna 820 for presence detection of an object. For example, receiver 840 can include an ADC to sample the received waves or signals from antenna 820 in accordance with a sampling rate, convert sampled signals to baseband signals, and feed results to processor 860. In some embodiments, receiver 840 can be further configured to perform operations of band-pass filter 220 shown in
Processor 860 can be configured by one or more programs stored in memory 880 to perform operations of apparatus 800 described above with respect to the methods shown in
In some embodiments, the signal energies and/or the baseline energies in the methods shown in
Another aspect of the disclosure is directed to a non-transitory processor-readable medium storing instructions which, when executed, cause one or more processors to perform the methods discussed above. The processor-readable medium may include volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of processor-readable medium or processor-readable storage devices. For example, the processor-readable medium may be the storage device or the memory module having the processor instructions stored thereon, as disclosed. In some embodiments, the processor-readable medium may be a disc or a flash drive having the processor instructions stored thereon.
It will be appreciated that the present disclosure is not limited to the exact construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. It is intended that the scope of the application should only be limited by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3932870 | Shapiro | Jan 1976 | A |
| 4187501 | Olesch | Feb 1980 | A |
| 8018371 | Paschen et al. | Sep 2011 | B1 |
| 9229102 | Wright | Jan 2016 | B1 |
| 20050078029 | Okamura | Apr 2005 | A1 |
| 20090154726 | Taenzer | Jun 2009 | A1 |
| 20110148689 | Filippi | Jun 2011 | A1 |
| 20110267222 | Craig | Nov 2011 | A1 |
| 20130041856 | Benitez | Feb 2013 | A1 |
| 20130307562 | Chen | Nov 2013 | A1 |
| 20140070983 | Maalouli | Mar 2014 | A1 |
| 20150301167 | Sentelle | Oct 2015 | A1 |
| 20160124084 | Lim | May 2016 | A1 |
| 20160307418 | Pantus | Oct 2016 | A1 |
| 20170123058 | Yavari | May 2017 | A1 |
| 20190317191 | Santra | Oct 2019 | A1 |
| 20200003864 | Chi | Jan 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 101414005 | Apr 2009 | CN |
| 102007428 | Apr 2011 | CN |
| 103109201 | May 2013 | CN |
| 0945854 | Sep 1999 | EP |
| 2940486 | Nov 2015 | EP |
| 201042961 | Dec 2010 | TW |
| 201711502 | Mar 2017 | TW |
| Entry |
|---|
| European Patent Office, European Search Report of Application No. EP19177157, dated Oct. 17, 2019. |
| Final Office Action issued by the USPTO dated Aug. 3, 2020, for U.S. Appl. No. 15/993,574. |
| Number | Date | Country | |
|---|---|---|---|
| 20190369225 A1 | Dec 2019 | US |
