Patent Application
20170074980
This invention relates to tracking objects via radio reflections.
Recently, technologies have emerged that can localize a user based purely on radio reflections off the person's body. Challenges related to such technologies include scaling to multiple people, accurately localizing them and tracking their gestures, and localizing static users as opposed to requiring the user to move to be detectable.
Alternative technologies for detecting presence of and/or tracking people in an indoor environment include camera-based approaches in which captured images or video is processed, and ultrasonic or passive infrared motion sensors. Each of these alternative technologies suffers from limited accuracy, computational complexity, privacy concerns, and/or cost of deployment.
In one aspect, in general, a reference transmit signal is distributed to each of one or more transmit antennas, and delayed by multiple different times before transmission from the transmit antennas. The reference transmit signal (or a delayed version of the reference signal) is also used at each of the receive antennas to determine propagation (time of flight) times reflecting from bodies from each of the transmit antennas to the receive antenna. In some examples, these propagation times are determined by modulating the received signals by the reference transmit signal. In some examples, the demodulated received signals are averaged (either as complex signals with magnitude and phase, or alternatively averaging magnitude of squared magnitude alone) over a number of sweep cycles to obtain an averaged received signal. In some examples, rather than distributing the reference transmit signal to each transmit and receive antenna station, a reference timing signal is distributed to the stations, and the reference transmit signal is synthesized separately at each station. Locations of multiple bodies can then be determined from the determined propagation times between multiple pairs for transmit antennas and receive antennas.
In another aspect, object are tracked in an environment by forming a first reference signal having a first component that has a varying frequency with time. A plurality of modified reference signals corresponding to the first reference signal are formed such that each modified reference signal comprising a modified component that is a shift of the first component in frequency and/or time. These modified reference signals are transmitted or caused to be transmitted from one or more transmitting antennas. One or more received signals are processed, with each received signal corresponding to a different receiving antenna of one or more receiving antennas. Each of the one or more received signals is processed according to the first reference signal to form corresponding one or more processed received signals. A plurality of propagation delays are then determined from the processed received signals. Each propagation time delay corresponds to one receiving antenna of the one or more receiving antennas and one transmitting antenna of the one or more transmitting antennas. The processing includes determining from each processed received signal one or more of the plurality of propagation delays. In some examples, the plurality of propagation delays are to track (e.g., localize and/or monitor motion of) one or more objects in the environment according to propagation delays of reflections of the transmitted modified reference signals from locations of the one or more objects.
Aspects can include one or more of the following features.
Causing the modified reference signals to be transmitted includes causing concurrent transmission different modified reference signals from at least two different transmitting antennas.
Each of the different modified reference signals comprises a different time delay of the first reference signal.
Forming the first reference signal comprising a first component having a varying frequency with time includes forming said reference signal such that the first component comprises a frequency sweep from a first frequency to a second frequency from a first time to a second time. In some examples, the frequency sweep comprises a linear sweep.
Forming the plurality of modified reference signals comprises, forming at least a first modified reference signal as a time delay of the first reference signal.
Forming the plurality of modified reference signals comprises forming at least a first modified reference signal as modulation of the first reference signal with a carrier signal (e.g., a sinusoid).
Processing the one or more received signals comprises forming at least a first processed received signal as a modulation of a received signal with the first reference signal.
Determining the plurality of propagation delays comprises performing a spectral analysis of the modulation of the received signal.
Determining the plurality of propagation delays comprises indentifying a plurality of spectral peaks in the spectral analysis, each spectral peak corresponding to a propagation delay.
The steps of any one of the methods set forth above are repeated to track motion of the one or more objects.
The one or more transmitting antennas are selected from a plurality of antennas for each repetition, the one or more transmitting antennas being different in at least some iterations.
At least one antenna is a receiving antenna in one iteration and a transmitting antenna in another iteration.
In another aspect, in general, software stored on a non-transitory machine-readable medium comprises instruction that when executed on a processor of a localization system causes said system to perform all the steps of any one of the method set forth above.
In another aspect, in general, a localization system includes a plurality of antennas, comprising one or more transmitting antennas and one or more receiving antennas. A signal processor is configured to perform all the steps of any one of the methods set forth above. In another aspect, in general, locations of multiple objects are determined by iteratively (a) determining a location of a body based on determined propagation times between multiple transmitter-receiver pairs, and (b) having determined a location, effectively removing the effect of reflections from that location from the remaining signals. In some examples, the step of removing the effect of reflections from a location includes removing the effect of reflections from a region around the locations, where that region is determined according to a spatial model of the object causing the reflections (e.g., a model of a human body). In some examples, this iteration is repeated after having identified N bodies in a region by, at each iteration, determining a location for the nth body having removed the effect of the other N−1 bodies in the received signals.
In another aspect, in general, a method for multi-object localization in an environment (e.g., an indoor environment) makes used to received signals that representing radio frequency reflections of signals emitted (e.g., concurrently) from one or more transmitting antennas and received at one or more receiving antennas after reflection from a plurality of objects (e.g., moving objects, people) in the environment. The received signals are used to determine a plurality of time profiles. Each time profile is associated with a different pair of a transmitting antenna and a receiving antenna. Each time profile characterizes a degree of reflection (e.g., a reflected power) for different propagation times from the transmitting antenna to the receiving antenna. A plurality of time profiles is thereby formed with each time profile being associated with a corresponding transmitting antenna and a corresponding receiving antenna. The plurality of time profiles and locations in the environment of the transmitting and the receiving antennas corresponding to said time profiles are used to identify a first location of reflection of a plurality of locations in the environment. One or more iterations are then performed, with each iteration including cancelling an effect of a reflection from a previously identified location of reflection in each of one or more of the time profiles, and using the time profiles in which effects of reflections from one or more identified locations have been cancelled and locations of the transmitting and the receiving antennas corresponding to said time profiles to identify a further location of reflection of the plurality of reflections in the environment. The plurality of locations of reflection identified in the environment is then provided.
Aspects can include one or more of the following features.
The method further includes determining, for an identified location of refection, a plurality of time portions of the received signals corresponding to reflections from the identified location of reflection. In some examples, the determined plurality of time portions of the received signals is further processed to determine motion characteristics of an object at the identified location.
The method further comprises causing concurrent emission of the signals from the one or more transmitting antennas of a plurality of antennas.
Determining a time profile for a pair of a transmitting antenna and a receiving antenna includes determining said time profile to characterize a distribution of reflected energy as a function of a propagation time from the transmitting antenna to the receiving antenna.
Using the time profiles to identify a location of reflection includes, for each time profile forming a spatial distribution of potential locations of reflections represented in said time profile, combining the spatial distributions for multiple time profiles to form a combined spatial distribution, and determining a location in the environment according to the combined spatial distribution.
Determining the location according to the combined spatial distribution includes determining a location of an extremum value in the combined distribution.
The spatial distribution comprises a two-dimensional spatial distribution and the determined location corresponds to a two-dimensional coordinate in the environment.
The locations of the transmitting and receiving antennas are non-coplanar.
The spatial distribution comprises a three-dimensional spatial distribution and the determined location corresponds to a three-dimensional coordinate in the environment.
Cancelling an effect of a reflection from a previously identified location of reflection in a time profile includes attenuating a portion of the time profile corresponding to a propagation time from the location of the transmitting antenna via the identified location to the location of the receiving antenna.
Attenuating a portion of the time profile comprises zeroing the time profile in a portion of said profile corresponding to the propagation time.
Attenuating a portion of the time profile includes using an estimated mean and standard deviation of an estimate of the propagation time from the location of the transmitting antenna via the identified location to the location of the receiving antenna.
The method further includes removing an effect a reflection from a static object in the plurality of time profiles.
The method further includes removing an effect of multipath reflection from at least one location of the plurality of reflections also reflecting from a static object.
In another aspect, in general, software stored on a non-transitory machine-readable medium comprises instruction stored thereon, that when executed on a processor of a localization system causes said system to perform all the steps of any method set forth above.
In another aspect, in general, a localization system includes a plurality of antennas, one or more signal generators coupled to multiple of the antennas, the one or more signal generators configured to provide signals for concurrent transmission from the multiple antennas, and one or more receivers coupled to corresponding antennas configured to receive signals comprising reflections of the concurrently transmitted signals. The system also includes a received signal processor configured to infer locations of multiple objects based on the received signals. The received signal processor is configured to perform all the steps of any of the methods set forth above.
Aspects can include one or more of the following features.
The effect of a fixed pattern of reflection (e.g., from walls, etc.) is effectively removed from the receive signals, for example, by subtracting a long-term average of the receive signals, or by taking a difference of receive signals from successive frequency sweeps.
The transmit and/or receive antennas are directional antennas.
The antennas are arranged in a fixed arrangement along a line, in a two-dimensional pattern (e.g., in a “T” pattern on a face of a region in which bodies are tracked), or in a three-dimensional arrangement. In some examples, the fixed arrangement is known, while in other examples, the geometric relationship of the antenna locations is inferred using radio propagation time between locations.
In some examples, the antennas are arranged in a room, for example, along a wall or a ceiling. In other examples, the antennas are arranged over multiple rooms of a multi-room space (e.g., a hospital wing or a retail store). In another aspect, in general, breathing of relatively motionless bodies (e.g., humans, pets, etc.) is detected using an oscillation in reflected power from a location of a detected body at a breath frequency. In some examples, the difference between a reference power for a body at a reference time and subsequent power for that body is used to detect the breathing related oscillations. In some implementations, multiple subtraction windows are used to identify and/or localize reflectors with different speeds. One or more persons can be localized based on their breathing. In some examples, the system counts the breaths of one or more people, and can detect occasions when one or more people hold their breaths.
In another aspect, in general, a technique referred to as Multi-Shift Frequency-Modulated Carrier Wave (FMCW) is used. The technique allows distinguishing between FMCW signals generated from different sources by shifting them in time or in frequency. This distinguishability can be achieved by shifting in time, frequency, or code (e.g., multiplying the signals transmitted from the different antennas by different spreading codes).
In another aspect, in general, a method and apparatus is used for distinguishing between frequency modulated carrier wave (FMCW) signals transmitted by different sources. The different sources may be connected to a same or to multiple different devices. In some examples, distinguishability is achieved by shifting the FMCW signals with respect to each other in the frequency domain. In some implementations, shifting in the frequency domain is achieved by mixing with a signal while in other implementations, the shifting in the frequency domain is achieved by adding a delay line. In some alternative implementations, distinguishability is achieved by shifting the FMCW signals with respect to each other by multiplying them with different codes.
Advantages of one or more aspects can include the following.
Detection of breathing provides a way of detecting a relatively stationary body, which could otherwise be mistaken for background. For example, in an application that monitors presence of people in a room, an ultrasonic motion detector may miss the motionless person whereas their breathing would nevertheless be detected.
Concurrent transmission from all (or sets of multiple fewer than all) transmit antennas can be less affected by motion of a body (e.g., less “smearing” of the estimated location) than sequential transmission from single antennas where the change in location may be significant between transmit times.
Use of a narrow band signal (i.e., a swept tone) provides a direct way of separating the transmissions from different transmitters in a receive signal, and can provide robustness to phase noise.
The approached described above for tracking people can be used to applications including the following:
Security applications to detect people in a building (e.g., intrusion detection) or taking particular paths in a building;
Energy efficiency approaches based on detected people and longer-term patterns of traffic of people in a building (e.g., energy usage can be based on the number of people in a room rather than mere presence of a person);
Traffic tracking in a retail space, for example, to determine traffic patterns of individual people as they shop in a store;
Adjusting building services (e.g., heating) according to a number of detected people in a space;
Control of appliances or other devices, for example, by pointing or wave gestures;
Tracking people for gaming and virtual reality applications;
Elderly monitoring and fall detections; and
Search and rescue missions.
Monitoring of sleep patterns, detection of sleep apnea, baby monitoring, and other applications of vital sign monitoring have an advantage of not requiring physical sensors or connections to a person.
Other features and advantages of the invention are apparent from the following description, and from the claims.
Referring to
Generally, the system 100 makes use of time-of-flight (TOF) (also referred to as “round-trip time”) information derived for various pairs of antennas 150. For schematic illustration in
For a particular path, the TOF, for example associate with path 185A, constrains the location of the object 180 to lie on an ellipsoid defined by the three-dimensional coordinates of the transmitting and receiving antennas of the path, and the path distance determined from the TOF. For illustration, a portion of the ellipsoid is depicted as the elliptical line 190A. Similarly, the ellipsoids associated with paths 185B-C are depicted as the lines 190B-C. The object 180 lies at the intersection of the three ellipsoids.
Continuing to refer to
Referring to
Referring to
The output of the frequency shifter is subject to a spectral analysis (e.g., a Fourier Transform) to separate the frequency components each associated with a different TOF. In this embodiment, the output of the frequency shifter is samples and a discrete time Fourier Transform implemented as a Fast Fourier Transform (FFT) is compute for each interval 212. Each complex value of the FFT provide a frequency sample with a frequency resolution Δf=1/Tsweep where Tsweep is the sweep duration (e.g., 2.5 milliseconds).
Continuing to refer to
The system addresses the first multipath effect, referred to as static multipath, using a time differencing approach to distinguish a moving object's reflections from reflections off static objects in the environment, like furniture and walls. Typically, reflections from walls and furniture are much stronger than reflections from a human, especially if the human is behind a wall. Unless these reflections are removed, they would mask the signal coming from the human and prevent sensing her motion. This behavior is called the “Flash Effect”.
To remove reflections from all of these static objects (walls, furniture), we leverage the fact that since these reflectors are static, their distance to the antenna array does not change over time, and therefore their induced frequency shift stays constant over time. We take the FFT of the received signal every sweep window and eliminate the power from these static reflectors by subtracting the (complex) output of the FFT in a given sweep from the FFT of the signal in a previous sweep. This process is called background subtraction because it eliminates all the static reflectors in the background. In some embodiments, the immediately previous sweep (i.e., 2.5 milliseconds previous), while in other embodiments, a greater delay may be used (e.g., 12.5 milliseconds, or even over a second, such as 2.5 seconds).
By eliminating all reflections from static objects, the system is ideally left only with reflections from moving objects. However, as introduced above, these reflections include both signals that travel directly from the transmitting antenna to the moving body (without bouncing off a static object), reflect of the object, and then travel directly back to the receiving antenna, as well as indirect paths that involve reflection from a static object as well as form a moving object. We refer to these indirect reflections as dynamic multi-path. It is quite possible that moving object reflection that arrives along an indirect path, bouncing off a side wall, is stronger than her direct reflection (which could be severely attenuated after traversing a wall) because the former might be able to avoid occlusion.
The general approach eliminating dynamic multi-path is based on the observation that, at any point in time, the direct signal paths to and from the moving object has traveled a shorter path than indirect reflections. Because distance is directly related to TOF, and hence to frequency, this means that the direct signal reflected from the moving object would result in the smallest frequency shift among all strong reflectors after background subtraction. We can track the reflection that traveled the shortest path by tracing the lowest frequency (i.e., shorted time of flight) contour of all strong reflectors.
Referring to
To determine the first local maximum that is caused by a moving body, we must be able to distinguish it from a local maximum due to a noise peak. We achieve this distinguishability by averaging the spectrogram across multiple sweeps. In this embodiment, we average over five consecutive sweeps, which together span a duration of 12.5 milliseconds, prior locating the first local maximum of the FFT power. For all practical purposes, a human can be considered as static over this time duration; therefore, the spectrogram (i.e., spectral distribution over time) would be consistent over this duration. Averaging allows us to boost the power of a reflection from moving body while diluting the peaks that are due to noise. This is because the human reflections are consistent and hence add up coherently, whereas the noise is random and hence adds up incoherently. After averaging, we can determine the first local maximum that is substantially above the noise floor and declare it as the direct path to the moving body (e.g., a moving human).
In practice, this approach using the first reflection time rather than the strongest reflection proves to be more robust, because, unlike the contour which tracks the closest path between a moving body and the antennas, the point of maximum reflection may abruptly shift due to different indirect paths in the environment or even randomness in the movement of different parts of a human body as a person performs different activities.
Note that the process of tracking the contour of the shortest time of flight is carrier out for each of the transmitting and receiving antenna pairs, in this embodiment, for the three pairs each between the common transmitting antenna and the three separate receiving antennas. After obtaining the contour of the shortest round-trip time for each receive antenna, the system leverages common knowledge about human motion to mitigate the effect of noise and improve its tracking accuracy. The techniques used include:
After contour tracking and de-noising of the estimate, the system obtains a clean estimate of the distance traveled by the signal from the transmit antenna to the moving object, and back to one of the receive antennas (i.e., the round trip distance). In this embodiment that uses one transmitting antenna and three receiving antenna, at any time, there are three such round trip distances that correspond to the three receive antennas. The system uses these three estimates to identify the three-dimensional position of the moving object, for each time instance.
The system leverages its knowledge of the placement of the antennas. In this embodiment, the antennas are placed in a “T” shape, where the transmitting antenna is placed at the crosspoint of the “T” and the receiving antennas are placed at the edges, with a distance of 1 meter between the transmitting antenna and each of the receiving antennas. For reference, the z axis refers to the vertical axis, the x axis is along the horizontal, and with the “T” shaped antenna array mounted to a wall the y axis extends into the room. Localization in three dimensions uses the intersection of the three ellipsoids, each defined by the known locations of the transmitting antenna and one of the receiving antennas, and the round-trip distance.
Note that in alternative embodiments, only two receiving antennas may be used, for example with all the antennas placed along a horizontal line. In such an embodiment, a two dimensional location may be determined using an intersection of ellipses rather than an intersection of ellipsoids. In other alternative embodiments, more than three receiving antennas (i.e., more than three transmitting-receiving antenna pairs) may be used. Although more than three ellipsoids do not necessarily intersect at a point, various approaches may be used to combine the ellipsoids, for example, based on a point that is closest to all of them.
Referring to
In
Referring to
Referring to
Referring again to
In
As introduced above, each transmit-receive pair of antennas yields one TOF profile. Each point of a TOF profile corresponds to one ellipsoid in three dimensions, or equivalently an ellipse in two dimensions (e.g., if the antennas and the objects are assumed to be in one plane). Therefore, an entire TOF profile may be mapped into a spatial distribution of estimated location, referred to below as a “heat map,” by setting the intensity along each ellipse (or ellipsoid) according to each point of the TOF profile. In an example generated with two people were present in an 5 meter by 7 meter room, difference TOF profile using one transmit-receive antenna pair is shown in
Note that with only one antenna pair as shown in
In practice the combination (i.e., superposition) of heat maps from different antenna pairs as described above may not be sufficient to accurately distinguish the location of bodies (i.e., people) in the environment of the system. One reason is that different users may exhibit a “near-far” problem. Specifically, the reflections of a nearby person are generally much stronger than the reflections of a faraway person or a person behind an obstruction.
In what follows, we describe each of these four steps in detail by walking through the example with four persons shown in
In the first step, Detection, the location of the highest power reflector in the two-dimensional heat map of
Given the (x, y) coordinates of a person determined in the Detection step, in the Remapping step, we map this location back to a corresponding TOF range separately for each transmit-receive pair. Although the heat map is computed in two dimensions, each person is not a point reflector and rather has an extent in the vertical (z) direction. Therefore, the effect of reflections off his entire body on the TOF profile of each transmit-receive pair is estimated. For an antenna pair at three-dimensional coordinates (xt, yt, zt) and (xr, yr, zr), the shortest TOF for an object at (x, y) is with a reflection from the three-dimensional point (x, y, (zt−zr)/2). This TOF(min) is computed using the propagation speed c. The greatest TOF for an object at (x, y) is assumed to correspond to a reflection from the three-dimensional point (x, y, 0), essentially assuming the point at floor level is farthest from the antennas. This TOF(max) is also computed using the propagation speed c. The range [TOF(min),TOF(max)] for each antenna pair is the product of the Remapping step. In other embodiments, other approaches to mapping and (x, y) into a range or other distribution over TOF for each antenna pair may be used, for example, using an estimated mean and standard deviation estimate of the TOF.
The Cancellation step is performed for each antenna pair. In one embodiment, the current TOF profile for an antenna pair is zeroed (or otherwise attenuated) in the estimated range of TOF corresponding to the estimates (x, y) of the person. The combined heat map is then calculated by summing the heat maps determined from the TOF profiles after cancellation.
On the second cycle, in step 1 using the heat map of
In some examples, a further Refocusing step is performed to improve the location estimates of the people found in the iteration described above as illustrated in
In some examples, time sequences of estimated (x, y) locations, or sets of locations, may be further processed. For example, trajectory smoothing and/or assignment of locations to individuals may be performed, for example, using Kalman filtering or other model based or heuristic techniques.
In some examples, the system can differentiate a hand motion from a whole-body motion (like walking) by leveraging the fact that a person's hand has a much smaller reflective surface than his entire body. Using the cancellation and focusing approaches described above, the system can track gestures even when they are simultaneously performed by multiple users. In particular, by exploiting the focusing step, the system focuses on each person individually and track his/her gestures. For example, a pointing gesture, where different users may point in different directions at the same time, can be detected. Subsequently, by tracking the trajectory of each moving hand, the system determines the direction in which each of the users is pointing. Note that the users perform these pointing gestures in three-dimensional and the system tracks the hand motion by using the TOFs from the different transmit-receive antenna pairs to construct a three dimensional point cloud in the refocusing step rather than a two-dimensional heat map.
The approach described above is effective at detecting and estimating the location of moving bodies. In some implementations, as second approach is used instead of or in addition to detect relatively stationary people (e.g., sitting in a chair or lying in bed) based on their periodic breathing motion. In the approach described above, a difference TOF profile for each antenna pair is determined by taking difference over a relatively small time interval, for example, over 12.5 milliseconds. Breathing occurs at a much slower rate, for example, with a breath time of a few seconds. Generally, an approach to detection of breathing people in the environment is based on detecting modulation of the TOF profile over time with a characteristic frequency consistent with breathing. A variety of approaches may be used to detect such modulation for different TOF, for example, by analysis of a series of TOF profiles over a sufficient time scale to detect the modulation. One way to do this is to apply the approach described above, but rather than can computing the different TOF profile with a short time difference (e.g., 12.5 milliseconds), a longer time difference (e.g., over a second, for example, 2.5 seconds) is used.
The SSC procedure described above enables focusing on each person while eliminating interference from all other users. This algorithm proves very effective to monitoring each person's breath in the presence of other people in the environment. Recall that the SSC focusing step allows us to focus on each person while eliminating interference from all other people in the scene (as shown in
There are a great many other potential applications of systems using one or more of the approaches described above, recognizing that references to “the system” above and below are to one or more embodiment of systems that incorporate one or more of the techniques described above. For example, not all systems make use of a each of short time scale differencing, longer time scale differencing, SCC, refocusing, for phase-based processing. Also, as is evident from the context, in some applications, a system may have multiple units each with its own antenna array to monitor large environments.
One potential application relates to energy saving and building automation. The system can identify the number of people in a room or a house and their locations. Therefore the system can be leveraged as an element of a building an automation system that proactively adjusts the temperature in a room based on the number of people or turns off lights/HVAC for empty rooms. This can help in saving energy as well as increase awareness to how people roam around buildings. It could also learn when people come in and when they leave and predicatively turn on, off, or adjust light/HVAC/temperature systems.
There is also a large interest in understanding how employees and visitors use enterprise environments: where they spend most of their time, how teams/groups form and generate ideas, what spaces are unused and what spaces are excessively used, etc. The system (or multiple systems distributed around the enterprise environment) can provide deep insights by tracking employees as well as visitors movements as they roam around spaces. Hence, it can be used to reduce or reorganize the spaces that are not used. Alternatively, it can be used to reorganize spaces where people typically meet and come up with ideas.
In home automation setting, the system can be used for a gesture-based interface whereby a person automates different household appliances by performing a gesture. The system can track the gesture and interpret it as a command; this will allow the person to control appliances such as: turn on and off the TV, write a letter or a word in the air, turn off other household appliances, etc. In addition, the system can learn people's everyday actions and automate tasks. For example, it can learn when a person wakes up (or detect when he/she gets out of bed) and immediately turn on the shower, then when she gets out of the shower, turn on the coffee machine. It can also know when a person is present and hence can determine when she comes into her house. So the system can learn the times at which a person typically returns home (by tracking these patterns across different days) and automatically adjust heating/HVAC/lighting based on previous patterns.
In a retail setting, tracking customers can be used to improve understanding of purchasing behavior. The system can track shoppers in retail stores with very high accuracy providing deep insights into their behavior. It localizes shoppers by relying on radio reflections off their bodies, without requiring them to have a smartphone. It generates wide-ranging in-store analytics such as the number of people who enter/exit the store, the paths they follow, the locations they stop at, and the time they wait in lines to pay. It also tracks hand movements and identifies when a user picks up or returns items to the shelves. These insights provide stores with deep awareness into customer behavior; for example, stores can analyze customer's reactions to their promotion campaigns, and adapt these campaigns in real-time. Further, by understanding customers' trajectories, stores can make products more accessible and more efficiently distribute their staff, thus increasing customer satisfaction, and hence retention. Generally, these analytics enable stores to identify opportunities to increase sales, enhance customer experience and loyalty, and optimize in-store marketing.
There are at least two potential users of the insights that the system can produce for Retail Analytics. First are retail merchandising directors; they increasingly rely on data to determine the most profitable mix of products, pricing and in-store promotions, such as displays, sales support, discounts, etc. Although bricks-and-mortar retail is struggling in the face of competition from e-commerce, leading retailers see a better understanding of customer behavior as the way to stem the bleeding, so spending is shifting to Retail Analytics and is expected to continue to do so over the coming decade.
The second set of Retail Analytics customers are the consumer goods suppliers (CPG companies, primarily). Today, Market Research Directors within CPG companies spend approximately $5.6Bn per year on market research, including insights from point-of-sale data and customer panel surveys, compiled by research firms such as ACNielsen. Location Analytics insights can help CPG Brand Managers (consumers of the Market Research) to be more efficient with sales promotion spend at retailers.
In health care, the system can track patients as they move about in the hospital. For example, patients fall in their rooms and bathrooms, and it is important to detect when they fall. It may also be important to detect when a patient leaves their room and track them as they move around in the hospital. This becomes even more important in psychiatric sections or psychiatric institutions. Further, because the system can monitor people in general, it can also provide information into visitor waiting times. In additional, it can track if a nurse or another caregiver came into the room and did their responsibility (e.g., checked/changed the patient's vitals)
Senior housing and assisted living facilities strive to offer freedom to their residents and to create an atmosphere focused on enjoying life, rather than being afraid of, so they tend to be wary of adopting any monitoring technology. This includes nursing homes, where the need for care is greater; or to assisted living facilities devoted to patients with dementia. The system can be used to track people (mainly the elderly) in senior housing facilities. It can be used to detect falls, track how many times they used the bathroom or went out of their room or bed, whether they came near the window and received enough sunshine. It can also track the visitors (e.g., their number and actions) when they come into the facility, and the caregivers as they take care (or not take care) of the residents.
The system can also be used for home or enterprise security. Specifically, because it can track human motion, it can know if an intruder enters the building, or even if he is roaming outside the house. For example, it can automatically turn off an alarm or send an alert when someone enters the building when the security system is turned on. Further, even if there are residents inside the house or enterprise and it senses a person roaming around outside the facility, it can detect their presence, track their movements, and/or alert about their presence.
Further details regarding two applications are provided below as examples, recognizing that the approaches described can be applied to a variety of the applications outlined above. These two applications are fall detection and detection of pointing gestures.
For fall detection, an objective is to automatically distinguish a fall from other activities including sitting on the ground, sitting on a chair and walking To do so, we build on the system's elevation tracking along the z dimension. Note that simply checking the person's elevation generally is not sufficient to distinguish falls from sitting on the floor. To detect a fall, the system requires two conditions to be met: First, the person's elevation along the z axis must change significantly (by more than one third of its value), and the final value for the person's elevation must be close to the ground level. The second condition is the change in elevation has to occur within a very short period to reflect that people fall quicker than they sit.
In applications that make use of a pointing gesture, the system can estimate a pointing angle, which can then be used to control devices such as electrical or electronic devices. Generally, the system first provides coarse estimation of a body part motion, and then further processes that coarse estimate to determine the pointing angle. We consider the following motion: the user starts from a state where her arm is rested next to her body. She raises the arm in a direction of her choice with the intention of pointing toward a device or appliance, and then drops her hand to the first position. The user may move around and at a random time perform the pointing gesture. We require however that the user be standing (i.e., not walking) when performing the pointing gesture. The goal is to detect the pointing direction.
To track such a pointing gesture, the system distinguishes between the movement of the entire body and the motion of an arm. To achieve this goal, we leverage the fact that the reflection surface of an arm is much smaller than the reflection surface of an entire human body. We estimate the size of the reflection surface from the spectrogram of the received signal at each of the antennas.
Once we detect it is a body part, the system estimates the direction of the motion to identify the pointing direction, which involves the following steps:
Examples of systems that estimate pointing direction can be to enable a user to control household appliances by simply pointing at them. Given a list of instrumented devices and their locations, the system can track the user's hand motion, determine the direction in which she points, and commands the device to change its mode (e.g., turn on or off the lights, or control our blinds).
The system described above may be embodied in various configurations. For example, the antenna array may form a linear array, multiple linear arrays (for example with a horizontal array and a vertical array in a “T’ or inverted “T” shape, of an “H” shape). In other examples, multiple linear arrays may be disposed about the environment.
It should be understood that the shifting of transmitted signals in time/frequency so that reflections from different transmitting antennas can be distinguished at a receiving antenna is only one possible approach. For example, multiplication by different codes (e.g., orthogonal pseudo-random sequences) can be used at the transmitters and then these codes can be uses to separate the signals at the receivers.
The approaches described above are generally implemented as part of the signal analysis module 170, shown in
It is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the appended claims. Other embodiments are within the scope of the following claims.
This application is a continuation of U.S. application Ser. No. 14/510,263 titled “MOTION TRACKING VIA BODY RADIO REFLECTIONS,” filed on Oct. 9, 2014, and claims the benefit of U.S. Provisional Application No. 61/943,957 titled “MULTI-PERSON MOTION TRACKING VIA BODY RADIO REFLECTIONS,” filed on Feb. 24, 2014, andU.S. Provisional Application No. 61/985,066 titled “MULTI-PERSON MOTION TRACKING VIA BODY RADIO REFLECTIONS,” filed on Apr. 28, 2014. The above-referenced applications are incorporated herein by reference.
This invention was made with government support under CNS-1117194 awarded by the National Science Foundation. The government has certain rights in the invention.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2015/017239 | 2/24/2015 | WO | 00 |
| Number | Date | Country | |
|---|---|---|---|
| 61943957 | Feb 2014 | US | |
| 61985066 | Apr 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14510263 | Oct 2014 | US |
| Child | 15120864 | US |
