The present disclosure relates to wireless detection and monitoring and more particularly to monitoring and differentiating individual interaction with products.
Understanding how individuals interact with consumer products is important for the development of product improvements, the creation of future products, ensuring consumer safety, etc. Continuously monitoring these interactions is challenging, however, particularly when sensors such as video monitors are impractical or unwanted. Alternative approaches to continuously monitor individual product interactions are desirable.
The background description provided here is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
In accordance with the present invention, a harmonic radar apparatus includes a transmitter configured to transmit a plurality of fundamental frequencies towards a scene. A further aspect of the harmonic radar apparatus includes a receiver configured to receive a reflected signal from the scene, the reflected signal being modulated based on the scene, and a re-radiated signal from a tag, the re-radiated signal being at a harmonic frequency of at least one of the plurality of fundamental frequencies transmitted by the transmitter.
Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.
The present disclosure will become more fully understood from the detailed description and the accompanying drawings.
A multi-tone harmonic radar can track individuals as well as tagged products and distinguish between an individual and tagged product to monitor individual-product interactions. In an exemplary implementation, two methods are used: (1) micro-Doppler radar for tracking individuals and (2) harmonic tagging for tracking products. Implementing two tracking methods assists the multi-tone harmonic radar in differentiating the individual and the product as well as distinguish how the individual and the product are interacting.
By combining techniques of radar measurement of moving individuals with wireless radio-frequency identification (RFID) technologies to monitor products, the multi-tone harmonic radar independently measures the movements of individuals and products with which individuals are interacting. By differentiating the motions of individuals and products, the nature of the interactions can be determined, leading to a robust and reliable method for analyzing numerous aspects, including length of interactions, range of motions, specific types of motions, etc. The multi-tone harmonic radar is furthermore capable of extending independent product monitoring to a theoretically unlimited number of individual items using RFID technologies.
As described above, the multi-tone harmonic radar is a wireless system combining two unique technologies to enable differentiable measurement of individual product interactions. Micro-Doppler radar measures moving individuals, and passive harmonic RFID tags measure product movement. Using a low-power transmitted signal, micro-Doppler radar detects and tracks the minute motions of the human body by measuring the small frequency shifts induced in the reflected signals from the various moving parts of the human body. By processing the returned reflected signals in the time-frequency domain, motions due to the movement of arms, legs, body, etc. can be measured over time.
Using the reflected signals, classification algorithms can differentiate activities of an individual as well as which body parts are in motion. Combined with the micro-Doppler measurements identifying individual movements, the multi-tone harmonic radar also receives signals from harmonic RFID tags adhered to the products. The monitored products may consumer products, for example, a razor, a phone, a toothbrush, etc. Harmonic RFID tags collect an incident signal (in this case the transmitted signal from the multi-tone harmonic radar) and retransmit a signal at a harmonic of the incident signal. For example, if the multi-tone harmonic radar transmits a signal operating at 2 GHz, a harmonic RFID tag will generate a 4 GHz retransmitted signal from the harmonic RFID tag. The received 4 GHz signal will be modulated in frequency by movement of the product, which can then be measured. Because this frequency is significantly separated from the 2 GHz scattered signal processed by the micro-Doppler radar receiver within the multi-tone harmonic radar, the motion of the product can be easily differentiated by the receiver.
Referring to
The multi-tone harmonic radar 104 includes a transmitter 108 configured to transmit a set of fundamental frequencies, shown in
A harmonic RFID tag 120 may be attached, such as by adhering, to a product 124. The harmonic RFID tag 120 may be configured to re-radiate a signal transmitted by the transmitter 108 at a particular harmonic of the fundamental frequency of the transmitted signal. The particular harmonic that the harmonic RFID tag 120 is configured to re-radiate indicates a type of product, providing differentiation between types of products. In various implementations, a similar approach may be used to identify a particular product, for example, by serial number or other identifying information. Harmonic RFID tags re-radiate incident wireless signals at a harmonic of the incident frequency, enabling easy discrimination of the motion of individuals and products. Traditional radars are only able to measure motion in the radial direction (towards or away from the radar), a new type of radar, interferometric radar, for continuous, 3D motion tracking can measure both radial and angular velocity, enabling full 3D motion tracking
In various implementations, the multi-tone harmonic radar 104 will be a software-defined radio (SDR) for signal generation and reception, combined with microwave hardware including amplifiers, filters, and antennas. The multi-tone harmonic radar 104 transmits a continuous-wave signal at a fundamental frequency f0 that reflects off the individual 116 and the product 124. The reflections from the various parts of the individual's body are strong enough to easily measure the micro-Doppler signature; however, the reflection from the product at f0 will generally be too weak to detect. A harmonic RFID tag placed on the product will re-radiate a signal at 2f0, which can easily be detected since the re-radiated signal will be the only signal at that frequency.
The re-radiated signal from the harmonic RFID tag 120 at the particular harmonic is collected by a harmonic radar antenna 128. Only those harmonic RFID tags that are moving with the individual will generate time-varying frequency shifts, shown in
In an exemplary aspect, the multi-tone harmonic radar 104 communicates with a computing device 140, for example, via a WiFi connection or a near field communications protocol, such as Bluetooth Low Energy. The computing device 140 includes a processor and user interface (not shown). The processor may be configured to analyze data received from the multi-tone harmonic radar 104. For example, the computing device 140 may receive the received signals from the interferometric receiver 132 and conduct analyses on the received signals. For example, the computing device 140 can determine a velocity or a position of the product 124 during each interaction with the individual 116, indicating how the individual 116 is using the product 124. The amount of product use, length of product use, and time of day the product is used may also be monitored.
In various implementations, the computing device 140 may include a classification module 144, an analyzer module 148, and a display module 152. The classification module 144 is configured to distinguish the received signals as the individual 116 moving and/or the product 124 being moved with the individual 116. The classification module 144 may further classify individual movements according to which body parts of the individual are moving. Similarly, the classification module 144 may distinguish between a type of product based on the harmonic of the re-radiated signal.
The analyzer module 148 is configured to analyze the received signals and compare the received signals to kinematic models. The comparison to kinematic models can assist the computing device 140 in determining the type of interaction being monitored by the multi-tone harmonic radar 104. For example, kinematic models may be generated and uploaded to a central database for comparison to monitored interactions. The computing device 140 may also include a storage 156 configured to store received signals for future analysis and processing. The computing device 140 may be configured to identify where the product 124 is located, how the individual 116 is holding the product 124 (if the individual 116 is holding the product 124), how often the product 124, is being used by the individual 116, and additional individual interactions with products.
Being better able to monitor how individuals interact with particular products provides insight with how able-bodied individuals interact with products as well as particular individuals, such as the elderly, handicapped, etc. For example, the multi-tone harmonic radar 104 may inform how an individual is interacting with a particular product when the individual has a limited range of motion, such as if the individual has difficulty lifting their arms or if the individual is wheelchair bound, their particular interactions with certain products may inform of options for product improvement. In various implementations, the multi-tone harmonic radar 104 may be incorporated into electronics commonly existing in living spaces, incorporated into the infrastructure of a living space, or as a separate device. Additionally, the multi-tone harmonic radar 104 may monitor a plurality of products, each product having a unique identifier (as described above) to re-radiate the transmitted signal at a different harmonic to differentiate the products.
In various implementations, the storage 156 may be accessed by a machine learning system configured to generate kinematic models of identified individuals. Once an individual has been monitored for an amount of time, the machine learning system is able to generate a kinematic model corresponding to that individual. In this way, the machine learning system may distinguish between different individuals in a living space.
A multi-tone micro-Doppler is a continuous-wave radar that is configured to transmit a set of tones separated in frequency by a bandwidth greater than the expected individual micro-Doppler signature, enabling up to tens of frequencies while still easily detecting the micro-Doppler signature. In various implementations, a pulse wave form may alternatively or additionally be used to also gather range data to determine a location or distance of a moving object from the radar.
Harmonic RFID tags may be matched to each tone. The spectrum captured by the harmonic radar then consists of many copies of the individual micro-Doppler signature (any of which can easily be processed for individual motion measurement) and the spectra of any present products at their specific harmonic frequencies. In this manner, the motion of a number of separate products can be tracked by observing the changes in frequency of the individual harmonic signatures. In various implementations, dipole-based harmonic tags may be used as harmonic RFID tags due to their simplicity. Additionally, electrically small, narrow-band antennas may be used to enable greater spectral density of the harmonic RFID tags, resulting in tracking of more individual products.
Implementing scalability to many thousands of products will require unique signatures on each harmonic RFID tag. Additional hardware on each harmonic RFID tag can implement unique ID signatures; however, embedded RFID signatures necessitates the capture and storage of more energy than simple harmonic tags.
Micro-Doppler frequencies are produced whenever a radar system is viewing an object with micro-motions in the radial direction. An individual walking towards a radar generates micro-Doppler through the backscattered radar signals from the torso, arms, and legs. The torso gives rise to the bulk Doppler frequency, while the periodic arm swing and leg swing gives rise to the micro-Doppler frequencies around the bulk Doppler frequency. The information contained in the micro-Doppler frequencies can therefore be used to discriminate between objects, and to classify the activities of individuals based on their velocity profiles. For example, the classification module 144 may discriminate between varying velocities to identify activities of an individual or motion of different body parts of the individual.
For example, a micro-Doppler signature of a walking individual exhibits a large oscillatory signal due to the movement of the legs, while the oscillatory signatures with less frequency spread are due to the movement of the arms. In a micro-Doppler signature, the vehicle produces a strong bulk frequency shift while the individual produces oscillating frequency sidebands. These types of signature features are used to classify individuals and vehicles, and notably, a long duration signal is not necessary for classification. The classification module 144 may be implemented using classification algorithms that discriminate between vehicles and individuals using micro-Doppler features in short time frames.
Unlike traditional radar sensors, which transmit and receive at the same frequency, detection of harmonic RFID tags utilizes a nonlinear device which retransmits or re-radiates a signal at a harmonic of the incident frequency. A simple tag can consist of a dipole antenna which is ½ wavelength long at the incident frequency and is 1 wavelength long at the first harmonic frequency, ensuring high radiation efficiency in both bands. For example, a diode located at the feed point of the dipole provides sufficient harmonic capabilities. In various implementations, Schottky barrier diodes (SBD) were used as the nonlinear device for tags affixed to insects for the purposes of tracking. Such tags can be small and lightweight, weighing 10 mg or less.
The multi-tone harmonic radar 104 transmitting using the transmitter 108 at a fundamental frequency receives the signal scattered off the object of interest in the presence of scattered signals from the background and any other objects in the scene. For a motion detection radar, clutter responses are not a problem except for objects moving with very low speed, where the Doppler shift is close to or within the Doppler spread of the clutter. For a radar operating in the 2-4 GHz band, movements of individuals are generally fast enough to easily discern the Doppler shift from the response of the background clutter. However, the movement of small objects or slow-moving objects can be challenging to detect.
The benefit of using harmonic tags for motion tracking is that the scattered harmonic signal has a better signal-to-clutter ratio than the signals scattered at the fundamental frequency, since the only signal present at the harmonic frequency is due to the harmonic RFID tag. With this method, small and slow-moving objects can be detected, such as insects.
While the radar can easily detect and monitor the movement of the individual 116, differentiating the movement of a product in the same frequency band is extremely challenging. As shown in
In various implementations, as described above, the harmonic RFID tag 120 may instead be a dipole-based tag. Dipole-based harmonic tags are simple and easy to fabricate as well as tune to the specific frequencies of the multi-tone harmonic radar 104. The drawback with dipole antennas is that the bandwidth is wide, limiting the number of products that can be simultaneously tracked. An electrically small antenna which not only has inherently narrow bandwidth, enabling greater scalability in frequency, but is also more compact may be used as a harmonic RFID tag as well.
The transmitter 108 transmits a multi-tone radar signal with N tones spaced evenly in frequency by δf and can be represented in general by Equation 1:
where f0 is the lowest frequency and an are the amplitudes of the tones. The received signal is composed of two summations, a first is the result of the scattering of the transmitted fundamental frequencies [f0, f0+δf, . . . Nf0+δf] off the individual, and a second the result of the harmonics generated by the harmonic RFID tags.
The received signal received by the interferometric receiver 132 is given in general by Equations 2 and 3:
s
rx(t)=srx1(t)+srx2(t)
where
is the signal scattered off the individual with M scattering centers of velocity vm (e.g. torso, arms, legs); the amplitude terms bn are due to propagation while bm account for differences in reflectivity of the separate body parts, and any differential multipath. The signal received from the N harmonic RFID tags is then given by Equation 4:
where dn accounts for propagation and effective scattering cross-section of the tags, hn represents the harmonic that the nth tag is tuned to (for example, the second harmonic may be used for all harmonic RFID tags, thus hn=h=2), and vn is the radial velocity of the nth tag.
Signal generation and reception may be implemented in SDR. The principal limitations on frequency separations will be the maximum expected micro-Doppler shift from the individual and the bandwidth of the dipole. At a carrier frequency of 3 GHz, the bandwidth of an individual micro-Doppler response will generally be less than 1 kHz, thus the tone separation can be quite small. The limitation will then be on the bandwidth of the harmonic RFID tag antenna, which must be small enough that adjacent tag channels do not overlap. This bandwidth depends on the design of the antenna.
Referring now to
By implementing the interferometric receiver 132 composed of two elements widely separated (D) and correlating the signals received by each antenna (for example, 128 and 136), the object 800 passing through the resulting interference pattern 804 generates an oscillatory signal response, shown in
Compared to the Doppler frequency,
where v is the radial velocity of the object, the interferometric frequency response is in a mathematically identical form. The difference is that, with the interferometric measurement, the frequency response is designed based not only on the wavelength of the transmitted signal, but also on the separation (D) of the antenna elements 128 and 136, enabling two degrees of freedom with which the response can be designed. Because the receiver implements a correlation process, the response is generally independent of the transmit waveform. This process works even when sensing only passive thermally-generated radiation from individuals. Therefore, the interferometric measurement can potentially be added to existing radar systems with the addition of a second antenna and receiver.
A primary benefit of the interferometric measurement technique is the ability to measure motion where the Doppler sensors cannot. When the radial motion is minimal or zero, the Doppler sensor cannot discern the motion of the object; however, the interferometric sensor measures the highest frequency shift when the motion is strictly angular (zero radial velocity). Thus, the interferometric sensor and the Doppler sensor provide complementary measurements. The complementary nature can be seen in terms of the total velocity vector of the moving point object, given by v=rvr+φvt, where vr is the radial velocity and vt is the tangential linear velocity. The Doppler frequency shift is then given by Equation 6:
where φ is the angle between the object trajectory vector and the radial vector towards the sensor. The angular velocity can be given in terms of the tangential linear velocity by
In terms of the total velocity, the frequency shift from the interferometric measurement is given by Equation 7:
Equations 6 and 7 show that the two measurements are complementary in terms of the direction of the object relative to the sensor. Thus, by combining both the interferometric measurement and the Doppler measurements, the angle of a moving object can be measured regardless of the direction of motion.
Referring now to
The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.
In this application, including the definitions below, the term “module” or the term “controller” may be replaced with the term “circuit.” The term “module” may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.
The module may include one or more interface circuits. In some examples, the interface circuit(s) may implement wired or wireless interfaces that connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of a LAN are Institute of Electrical and Electronics Engineers (IEEE) Standard 802.11-2016 (also known as the WIFI wireless networking standard) and IEEE Standard 802.3-2015 (also known as the ETHERNET wired networking standard). Examples of a WPAN are the BLUETOOTH wireless networking standard from the Bluetooth Special Interest Group and IEEE Standard 802.15.4.
The module may communicate with other modules using the interface circuit(s). Although the module may be depicted in the present disclosure as logically communicating directly with other modules, in various implementations the module may actually communicate via a communications system. The communications system includes physical and/or virtual networking equipment such as hubs, switches, routers, and gateways. In some implementations, the communications system connects to or traverses a wide area network (WAN) such as the Internet. For example, the communications system may include multiple LANs connected to each other over the Internet or point-to-point leased lines using technologies including Multiprotocol Label Switching (MPLS) and virtual private networks (VPNs).
In various implementations, the functionality of the module may be distributed among multiple modules that are connected via the communications system. For example, multiple modules may implement the same functionality distributed by a load balancing system. In a further example, the functionality of the module may be split between a server (also known as remote, or cloud) module and a client (or, user) module.
While various embodiments have been disclosed, it should be appreciated that additional variations of the wireless monitoring system and method are also envisioned. For example, additional or different hardware components may be used although certain of the present advantages may not be fully realized. It is also noteworthy that any of the preceding features may be interchanged and intermixed with any of the others. Accordingly, any and/or all of the dependent claims may depend from all of their preceding claims and may be combined together in any combination. Variations are not to be regarded as a departure from the present disclosure, and all such modifications are entitled to be included within the scope and spirit of the present invention.
The dimensions and values disclosed herein are not to be understood as being strictly limited to the exact numerical values recited. Instead, unless otherwise specified, each such dimension is intended to mean both the recited value and a functionally equivalent range surrounding that value. For example, a dimension disclosed as “40 mm” is intended to mean “about 40 mm.”
Every document cited herein, including any cross referenced or related patent or application and any patent application or patent to which this application claims priority or benefit thereof, is hereby incorporated herein by reference in its entirety unless expressly excluded or otherwise limited. The citation of any document is not an admission that it is prior art with respect to any invention disclosed or claimed herein or that it alone, or in any combination with any other reference or references, teaches, suggests or discloses any such invention. Further, to the extent that any meaning or definition of a term in this document conflicts with any meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to that term in this document shall govern.
While particular embodiments of the present invention have been illustrated and described, it would be obvious to those skilled in the art that various other changes and modifications can be made without departing from the spirit and scope of the invention. It is therefore intended to cover in the appended claims all such changes and modifications that are within the scope of this invention.
This application claims the benefit of U.S. Provisional Application No. 62/903,005 filed Sep. 20, 2019, and U.S. Provisional Application No. 62/814,067 filed Mar. 5, 2019, the substance of which are incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
62814067 | Mar 2019 | US | |
62903005 | Sep 2019 | US |