Patent Application
20160345907
Photoplethysmography (PPG) provides a non invasive optical technique to detect changes in blood volume and blood composition in a biological being. However, PPG readings are susceptible to contamination by noise in the form of motion artifacts that can impact the usefulness of PPG data for biometric interpretation. Specifically, within a vehicle environment, motion artifacts can be enhanced based on noise, road vibration, individual movement, vehicle movement, inertial movement, among others. Motion artifacts become an exceptional property of captured PPG signals that pollute PPG readings and provide a skewed biometric interpretation.
According to one aspect, a computer implemented method for processing photoplethysmography (PPG) signals in a vehicle includes receiving a PPG waveform signal from an optical sensor. The method also includes processing a PPG measurement signal based on the PPG waveform signal. The method additionally includes receiving a noise waveform signal from at least one of: a seat assembly sensor, a vehicle sensor, and a vehicle system. Additionally, the method includes processing a motion artifacts measurement signal based on the noise waveform signal. The method further includes processing a refined PPG signal to suppress the motion artifacts measurement signal from the PPG measurement signal.
According to a further aspect, a system for processing photoplethysmography (PPG) signals in a vehicle includes a computing device that includes a processor. The system also includes a PPG determinant module that is included as a module of the computing device that receives a PPG waveform signal from an optical sensor and processes a PPG measurement signal based on the PPG waveform signal. Additionally, the system includes a motion artifact determinant module that is included as a module of the computing device that receives a noise waveform signal from at least one of: a seat assembly sensor, a vehicle sensor, and a vehicle system and processes a motion artifacts measurement signal based on the noise waveform signal. The system further includes a PPG signal filtering module that is included as a module of the computing device that processes a refined PPG signal to suppress the motion artifacts measurement signal from the PPG measurement signal.
According to still another aspect, a computer readable medium a computer readable medium including instructions that when executed by a processor executes a method for processing photoplethysmography (PPG) signals in a vehicle includes receiving a PPG waveform signal from an optical sensor. The method also includes processing a processing a PPG measurement signal based on the PPG waveform signal. The method additionally includes receiving a noise waveform signal from at least one of: a seat assembly sensor, a vehicle sensor, and a vehicle system. Additionally, the method includes processing a motion artifacts measurement signal based on the noise waveform signal. The method further includes processing a refined PPG signal to suppress the motion artifacts measurement signal from the PPG measurement signal.
The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that can be used for implementation. The examples are not intended to be limiting.
A “bus”, as used herein, refers to an interconnected architecture that is operably connected to other computer components inside a computer or between computers. The bus can transfer data between the computer components. The bus can be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus can also be a vehicle bus that interconnects components inside a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area network (CAN), Local Interconnect Network (LIN), among others.
“Computer communication”, as used herein, refers to a communication between two or more computing devices (e.g., computer, personal digital assistant, cellular telephone, network device) and can be, for example, a network transfer, a file transfer, an applet transfer, an email, a hypertext transfer protocol (HTTP) transfer, and so on. A computer communication can occur across, for example, a wireless system (e.g., IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local area network (LAN), a wide area network (WAN), a point-to-point system, a circuit switching system, a packet switching system, among others.
A “disk”, as used herein can be, for example, a magnetic disk drive, a solid state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, and/or a memory stick. Furthermore, the disk can be a CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and/or a digital video ROM drive (DVD ROM). The disk can store an operating system that controls or allocates resources of a computing device.
A “database”, as used herein can refer to table, a set of tables, a set of data stores and/or methods for accessing and/or manipulating those data stores. Some databases can be incorporated with a disk as defined above.
A “memory”, as used herein can include volatile memory and/or non-volatile memory. Non-volatile memory can include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory can include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM). The memory can store an operating system that controls or allocates resources of a computing device.
A “module”, as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another module, method, and/or system. A module may also include logic, a software controlled microprocessor, a discrete logic circuit, an analog circuit, a digital circuit, a programmed logic device, a memory device containing executing instructions, logic gates, a combination of gates, and/or other circuit components. Multiple modules may be combined into one module and single modules may be distributed among multiple modules.
An “operable connection”, or a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications can be sent and/or received. An operable connection can include a wireless interface, a physical interface, a data interface and/or an electrical interface.
A “processor”, as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor can include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that can be received, transmitted and/or detected. Generally, the processor can be a variety of various processors including multiple single and multicore processors and co-processors and other multiple single and multicore processor and co-processor architectures. The processor can include various modules to execute various functions.
A “portable device”, as used herein, is a computing device typically having a display screen with user input (e.g., touch, keyboard) and a processor for computing. Portable devices include, but are not limited to, handheld devices, mobile devices, smart phones, laptops, tablets and e-readers. In some embodiments, a “portable device” could refer to a remote device that includes a processor for computing and/or a communication interface for receiving and transmitting data remotely.
A “vehicle”, as used herein, refers to any moving vehicle that is capable of carrying one or more human occupants and is powered by any form of energy. The term “vehicle” includes, but is not limited to: cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, go-karts, amusement ride cars, rail transport, personal watercraft, and aircraft. In some cases, a motor vehicle includes one or more engines. Further, the term “vehicle” can refer to an electric vehicle (EV) that is capable of carrying one or more human occupants and is powered entirely or partially by one or more electric motors powered by an electric battery. The EV can include battery electric vehicles (BEV) and plug-in hybrid electric vehicles (PHEV). The term “vehicle” can also refer to an autonomous vehicle and/or self-driving vehicle powered by any form of energy. The autonomous vehicle may or may not carry one or more human occupants. Further, the term “vehicle” can include vehicles that are automated or non-automated with pre-determined paths or free-moving vehicles.
A “vehicle system”, as used herein can include, but is not limited to, any automatic or manual systems that can be used to enhance the vehicle, driving and/or safety. A “vehicle system”, as used herein can include, but is not limited to, any automatic or manual systems that can be used to enhance the vehicle, driving and/or safety. Exemplary vehicle systems include, but are not limited to: an electronic stability control system, an anti-lock brake system, a brake assist system, an automatic brake prefill system, a low speed follow system, a cruise control system, a collision warning system, a collision mitigation braking system, an auto cruise control system, a lane departure warning system, a blind spot indicator system, a lane keep assist system, a navigation system, a transmission system, brake pedal systems, an electronic power steering system, visual devices (e.g., camera systems, proximity sensor systems), a climate control system, an electronic pretensioning system, a monitoring system, a passenger detection system, a vehicle suspension system, a vehicle seat configuration system, a vehicle cabin lighting system, an audio system, a sensory system, among others.
A “wearable computing device”, as used herein can include, but is not limited to, a computing device component (e.g., a processor) with circuitry that can be worn by and/or in possession of a user. In other words, a wearable computing device is a computer that is subsumed into the personal space of a user. Wearable computing devices can include a display and can include various sensors for sensing and determining various parameters associated with a user. For example, location, motion, and biosignal (physiological) parameters, among others. Some wearable computing devices have user input and output functionality. Exemplary wearable computing devices can include, but are not limited to, watches, glasses, clothing, gloves, hats, shirts, jewelry, rings, earrings necklaces, armbands, shoes, earbuds, headphones and personal wellness devices.
Referring now to the drawings, wherein the showings are for purposes of illustrating one or more exemplary embodiments and not for purposes of limiting the same,
The system 100 can be implemented alone or in combination with a computing device 104 (e.g., controller, a head unit, etc.). The computing device 104 includes a processor 106, a memory 108, and a disk 110 which are operably connected for computer communication via a bus (not shown) and/or other wired and wireless technologies.
The computing device 104 can execute software serving to monitor and supervise various parameters of the engine (not shown) of the vehicle 102, as well as other components or systems of the vehicle 102. For example, the computing device 104 is capable of receiving signals from components of the vehicle 102 including sensors and devices. Signals output from sensors and devices can be sent to the computing device 104 and can be stored on the memory 108 and/or the disk 110. Further, the computing device 104 can facilitate information transfer between components of the vehicle 102 and/or control of the components of the vehicle 102. Both real-time and electronically stored signals can be processed by the processor 106 in accordance with software stored on the memory 108 and/or the disk 110.
In some embodiments, the computing device 104 can process signal output by sensors and devices into data formats that include values and levels. Such values and levels can include, but are not limited to, a numerical or other kind of value or level such as a percentage, a non-numerical value, a discrete state, a discrete value, a continuous value, among others. For example, in some cases, the value or level of X can be provided as a percentage between 0% and 100%. In other cases, the value or level of X can provided as a value in the range between 1 and 10. In still other cases, the value or level of X may not be a numerical value, but could be associated with a determined state, such a driving state.
In an exemplary embodiment, the computing device 104 also includes a PPG determinant module 112, a motion artifact determinant module 114, and a PPG signal filtering module 116. As will be described in more detail below, the PPG determinant module 112 communicates with one or more components of the vehicle 102 in order to process a PPG measurement signal (illustrated in
In the illustrated embodiment of
The sensor assembly 120 can include one or more sensor assembly sensors (not all individually shown) that can include contact sensors and/or non contact sensors. The plurality of sensor assembly sensors can include electric current/potential (e.g., proximity sensors, inductive, capacitive), ultrasonic (e.g., piezoelectric, electrostatic), vibration, optical, vision, photoelectric or oxygen sensors, among others. It is appreciated that the one or more sensor assembly sensors are operable to sense a measurement of data associated with the driver 118, the vehicle 102, the vehicle environment, one or more vehicle systems 124, and/or occupants of the vehicle 102, and can output one or more data signals indicating one or more measurements of data to the computing device 104. The computing device 104 can convert the data signals into other data formats in order to generate other data metrics and parameters such as values and levels, as described above.
In an exemplary embodiment, one or more specific sensor assembly sensors can include an optical sensor (illustrated in
As shown within the embodiment of
In an exemplary embodiment, the vehicle seat 122 can include a headrest 132, a seat back 134, and a seat base 136, although other configurations of the vehicle seat 122 are contemplated. As shown in the illustrated embodiment of
In one or more embodiments, the plurality of sensor assemblies 120 disposed within the vehicle seat 122 can be mechanically coupled to a common structural coupling material that allows for distribution of noise (e.g., engine noise, road noise, road vibration, driver movement, etc.) equally to all of the sensor assemblies 120. The mechanical coupling of the plurality of sensor assemblies 120 can reduce the effects of motion artifacts, since artifacts are distributed to impact each of the sensor assemblies 120 equally.
In some embodiments, the plurality of sensor assemblies 120 can be disposed in any portion of the vehicle 102. For example, a seat belt 138, floor board 140, steering wheel 142, dashboard 144, rear-view mirror 146, etc. can include one or more sensor assemblies 120 that include different types of sensor assembly sensors (e.g., optical, visual, capacitive sensors, electrodes, etc.). Moreover, in some cases, one or more sensor assemblies 120 can be included within one or more wearable devices (not shown) worn by the driver 118. The wearable devices can include, but are not limited to, wearable rings, watches, eye glasses, and articles of clothing. In other embodiments, one or more sensor assemblies 120 can be included with a portable device (not shown) located in proximity to the driver 118 such as a smart phone or similar device, or associated with an article of clothing worn by the driver 118.
In one embodiment, at least one of the plurality of sensor assemblies 120 may be disposed on the left front side of vehicle seat 122, for example, sensor 120a. This location may be best suited to receive the highest amount of contact with the driver's left leg 119. The left leg 119 is typically moved less than the right leg since the right leg is often used for the accelerator and brake pedals. The less movement of the leg, the less noise imparted to the sensor 120a. In one exemplary embodiment, the sensor 120a may be located between about 10 centimeters and about 30 centimeters from the front edge 122a of the vehicle seat 122 and between about 10 centimeters and about 30 centimeters from the left edge 122b of the vehicle seat 122. In another embodiment, the sensor 120a may be located between about 20 centimeters and about 25 centimeters from the front edge 122a and between about 15 centimeters and about 20 centimeters from the left edge 122b.
The vehicle 102 can additionally include the one or more vehicle sensors 148. The vehicle sensors 148 can include sensors associated with one or more vehicle systems 124 and/or other components of the vehicle 102. The vehicle sensors 148 can sense and measure a stimulus (e.g., a signal, a property, a measurement, or a quantity) associated with the vehicle 102 and/or one or more particular vehicle systems 124. In some embodiments, the vehicle sensors 148 can also sense and measure a stimulus associated with the driver 118, as described in more detail below. The vehicle sensors 148 can output one or more data signals representing one or more stimulus from the vehicle sensors 148. The vehicle sensors 148 can be disposed at various locations within a passenger cabin of the vehicle 102, including, but not limited to a head unit (not shown), the floor board 140, the dashboard 144, etc. Additionally, vehicle sensors 148 can be located at exterior portions of the vehicle 102 such as side view mirrors (not shown), door panels (not shown), front and rear bumpers (not shown), vehicle wheels (not shown), vehicle engine (not shown), etc.
In particular, the one or more vehicle sensors 148 (not all individually shown) can include, but are not limited to, an accelerometer, a magnetometer, a gyroscope, an ambient light sensor, a proximity sensor, a global positioning sensor system, a lateral acceleration sensor, and the like. Additionally, vehicle sensors 148 can include, but are not limited to a vehicle speed sensor, a steering angle sensor, accelerator pedal sensor, a brake sensor, a throttle position sensor, a wheel sensor, a camshaft sensor, an electronic parking sensor, among others. The vehicle sensors 148 can also include visual sensors in the form of cameras 150 mounted to the interior of the vehicle 102 and cameras, radar sensors, and laser sensors mounted to the exterior of the vehicle 102. Further, vehicle sensors 148 can include sensors located external to the vehicle 102 and accessed, for example, via a network. These sensors can include external cameras, radar and laser sensors on other vehicles in a vehicle-to-vehicle network, street cameras, surveillance cameras, blind spot indicator system, lane keep assist system, among others.
In an exemplary embodiment, one or more vehicle systems 124 (not all individually shown), discussed above can include, a data storage mechanism (e.g., memory) for storing data utilized by said vehicle systems 124, for example, sensitive data such as contact data, route data, password data, driver behavior profiles, driver physiological data profiles, among others.
The vehicle sensors 148 and vehicle systems 124 can provide vehicle data to the computing device 104 that can be utilized to determine various metrics with respect to the driver 118 and the vehicle 102. Specifically, vehicle data can include driver and/or vehicle conditions, states, statuses, behaviors, and associated information. As discussed in detail below, in an exemplary embodiment, the vehicle sensors 148 and vehicle systems 124 can capture one or more artifacts and can output respective signals that are associated with the movement of the driver 118 and/or the vehicle 102.
II. Processing a PPG Measurement Signal Associated with the Driver
Referring again to
In an exemplary embodiment, one or more optical sensors 202 are configured to emit a near-infrared or infrared LED light source in order to read and measure PPG signals of the driver 118 and/or occupants of the vehicle 102. The one or more optical sensors 202 can be configured to provide a volumetric measurement of the driver's blood volume and blood composition to determine metrics relating to the driver's blood oxygen levels that can have an effect on the driver's heart rate. Specifically, the one or more optical sensors 202 can utilize pulse oximetry that provides a reflected PPG measurement of the absorption of various wavelengths of infrared or near-infrared light by tissue within the driver's body. The one or more optical sensors 202 can each measure the amount of light that is reflected by the tissue in order to determine an amount of light that is absorbed by the driver's body. In other words, the optical sensors 202 can measure the pulsation change in the driver's blood volume with respect to oxygen saturation as more blood will absorb a higher amount of light, and less blood will absorb a lesser amount of light.
In an exemplary embodiment, each optical sensor 202 is located within one or more sensor assemblies 120 that are disposed within the (driver's) vehicle seat 122 and other locations within the vehicle cabin. The plurality of sensor assemblies 120 can be specifically positioned in areas that are situated to contact or not contact the surface of the driver's skin and/or clothing in order for the optical sensors 202 to clearly measure the driver's PPG signals. For example, one or more sensor assemblies 120 can be disposed at areas of the vehicle seat 122 and/or the vehicle 102 that are located near areas of the driver's body that have thinner layers of skin (e.g., ear lobes, finger tips). Additionally, one or more sensor assemblies 120 may be disposed at areas that are near areas of the driver's body that have thick blood vessels (e.g., back, thighs).
In one embodiment, the optical sensor 202 can include source circuitry 204 and detector circuitry 206. Specifically, the source circuitry 204 can include the near-infrared or infrared LED light source and/or a laser light source, etc. that emits light toward various areas of the driver's body as the driver 118 is seated within the vehicle seat 122. More particularly, the source circuitry 204 can include a plurality of LED and laser light sources (not shown) that are configured to provide various light colors and intensities. For example, the plurality of LED light sources can emit light with different wavelengths (e.g., 660-1600 nm) and different frequencies (430 THz-300 GHz) that illuminate through the driver's skin. The source circuitry 204 of each of the optical sensors 202 can be configured to increase or decrease the intensity of emitted light in order to emit a plurality of wavelengths based on the location of the optical sensor(s) 202 and the type of measurement that is being output by the optical sensor(s) 202. For example, with respect to the location of the optical sensors 202, the source circuitry 204 can utilize shorter wavelengths of light at areas where the optical sensors 202 are located that emit light where the driver 118 is expected to be wearing clothing (e.g., back, sides) as oppose to areas where light can typically be emitted directly to the driver's skin (e.g., neck, hands). In some embodiments, the source circuitry 204 can calibrate the intensity of various types of light by initially emitting less intense longer wavelengths that can be used to capture one or more measurements such as PPG signals on skin. The source circuitry 204 can then incrementally modify the intensity of the light wavelengths in order to penetrate through the driver's clothing to allow for the interrogation of blood vessels. In some embodiments, the intensity of light can also be influenced by the opacity of the driver's skin. For example, the effect of the driver's skin color can be utilized as a factor to modify the intensity of the LED light source during calibration by the source circuitry 204.
In one or more embodiments, the detector circuitry 206 can include a photodiode that can be configured to read an amount of scattered light that is transmitted through blood perfused tissue and measured on the opposite side of the tissue as the light provided by the source circuitry 204 and/or reflected back to the same side of the tissue as the light provided by the source circuitry 204. In some embodiments, the detector circuitry 206 can include one or more cameras (in lieu of or in addition to the photodiode) that are configured to capture images in order to analyze and provide measurements with respect to the transmitted light and/or the reflected light. In some configurations, the detector circuitry 206 can be positioned in order to measure one or more paths of light from the light source(s) of the source circuitry 204 that reflects back the source circuitry 204. Upon reading the amount of scattered light that is reflected back to the detector circuitry 206, each optical sensor 202 can provide a representation of its reading in the form of one or more PPG waveform signals. In alternate configurations, the detector circuitry 206 can be positioned in order to measure one or more paths of light from the light source(s) of the source circuitry 204 that pass through the tissue. Upon reading the amount of scattered light that passes through the tissue to the detector circuitry 206, each optical sensor 202 can provide a representation of its reading in the form of one or more PPG waveform signals.
In an exemplary embodiment, one or more optical sensors 202 can each output a respective PPG waveform signal (not shown) to the PPG determinant module 112 within a predetermined frequency of time (e.g., 10 ms). Each PPG waveform signal output by each of the optical sensors 202 can include a plurality of signal segments (not shown). The signal segments can include the measurement of PPG signals of the driver 118 along with artifacts caused by driver and vehicle movement that have an effect on the reading of the PPG signals by the optical sensor 202. Accordingly, in some situations, the PPG waveform signals output by the one or more of optical sensors 120 can be contaminated by the artifacts and can provide skewed PPG signals. Additionally, each PPG waveform signal can include signal features (not shown) such as signal peaks that can be further evaluated to determine PPG signal measurements and noise measurements. Additional signal features that can be evaluated include frequency, time duration, wave amplitude, among others. It is appreciated that other characteristics of the PPG waveform signal can also be identified as a signal feature.
Each PPG measurement signal 208 can include a plurality of signal segments 214 (only one signal segment 214 is shown). It is to be appreciated that one or more signal segments 214 can include any size and/or portion of the PPG measurement signal 208. The signal segments 214 can include the measurement of PPG signals of the driver 118 (compiled from one or more PPG waveform signals) along with artifacts caused by driver and vehicle movement. Additionally, each PPG measurement can include signal features such as signal peaks 216 that can be further evaluated to determine PPG signal measurements and noise measurements. Additional signal features that can be evaluated include frequency, time duration, wave amplitude, local maximum and minimum points, and inflection points (related to the second derivative of PPG) among others (not shown). It is appreciated that other characteristics of the PPG measurement signal 208 can also be identified as a signal feature.
Referring now to
At block 302, the method includes receiving one or more PPG waveform signals. In one embodiment, the PPG determinant module 112 communicates with one or more sensor assemblies 120 that are disposed at various areas of the vehicle seat 122 and/or the vehicle 102 in order to receive the one or more PPG waveform signals that are output by the one or more of optical sensors 202. Upon receipt of the one or more PPG waveform signals, the PPG determinant module 112 can store data pertaining to the signal features of the one or more PPG waveform signals onto the memory 108 and/or the disk 110 of the computing device 104 in order to be further evaluated.
At block 304, the method includes evaluating the one or more PPG waveform signals in order to determine the most consistent PPG waveform signals. In one embodiment, the PPG determinant module 112 accesses data pertaining to the signal features of the one or more PPG waveform signals from the memory 108 and/or the disk 110 and can determine one or more PPG waveform signals that include similar waveform patterns based on one or more signal features that are within one or more determined mean signal feature categories. More specifically, the PPG determinant module 112 can evaluate signal features of each of the PPG waveform signals including frequency, time duration, wave amplitude, measurement between signal peaks, local maximum and minimum points, and inflection points (related to the second derivative of PPG), etc. Upon evaluating the signal features, the PPG determinant module 112 can determine measurement values associated with each of the signal features (e.g., distance/time measurements) in order to compute mean values associated to each of the signal features.
Upon determining mean values associated to each of the signal features, the PPG determinant module 112 can determine one or more signal feature categories associated with each of the signal features. The one or more signal feature categories can include a categorization of the signal features of the one or more PPG waveform signals that are within a predetermined range from the mean signal feature value. For example, a mean signal feature category can include a peak signal measurement category that includes a predetermined range of values that are within a range from a mean value of a measurement between peak signals of each of the plurality of waveform signals. The PPG determinant module 112 can further evaluate PPG waveform signals that include signal features that fall within a predetermined amount of signal feature categories in order to determine the most consistent PPG waveform signals. It is to be appreciated that the PPG determinant module 112 can utilize various other methods to evaluate the one or more PPG waveform signals in order to determine the most consistent PPG waveform signals
By determining the most consistent PPG waveform signals, the PPG determinant module 112 can capture the most accurate representation of the driver's PPG signals that have been captured from the vehicle seat 122 and/or the vehicle 102. For example, the PPG determinant module 112 can evaluate ten PPG waveform signals provided by ten optical sensors 202 in order to determine three PPG waveform signals that include signal features that include values that fall within the predetermined amount of signal feature categories.
At block 306, the method includes discarding inconsistent PPG waveform signals. In one embodiment, the PPG determinant module 112 removes the inconsistent PPG waveform signals stored on the memory 108 and/or the disk 110 in order ensure such data does not influence the measurement of the driver's PPG signals. Specifically, the PPG determinant module 112 removes PPG waveform signals that are not determined to fall within the predetermined amount of signal feature categories. In particular, by discarding the inconsistent PPG signal waveforms 208 based on the evaluation conducted at block 304, the PPG determinant module 112 ensures that optical sensors 202 that are located in areas of the vehicle seat 122 and/or the vehicle 102 from which PPG readings may not have been completely or accurately captured are not accounted for when processing the PPG measurement signal 208.
At block 308, the method includes aggregating the most consistent PPG waveform signals and outputting a PPG measurement signal 208. In an exemplary embodiment, the PPG determinant module 112 aggregates the most consistent PPG waveform signals as determined at block 304 into the PPG measurement signal 208 that represents the most consistent measurement of the driver's PPG signals.
With reference again to
In another embodiment, the PPG determinant module 112 can determine the optimum PPG waveform signal by communicating with one or more pressure sensors included within the one or more sensor assemblies 120 in order to determine the pressure sensor that measures the highest pressure measurement caused by the driver 118 seated within the vehicle seat 122. In particular, one more sensor assemblies 120 that include the pressure sensor(s) can be disposed at particular locations of the vehicle seat 122 in which the change in pressure caused by the driver 118 is detected. The pressure sensor(s) can communicate pressure measurement data to the PPG determinant module 112 with respect to which pressure sensor measures the highest pressure measurement caused by the driver 118 in order to determine the optimum PPG waveform signal. In other words, the optimum PPG waveform signal is determined to be the PPG waveform signal that is most likely to be captured by an optical sensor 202 included within the sensor assembly 120 that is determined to be at a location of the vehicle seat 122 that is most utilized by the driver 118 (i.e., a location where a consistent/constant measurement can take place). Upon determining the optimum PPG waveform signal, the PPG determinant module 112 can convert the optimum PPG waveform signal into the PPG measurement signal 208.
In additional embodiments, the PPG determinant module 112 can process the PPG measurement signal 208 by separately evaluating PPG waveform signals that are captured from respective areas of the vehicle seat 122 in order to determine the optimum PPG waveform signal captured from each area of the vehicle seat 122. For example, the PPG determinant module 112 can determine four optimum PPG waveform signals captured by the optical sensors 202 located at the each of the seat back 134, and the seat base 136. Upon determining the optimum PPG waveform signals, the PPG determinant module 112 can aggregate the optimum PPG waveform signals captured from each area of the vehicle seat 122 into the PPG measurement signal 208. It is to be appreciated that the PPG determinant module 112 can utilize various other methods to process the PPG measurement signal 208 from one or more PPG waveform signals provided by the plurality of optical sensors 202.
As discussed above, the PPG waveform signals can be contaminated with motion artifacts caused by the driver 118 and/or the vehicle 102. Therefore, one or more signal segments 214 of the PPG measurement signal 208 that is processed by the PPG determinant module 112 also includes motion artifacts. Accordingly, the PPG determinant module 112 sends the processed PPG measurement signal 208 to the PPG signal filtering module 116 in order filter the motion artifacts and extract a refined PPG signal 212.
Apart from being configured to read and measure PPG signals of the driver 118, as discussed above, the plurality of optical sensors 202 can also emit a plurality of light sources at a plurality of frequencies in order to read signals that are associated with additional physiological data related to the driver 118 and/or occupants of the vehicle 102. For example, the optical sensors 202 can emit signals to determine physical information related to the driver 118 and/or occupants including biometric identification of the driver 118 and/or occupants based on sensing signals associated to physical characteristics (e.g., posture, position, movement) and biological characteristics (e.g., blood pressure, blood flow, oxygen content in the blood, etc.).
In one embodiment, the optical sensors 202 can emit the plurality of light sources at the plurality of frequencies in order to non invasively measure the driver's and/or occupant's blood alcohol levels. For example, the source circuitry 204 of the optical sensors 202 can emit light into the driver's skin. The optical sensors 202 can measure a tissue alcohol concentration based on the amount of light that is reflected back by the skin to the detector circuitry 206. Additionally, the optical sensors 202 can noninvasively monitor a condition of the driver 118 through the determination of biological signals, such as body-trunk plethysmograph and respiration that are detected from the driver's back from one or more optical sensors 202 included within sensor assemblies 120 disposed at the seat back 134 of the vehicle seat 122. In particular, one or more filtered signals can be evaluated to determine the driver's PPG signals that fall between normal and intoxicated states in order to determine driver intoxication. Such a technique is described by K. Murata, et. al in “Noninvasive Biological Sensor System for Detection of Drunk Driving,” IEEE Transactions on Information Technology in Biomedicine, Vol. 15, No, 1, 2011, the entire contents of which are incorporated by reference.
In an additional embodiment, the optical sensor 202 can emit various light sources in order to non invasively identify the driver 118 and/or occupant(s) of the vehicle 102 via biometric identification. For example, one or more physiological signals (e.g., PPG signals, etc.) can be measured by the optical sensors 202 in order to determine physiological signals that can be matched against a database of enrolled biometric templates. The database of enrolled biometric templates can include a biometric template from the driver 118 and/or occupants of the vehicle 102. Examples of biometric identification techniques are described within the parent application of the present application. Additional, exemplary biometric techniques are described by A. Reit Kaysao{hacek over (g)}lu et. al in “A novel feature ranking algorithm for biometric recognition with PPG signals,” Computer in Biology and Medicine, Vol. 49, 2014, 1-14, and as well as Agrafito et al. filed May 10, 2012, the entirety of both being hereby incorporated by reference.
III. Processing a Motion Artifacts Measurement Signal Associated with the Driver and the Vehicle
Referring back to
In an exemplary embodiment, data can be provided by various types of seat assembly sensors, vehicle sensors 148, and/or vehicle systems 124 in the form of one or more noise waveform signals (not shown) that are output within a predetermined frequency of time (e.g., 10 ms) to the motion artifact determinant module 114. As discussed below, the motion artifact determinant module 114 can convert one noise waveform signal or aggregate a plurality of noise waveform signals output by a plurality of sensor assembly sensors, vehicle sensors 148, and vehicle systems 124 into the motion artifacts measurement signal 210.
Some exemplary embodiments utilizing specific types of sensor assembly sensors, vehicle sensors 148, and vehicle systems 124 to determine motion artifacts associated with the driver 118 and the vehicle 102 will now be discussed below with reference to
In one or more embodiments, the optical sensor 202 can emit a blue/near UV visible light source that is configured to emit light towards the surface of the driver's skin. In particular, the blue/near UV visible light source can be configured to increase or decrease the intensity of emitted light in order to emit a plurality of wavelengths based on the location of the optical sensor 202 with respect to the driver 118. For example, the blue/near UV visible light source can emit a shorter wavelength of light at areas where the driver 118 is expected to be wearing clothing (e.g., back, sides) as oppose to areas where light can typically be transmitted directly to the driver's skin (e.g., neck, hands). The detector circuitry 206 of the optical sensor 202 can determine a reflectance of light that is absorbed by the surface of the driver's skin in order to determine the driver's movements within the vehicle seat 122. One or more optical sensors 202 can output one or more of noise waveform signals that are indicative of the driver's movements within the vehicle seat 122.
In one or more embodiments, seat assembly sensors including, but not limited to, accelerometers (not shown), gyroscopes (not shown), proximity detectors (not shown), magnetometers (not shown), etc., can be utilized alone or in combination to output one or more noise waveform signals to the motion artifact determinant module 114. In one embodiment, one or more accelerometers included within one or more sensor assemblies 120 can determine motion artifacts associated to the driver 118. In particular, the one or more accelerometers can communicate data that is indicative of the driver's movement within the vehicle seat 122. In some embodiments, the one or more accelerometers can be positioned atop the optical sensor 202, each oriented perpendicular to each other in order to capture motion artifacts that can most directly disturb the measurement of the driver's PPG signals by the optical sensor 202. Specifically, the one or more accelerometers can also include capacitive accelerometers that are used to determine and measure changes in a degree of movement of the driver 118 within the vehicle seat 122. For example, as the driver 118 accelerates and brakes the vehicle 102, the accelerometer(s) can determine a rate of change of the movement of the driver 118 as the driver 118 may shift back and forth within the vehicle seat 122.
Additionally, in some embodiments, the one or more pressure sensors included within the one or more sensor assemblies 120, discussed above, can be utilized to determine motion artifacts associated to the driver 118. In particular, one or more sensor assemblies 120 that include the pressure sensor(s) can be disposed at particular locations of the vehicle seat 122 in which the driver's movement can be determined based on the change in pressure as the driver 118 moves within the vehicle seat 122. The pressure sensor(s) can include hardware configured to determine movement of the driver 118 based on the change in pressure as detected by the shifting of the driver's weight within the vehicle seat 122. For example, one or more pressure sensors located within the seat back 134 of the vehicle seat 122 can determine and measure a change in movement of the driver 118 whose weight distributes away and toward the seat back 134 as the driver 118 moves forward or backward within the vehicle seat 122. The pressure sensor(s) can output one or more noise waveform signals to the motion artifact determinant module 114.
In an additional embodiment, one or more vibration sensors (not shown) included within the one or more sensor assemblies 120 and/or vehicle sensors 148 can also be utilized to determine motion artifacts associated to the driver 118 and/or the vehicle 102. The one or more vibration sensors can include piezoelectric sensors for detecting mechanical vibrations associated to the vehicle seat 122 and/or the vehicle 102. For example, the one or more vibration sensors can sense vibrations that can be attributed to the engine, the roadway on which the vehicle 102 is being driven, and/or the driver's movement within the vehicle seat 122. The vibration sensor(s) can output one or more noise waveform signals to the motion artifact determinant module 114.
In some embodiments, one or more proximity sensors (not shown) included within the one or more sensor assemblies 120 and/or vehicle sensors 148 can also be utilized to determine motion artifacts associated with the driver 118 and/or the vehicle 102. For example, a plurality of proximity sensors can be operable to determine the location of the driver 118 as he or she moves within the vehicle seat 122. Specifically, each proximity sensor can output a proximity measurement based on the proximity of the driver 118 to the respective sensor. The proximity sensor(s) can output one or more noise waveform signals to the motion artifact determinant module 114.
In an alternate embodiment, one or more wearable devices can be worn by the driver 118 that can also measure the driver's movement within the vehicle seat 122. The wearable device(s) can include one or more movement tracking sensors (e.g., accelerometer, gyroscope, etc.). The wearable device(s) can include device logic that is configured to track and measure movements of the driver 118 wearing the wearable device(s). The wearable device(s) can output one or more noise waveform signals to the motion artifact determinant module 114.
In some embodiments, the (internal) cameras 150 included as part of the vehicle sensors 148 can be located throughout the vehicle 102. Specifically, one or more cameras 150 can be positioned in various areas in front of, above, and/or around the vehicle seat 122 in order to capture real time images of the driver 118 as the vehicle 102 is being driven. The one or more cameras 150 can include hardware configured to interpret video or image data sensed by the camera(s) 150 to recognize any movement associated with the driver 118 within the vehicle seat 122. In one embodiment, the processor 106 can include camera logic that can evaluate image data output by one or more cameras 150 and can compile the image data to determine and measure changes in the movement of the driver 118 within the vehicle seat 122. The compiled data can be provided to the motion artifact determinant module 114 in the form of one or more noise waveform signals.
In one exemplary embodiment, the cameras 150, hardware and/or camera logic can evaluate the image data and/or video data to determine the position of the driver 118 within the vehicle seat 122. The position of the driver 118 can be provided to the PPG determinant module 112 in the form of position data. The position data can assist the PPG determinant module 112 in determining which sensor assembly 120 is likely to provide the optimum PPG waveform signal. In other words, the optimum PPG waveform signal is determined to be the PPG waveform signal that is most likely to be captured by an optical sensor 202 included within the sensor assembly 120 that is determined to be at a location of the vehicle seat 122 that is most utilized by the driver 118 (i.e., a location where a consistent/constant measurement can take place). Upon determining the optimum PPG waveform signal, the PPG determinant module 112 can convert the optimum PPG waveform signal into the PPG measurement signal 208. For example, if the driver 118 is leaning to the left then the sensor assemblies 120 and the optical sensors 202 on the left side of the vehicle seat 122 are likely to provide the optimum PPG waveform signal. The PPG determinant module 112 can then ignore the sensor assemblies on the right side of the vehicle seat 122.
In another exemplary embodiment, the cameras 150, hardware and/or camera logic can evaluate the image data and/or video data to determine the characteristics of the clothing worn by the driver 118. The characteristics can be provided to the source circuitry 204 in the form of characteristic data. The source circuitry 204 of each of the optical sensors 202 can be configured to increase or decrease the intensity of emitted light in order to emit a plurality of wavelengths based on the characteristic data. For example, the source circuitry 204 can utilize shorter wavelengths of light at areas where the optical sensors 202 are located where the driver 118 is wearing darker colored or thicker clothing, as opposed to areas where the driver 118 is wearing lighter colored or thinner clothing. In some embodiments, the source circuitry 204 can be calibrated to the characteristics of various types of clothing. The intensity of the emitted light can then be selected based on the characteristic data received from the cameras 150, hardware and/or camera logic and the calibration. In some embodiments, the intensity of light can also be influenced by the opacity of the driver's skin. For example, the effect of the driver's skin color in combination with the characteristic data of the clothing can be utilized as a factor to modify the intensity of the LED light source by the source circuitry 204.
Additional vehicle sensors 148 can be used in conjunction with one another to provide data regarding body movement of the vehicle 102 as the vehicle 102 is being driven on a roadway. For example, vehicle sensors 148 disposed at each wheel of the vehicle 102 can measure a ride level of the vehicle 102 and one or more accelerometers included as part of the vehicle sensors 148 can measure vertical body acceleration of the vehicle 102 in order to accurately measure road noise. The vehicle sensors 148 can also be used to determine the steering angle, roll, pitch, lateral acceleration and yaw of the vehicle 102 as an indication of the vehicle 102 reacting to turns, acceleration, braking, and road noise that can affect the capturing of one or more PPG signals by the optical sensors 202. For example, one or more vehicle sensors 148 that include the steering sensor, gyroscope, lateral acceleration sensors, acceleration pedal sensors, brake sensors, wheel speed sensors, etc. can be utilized alone or in combination with one another to output noise waveform signals to the motion artifact determinant module 114.
In some embodiments, one or more vehicle systems 124 can provide one or more noise waveform signals to the motion artifact determinant module 114. For example, the electronic stability control system (not shown) can monitor the yaw rate of the vehicle 102 and can output movement data in the form of a noise waveform signal.
In an exemplary embodiment, upon receiving one or more noise waveform signals from the sensor assembly sensors, vehicle sensors 148, and/or vehicle systems 124, the motion artifact determinant module 114 can process the one or more noise waveform signals into the motion artifacts measurement signal 210. Processing completed by the motion artifact determinant module 114 generally includes converting one noise waveform signal or aggregating the plurality of noise waveform signals into the motion artifacts measurement signal 210. Processing can include amplification, mixing, and filtering of one or more noise waveform signals, as well as other signal processing techniques known in the art. The motion artifacts measurement signal 210 can include signal segments (not shown) and signal features (not shown) similar to the PPG measurement signal 208 that are representative of motion artifacts associated with the driver 118 and the vehicle 102.
In some embodiments, upon receiving the plurality of noise waveform signals from the sensor assembly sensors, vehicle sensors 148, and/or vehicle systems 124, the motion artifact determinant module 114 can assign a weight to each of the received noise waveform signals. The weight assigned to each of the noise waveform signals can be associated to a determined level of impact that each sensed noise waveform signal has to the motion artifacts that are contaminating the PPG measurement signal 208. The level of impact can be determined by evaluating one or more noise waveform signals that are output by one or more sensor assembly sensors, vehicle sensors 148, and/or vehicle systems 124. Upon evaluating the one or more noise waveform signals, the motion artifact determinant module 114 can apply a respective weight to each noise waveform signal based on the type of sensors/systems that output the noise waveform signal, the location of the sensors/systems that output the noise waveform signal, the type of data that the sensors/systems that output the noise waveform signal are capturing/measuring, and the impact of the noise waveform signal on the capturing of PPG waveform signals output by one or more optical sensors 202. For example, one or more noise waveform signals that are output by biological sensors located within the one or more sensor assemblies 120 within the vehicle seat 122 can be determined to have a higher level impact and can therefore be assigned a higher weight than other noise waveform signals provided by non-biological sensors located at other areas of the vehicle 102. Upon determining the weight of each of the noise waveform signals, the motion artifact determinant module 114 can aggregate the plurality of noise waveform signals to each impact the processed motion artifacts measurement signal 210 based on an assigned weight.
In yet another embodiment, upon receiving a plurality of noise waveform signals, the motion artifact determinant module 114 can separately evaluate noise waveform signals that are attributed to movement of the driver 118 apart from noise waveform signals that are attributed to the movement of vehicle 102 itself to determine a driver motion artifacts measurement signal (not shown) and a vehicle motion artifacts measurement signal (not shown). Specifically, the motion artifact determinant module 114 can aggregate the noise waveform signals that are attributed to the movement of the driver 118 into the driver motion artifacts measurement signal. Similarly, the motion artifact determinant module 114 can aggregate the noise waveform signals that are attributed to the movement of the vehicle 102 in the vehicle motion artifacts measurement signal.
It is to be appreciated that additional methods can be utilized by the motion artifact determinant module 114 in order to process the motion artifacts measurement signal 210. In an exemplary embodiment, upon determining the motion artifacts measurement signal 210, the motion artifact determinant module 114 can send the processed motion artifacts measurement signal 210 to the PPG signal filtering module 116 in order filter the PPG measurement signal 208 and process the refined PPG signal 212.
The PPG signal filtering module 116 can utilize one or more techniques to decontaminate the PPG measurement signal 208 processed by the PPG determinant module 112 from motion artifacts associated with the driver 118 and the vehicle 102 itself. Processing completed by the PPG signal filtering module 116 can include amplification, mixing, and filtering of the PPG measurement signal 208 and the motion artifacts measurement signal 210, as well as other signal processing techniques known in the art, some of which are discussed below.
In an exemplary embodiment, upon receiving the motion artifacts measurement signal 210, the PPG signal filtering module 116 can apply a filter on the PPG measurement signal 208 output by the PPG determinant module 112 in order to process the refined PPG signal 212. The refined PPG signal 212 is indicative of a measurement of the driver's blood volume and blood composition that is decontaminated from the one or more motion artifacts associated with the driver 118 and the vehicle 102.
With reference to
Upon determining signal segments 214 of the PPG measurement signal 208 that includes the motion artifacts, the PPG signal filtering module 116 can filter one or more signal segments 214 of the PPG measurement signal 208 that are attributed to the motion artifacts as represented by the motion artifacts measurement signal 210 in order to extract one or more signal segments 214 that are representative of the refined PPG signal 212. Upon extraction of the one or more signal segments 214 that are representative of the refined PPG signal 212, the PPG signal filtering module 116 can process and output the refined PPG signal 212 (this process is best illustrated by the best example as presented in
In an alternate embodiment, as discussed above, the motion artifact determinant module 114 can provide separate motion artifacts measurement signals in the form of the driver motion artifacts measurement signal and the vehicle motion artifacts measurement signal that are each individually associated with the driver 118 and the vehicle 102. Upon receiving the driver motion artifacts measurement signal and the vehicle motion artifacts signal, the PPG signal filtering module 116 can apply multiple levels of filtering to the PPG measurement signal 208 in order to filter one or more signal segments 214 of the PPG measurement signal 208 that are attributed to the driver motion artifacts measurement signal separately from one or more signal segments 214 that are attributed to the vehicle motion artifacts measurement signal. It is to be appreciated that additional methods can be utilized by the PPG signal filtering module 116 in order to process the refined motion artifacts signal.
In one or more embodiments, the refined PPG signal 212 can be provided by the PPG signal filtering module 116 to the processor 106 of the computing device 104 in order to be utilized for biometric interpretation by one or more vehicle systems 124. Specifically, the computing device 104 can evaluate the refined PPG signal 212 in order to determine biometric data that is associated with the driver 118. The computing device 104 can further utilize such biometric data in order to determine one or more driver states that can be utilized to control vehicle HMI output, vehicle systems 124, and/or autonomous driving of the vehicle 102. The “driver state,” as used herein, refers to a measurement of a state of the biological system of the driver. The driver state can be one or more of alert, vigilant, drowsy, inattentive, distracted, stressed, intoxicated, other generally impaired states, other emotional states and/or general health states, among others.
In alternate embodiments, alternative methods can be utilized by the system 100 in order to process the refined PPG signal 212. In one embodiment, a method of signal modulation and demodulation can be utilized in order to decontaminate the PPG measurement signal 208 from motion artifacts. More specifically, a carrier frequency can be generated at a calculated frequency of a harmonic of an identified noise component (e.g., noise waveform signal) that can be utilized to generate and demodulate an amplitude modulated signal (e.g., PPG measurement signal 208) in order to reduce signal interference. Such a method is described in Anderson et al., U.S. Pat. No. 7,623,990, filed Nov. 3, 2005, the entirety of which is hereby incorporated by reference.
In another embodiment, instead of filtering the PPG measurement signal 208 from the motion artifacts measurement signal 210 in order to process the refined PPG signal 212, the system 100 can utilize methods for minimizing the influence of motion artifacts during the measurement of PPG signals. For instance, a method of using amplitude modulated light can be used to encode the PPG signal (e.g., PPG waveform signals, PPG measurement signal 208) in order to distinguish the measured PPG signal from noise. In one embodiment, the source circuitry 204 of the optical sensor 202 can generate an amplitude modulated code sequence in the form of a Barker binary code sequence that is received by the detector circuitry 206. The Barker binary code sequence can be used to detect the segment of the signal associated to noise based on determining an activated light source and deactivated light source in order to differentiate the PPG signal from noise. Such a technique is described by R. Gircys, et. al in “Movement Artefact Resistant Photoplethysmographic Probe,” Elektronika IR Elektrontechnika, ISSN 1392-1215, Vol. 20, 2014, the entire contents of which are incorporated by reference.
Referring now to
At block 404, the method includes processing a PPG measurement signal 208 based on the PPG waveform signal. In an exemplary embodiment, upon receiving one or more PPG waveform signals, the PPG determinant module 112 can convert one PPG waveform signal (e.g., the optimum PPG waveform signal) or aggregate the plurality of PPG waveform signals in order to process the PPG measurement signal 208. At block 406, the method includes receiving a noise waveform signal from at least one of: a seat assembly sensor, a vehicle sensor, and a vehicle system. As discussed in detail above, one or more seat assembly sensors, vehicle sensors 148, and/or vehicle systems 124 can output one or more noise waveform signals to the motion artifact determinant module 114.
At block 408, the method includes processing a motion artifacts measurement signal 210 based on the noise waveform signal. In an exemplary embodiment, upon receiving one or more noise waveform signals, the motion artifact determinant module 114 can convert one noise waveform signal or aggregate the plurality of noise waveform signals in order to process the motion artifacts measurement signal 210.
At block 410, the method includes processing a refined PPG signal 212 to suppress the motion artifacts measurement signal 210 from the PPG measurement signal 208. As discussed, in one embodiment, the PPG signal filtering module 116 can receive the PPG measurement signal 208 and the motion artifacts signal respectively from the PPG determinant module 112 and the motion artifact determinant module 114. The PPG signal filtering module 116 can filter the PPG measurement signal 208 by determining one or more segments of the PPG measurement signal 208 that include data that is attributed to the motion artifacts measurement signal 210. The PPG signal filtering module 116 can filter one or more signal segments 214 of the PPG measurement signal 208 that are attributed to the motion artifacts as represented by the motion artifacts measurement signal 210, in order to process the refined PPG signal 212. The refined PPG signal 212 is indicative of a measurement of the driver's blood volume and blood composition that is decontaminated from the one or more motion artifacts associated with the driver 118 and the vehicle 102.
The embodiments discussed herein may also be described and implemented in the context of non-transitory computer-readable storage medium storing computer-executable instructions. Non-transitory computer-readable storage media includes computer storage media and communication media. For example, flash memory drives, digital versatile discs (DVDs), compact discs (CDs), floppy disks, and tape cassettes. Non-transitory computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, modules or other data. Non-transitory computer readable storage media excludes transitory and propagated data signals.
It will be appreciated that various embodiments of the above-disclosed and other features and functions, or alternatives or varieties thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations, or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.
The application is a continuation-in-part of, and claims priority to, U.S. application Ser. No. 14/697,593, filed on Apr. 27, 2015, the entire application of which is incorporated herein by reference; U.S. application Ser. No. 14/697,593 is a continuation-in-part of, and claims priority to, U.S. application Ser. No. 13/858,038, filed on Apr. 6, 2013, now issued as U.S. Pat. No. 9,272,689, the entire application of which is also incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14697593 | Apr 2015 | US |
| Child | 15235808 | US | |
| Parent | 13858038 | Apr 2013 | US |
| Child | 14697593 | US |
