Patent Grant
10726846
The present disclosure relates, generally, to systems and methods for providing a virtual health assistant. More particularly, the present disclosure relates to improved systems and methods for providing a virtual health assistant that can adaptively monitor a person, for example, a patient being monitored for medical reasons.
One exemplary target population who may benefit from the disclosed system and methods is the elderly population (i.e., individuals aged 65 and over). The population aged 65 and over is the fastest growing in the US. It is expected to enlarge from 50 million in 2016 to 56 million by 2020 and 73 million by 2030. More than a million new caregivers are required to support this population in the next eight years. Although home-based primary elderly care has shown promise as a cost saving measure in pilot programs, caregivers need technological solutions that would enable them to prioritize visits in order to enhance efficiency and free up bandwidth to do more.
Disclosed are example systems and methods that can serve as virtual caregiver assistants and physician extenders that help ensure the health and security of persons (such as seniors and the infirm) who may spend a significant amount of time in their homes (or other settings, such as healthcare facilities). The sounds of a person and the sounds in a person's environment can be monitored in real-time, as part of “passive” monitoring, to detect anomalous or unexpected sound signals that could indicate threatening circumstances or high-risk situations. If such an sound signal is detected, “active” monitoring may engage to implement automated solutions in a closed loop fashion through, for example, structured bidirectional interactions with the person, assessment of pathologic speech/response patterns, analysis of data about the person and his/her environment (current and historical), etc., to facilitate timely medical evaluations for acute and chronic conditions, and otherwise to better monitor and enhance the person's well-being. By capturing and analyzing sounds, a user can be monitored continuously and in a greater number of situations (even while, for example, the user is in a lavatory where he or she may not want a person to follow, or a camera to be installed). The disclosed systems and methods can assist health professionals and family members take care of persons in their homes or elsewhere by better monitoring their environment while respecting their privacy.
In an illustrative embodiment a system for providing health information or assistance includes at least one electronic device configured to engage in verbal communication with at least one user based upon at least one type of signal selected from the group consisting of 1) sound signals from the at least one user's environment, 2) non-verbal signals from the at least one user, 3) verbal signals from the user; and 4) any combination of (1), (2), and (3). The electronic device may include an input configured to receive the sound signals from the at least one user's environment, the non-verbal signals from the at least one user, and the verbal signals for the at least one user and a communication module configured to send information related to the at least one user and the at least one user's environment, including the sound signals, the non-verbal signals, and the verbal signals, to a remote device, and to subsequently receive condition signals from the remote device. The electronic device may further include a processing module configured to receive the condition signals and to cause the electronic device to operate in a passive monitoring mode or an active engagement and monitoring mode, the active engagement and monitoring mode including, but not limited to, verbal communication with the at least one user and an output configured to engage the at least one user in specific types of verbal communication selected from a group consisting of: 1) asking the at least one user a question; 2) providing related suggestions to the user; and 3) provide information to the user. The system may further include a remote that may include a remote communication module configured to receive the information related to the sound signals, the non-verbal signals, and the verbal signals from the at least one electronic device, and to subsequently send the condition signals to the at least one electronic device and a rules module configured to determine, based on the sound signals, the non-verbal signals, and the verbal signals, a responsive action, wherein the responsive action is communicated from the remote device to the at least one electronic device in the form of condition signals
In some embodiments, at least one of the electronic device and the remote device further includes a speaker identification module configured to identify and authenticate the at least one user.
In some embodiments, the remote device includes a temporal database configured to receive and store the information related to the at least one user and the at least one user's environment and a decisions database configured to receive and store decisions from at least one health care provider.
In some embodiments, the remote device includes a training module configured to selectively retrieve the information related to the at least one user and the at least one user's environment from the temporal database, selectively retrieve the decisions from the at least one health care provider from the decisions database, and adapt the rules module using machine learning based on the retrieved information and decisions to increase accuracy of the responsive action.
In some embodiments, the at least one electric device includes a plurality of electronic devices, the at least one user includes a plurality of users using the plurality of electronic devices, and wherein the training module is further configured to selectively retrieve information about each of the plurality of users and each of a plurality of users' environments, selectively retrieve decisions from at least one health care provider, and adapt the rules module using machine learning based on the retrieved information and decisions to increase accuracy of the responsive action.
In some embodiments, the communication module of the at least one electronic device is configured to retrieve the information related to the at least one user, the at least one user's environment, and the decisions from the at least one health care provider, and the at least one electronic device includes a report module configured to produce a report to the at least one user regarding the information related to the at least one user, the at least one user's environment, and the decisions from the at least one health care provider.
In some embodiments, when operating in the passive monitoring mode, t a remote device processing module is configured to analyze the sound signals to extract passive user information, determine, from the passive user information, whether an active trigger event has occurred, and upon determining that the active trigger event has occurred, cause the electronic device transition to the active engagement and monitoring mode.
In some embodiments, upon transitioning to the active engagement and monitoring mode, the electronic device processing module is configured to engage the at least one user via the verbal communication through the output and the remote device processing module is configured to verify and augment the passive user information through the communication with the user.
In some embodiments, upon transitioning to the active engagement and monitoring mode, the remote device processing module is configured to analyze the sound signal from the input to extract active user information associated with a communication with the at least one user and determine, from both the passive user information and the active user information, an active engagement response.
In some embodiments, upon transitioning to the active engagement and monitoring mode, the remote device processing module is configured to determine, via the verbal communication, whether the active engagement response has been completed and upon determining that the active engagement response has been completed, cause the electronic device to transition to the passive monitoring mode.
In some embodiments, the rules module adapts the verbal communication through the output in real time based on one of the passive user information and the active user information.
In some embodiments, the adaptation of the verbal communication includes changes in questions asked or the way the questions are asked to the at least one user based on changes in a condition of the at least one user detected in one of the passive user information and the active user information.
In some embodiments, the non-verbal signals are selected from the group consisting of: pitch, speed, tone, volume of voice, intonation, inflection, or other sounds that do not include words.
In some embodiments, the step of analyzing the sound signals includes at least one of analyzing speech pattern, intonation, data about the at least one user, data about the at least one user's environment, sound signals in the at least one user's environment, sounds created by the at least one user, and does not include analyzing key words.
In some embodiments, while engaging the at least one user via the verbal communication regarding the active trigger event, the rules module adapts future engagements regarding related trigger events based on information acquired during the verbal communication regarding the active trigger event.
In some embodiments, while engaging the at least one user via the verbal communication, the processing module is configured to analyze both verbal signals and non-verbal signals.
In some embodiments, each of the plurality of electronic devices is configured to produce a report and, subsequent to the training module adapting the rules module, the at least one health care provider makes a decision based on the report.
In some embodiments, the machine learning performed by the training module uses a model based on information retrieved from the temporal database and decisions retrieved from the decisions database, and the model is subsequently trained based on input from the at least one health care provider.
In some embodiments, the model is based on information from the multiple users, thereby providing a larger data set.
In some embodiments, each of the plurality of electronic devices is configured to produce a report and each of the plurality of users inputs information into a corresponding one of the plurality of electronic devices and receives output information from the corresponding one of the plurality of electronic devices.
In some embodiments, an alert is sent to a caretaker or health care professional that a responsive action was sent to the at least one user.
In another illustrative embodiment, an electronic device is provided for providing health information or assistance, wherein the electronic device is configured to engage in verbal communication with a user. The electronic device includes an input configured to receive at least one type of signal selected from the group consisting of: (1) sound signals from at least one user's environment, (2) non-verbal signals for the at least one user, 3) verbal signals from the user, and 4) any combination of (1), (2), and (3). The electronic device may further include a communication module configured to send information related to the at least one user and the at least one user's environment to a remote device, including the sound signals, non-verbal signals, and verbal signals, the remote device being configured to analyze a condition of the at least one user and subsequently communicate condition signals to the electronic device. The electronic may still further include a processing module configured to receive the condition signals and to cause the electronic device to engage in a passive monitoring mode or an active engagement and monitoring mode, the active engagement and monitoring mode including, but not limited to, verbal communication with the at least one user and an output configured to engage the at least one user in specific types of verbal communication selected from a group consisting of: 1) asking the at least one user a question; 2) providing related suggestions to the user; and 3) provide information to the at least one user.
In some embodiments, the electronic device includes a plurality of electronic devices, the at least one user includes a plurality of users using the plurality of electronic devices, and wherein the communication module of each electronic devices is configured to send information related to a respective user and the user's environment to the remote device to analyze a condition of the respective user.
In some embodiments, the communication module of the electronic device is further configured to retrieve the information related to the at least one user, the at least one user's environment, and decisions from at least one health care provider, and the at least one electronic device includes a report module configured to produce a report to the at least one user regarding the information related to the at least one user, the at least one user's environment, and the decisions from the at least one health care provider.
In some embodiments, the processing module is further configured to, upon transitioning to the active engagement and monitoring mode, engage the at least one user via communication through the output and the remote device processing module is configured to analyze the sound signal from the input to extract active user information associated with a communication with the at least one user and determine, from both the passive user information and the active user information, an active engagement response.
In some embodiments, the non-verbal signals are selected from the group consisting of: pitch, speed, tone, volume of voice, intonation, inflection, or other sounds that do not include words.
In a further illustrative embodiment, a method of providing health information or assistance includes the steps of receiving, at an input of an electronic device, at least one type of signal selected from the group consisting of: 1) sound signals from the at least one user's environment, 2) non-verbal signals from the at least one user, 3) verbal signals from the user; and 4) any combination of (1), (2), and (3) and communicating information related to the at least one user and the at least one user's environment to a remote device, the remote device being configured to analyze a condition of the at least one user and subsequently communicate condition signals to the electronic device. The method further includes the step of receiving, at the electronic device, a condition signal from the remote device to engage in a passive monitoring mode or an active engagement and monitoring mode, the active engagement and monitoring mode including, but not limited to, verbal communication with the at least one user selected from a group consisting of: 1) asking the at least one user a question; 2) providing related suggestions to the user; and 3) provide information to the at least one user.
In some embodiments, in the passive monitoring mode, the method includes the steps of analyzing the information related to the at least one user and the at least one user's environment, extracting passive user information, determining, from the passive user information, whether a trigger event has occurred, and upon determining that the trigger event has occurred, transitioning the electronic device to the active engagement and monitoring mode.
In some embodiments, in the active engagement and monitoring mode, the method includes the steps of engaging the at least one user via verbal communication through an output of the electronic device, analyzing, at the remote device, the information related to the at least one user and the at least one user's environment to extract active user information associated with the verbal communication with the at least one user, and determining, at the remote device, from both the passive user information and the active user information, an active engagement response.
In some embodiments, the method includes the steps of determining, via verbal communication, whether the active engagement response has been completed and upon determining that the active engagement response has been completed, transitioning to the passive monitoring mode.
In some embodiments, the method includes the step of adapting the verbal communication with the at least one user in real time based on one of the passive user information and the active user information.
In some embodiments, the step of adapting the verbal communication includes the step of changing questions asked of the at least one user based on changes in a condition of the at least one user detected in one of the passive user information and the active user information.
In some embodiments, the step of analyzing the information related to the at least one user and the at least one user's environment includes analyzing, at the remote device, speech pattern, intonation, data about the at least one user, data about the at least one user's environment, sound signals in the at least one user's environment, or sounds created by the at least one user, and does not include analyzing key words.
In some embodiments, the method includes the steps of receiving and storing the information related to the at least one user and the at least one user's environment and receiving and storing decisions from at least one health care provider.
In some embodiments, the method includes the steps of retrieving information related to the at least one user and the at least one user's environment from a temporal database, retrieving the decisions from the at least one health care provider from a decisions database, and adapting rules using machine learning based on the retrieved information and decisions to increase accuracy of an action to be taken with respect to the at least one user.
In some embodiments, the at least one electronic device includes a plurality of electronic devices, the at least one user comprises a plurality of users using the plurality of electronic devices, and the method includes the steps of retrieving information about each of the plurality of users and each of a plurality of users' environments, retrieving decisions from the at least one health care provider, and adapting rules using machine learning based on the retrieved information and decisions to increase accuracy of the action to be taken.
In some embodiments, the method includes the step of producing a report regarding the information related to the at least one user, the at least one user's environment, and the decisions from the at least one health care provider.
In yet another illustrative embodiment, a non-transitory computer-readable medium tangibly comprising computer program instructions in which the instructions, when executed by a process, cause the processor to receive input in the form of at least one type of signal selected from the group consisting of: 1) sound signals from the at least one user's environment, 2) non-verbal signals from the at least one user, 3) verbal signals from the user; and 4) any combination of (1), (2), and (3). The instructions further cause the processor to send information related to the at least user and the at least one user's environment, including the sound signals, the non-verbal signals, and the verbal signals, to a remote device, the remote device being configured to analyze a condition of the at least one user and subsequently communicate condition signals to the processor and receive the condition signals at the processor. The instructions further cause the processor to cause the processor to engage in a passive monitoring mode or an active engagement and monitoring mode, the active engagement and monitoring mode including, but not limited to, verbal communication with the at least user, wherein the electronic device is configured to transition from the passive monitoring mode to the active engagement and monitoring mode when the condition signals indicate that the at least user is in need of assistance and is further configured to transition from the active engagement and monitoring mode to the passive monitoring mode when the condition signals indicate that the at least user is no longer in need of assistance.
The foregoing and other aspects and advantages of the present disclosure will appear from the following description. In the description, reference is made to the accompanying drawings that form a part hereof, and in which there is shown by way of illustration one or more exemplary versions. These versions do not necessarily represent the full scope of the invention.
This disclosure is illustrated by way of example and not by way of limitation in the accompanying figures. The figures may, alone or in combination, illustrate one or more embodiments of the disclosure. Elements illustrated in the figures are not necessarily drawn to scale. Reference labels may be repeated among the figures to indicate corresponding or analogous elements.
The foregoing and other aspects and advantages of the invention will appear from the following description. In the description, reference is made to the accompanying drawings that form a part hereof, and in which there is shown by way of illustration preferred embodiments of the invention. Such embodiments do not necessarily represent the full scope of the invention, however, and reference is made therefore to the claims and herein for interpreting the scope of the invention.
Disclosed are example systems and methods that can use speech and sound analysis as well as virtual personal assistant capabilities to assist in monitoring and promoting the health of occupants of a home, healthcare facility, or other location. Referring to
In general, the system 100 may include one or more inputs 102, processors 104, memories 106, and outputs 108, and may be configured to carry out steps for adaptively monitoring and analyzing sound signals created by a user or sound signals occurring in a user's environment or both, in accordance with aspects of the present disclosure. Sound signal, as used herein, refers to any sound at any frequency, including audio signals. Such sounds may include, but are not limited to, speech, snoring, coughing, sneezing, and various other sound signals, as described below.
The system 100 may include, access, or communicate with one or more databases 110, storage servers 112, auditory systems 114, user interfaces 118, and, in some implementations, an imaging system 120, by way of a wired or wireless connection to the inputs 102. In various implementations, the system 100 may include any computing device, electronic device, apparatus or system configured for carrying out instructions and providing input/output capabilities, and may operate as part of, or in collaboration with other computing devices and sensors/detectors (local and remote). In certain instances, the system 100 may be a portable or mobile device, such as a smartphone, laptop, tablet, and the like. In alternative embodiments, it may be a standalone appliance with these capabilities. In this regard, the system 100 may be a system that is designed to integrate a variety of software and hardware capabilities and functionalities, and/or may be capable of operating autonomously. In addition, in various configurations, the components illustrated in
The input 102 may include any one or more different input elements, such as a mouse, keyboard, touchpad, touch screen, buttons, microphone, a personal tracking device (e.g., a Fitbit or other tracking device), other electronic inputs, and the like, for receiving various sound signals, selections, and operational instructions from a user through touch, movement, speech, etc. The input 102 may also include various drives and receptacles, such as flash-drives, USB drives, CD/DVD drives, and other computer-readable medium receptacles, for receiving various data and information. To this end, input 102 may also include various communication ports and modules, such as Ethernet, Bluetooth, or WiFi, for exchanging data and information with these, and other external computers, systems, devices, machines, mainframes, servers or networks.
In addition to being configured to carry out various steps for operating the system 100, the processor 104 may be configured to execute instructions, stored in the memory 106 in a non-transitory computer-readable media 116. The instructions executable by the processor 104 may correspond to various virtual health assistant (VHA) systems, examples of which will be described below. Although the non-transitory computer-readable media 116 is shown in
In some aspects, the processor 104 may be configured to receive and process a sound signal and/or a picture or video signal to generate a variety of information, including patient responses, signals from personal tracking device, user events, hypothesis confidence levels, as well as other data. In some aspects, the processor 104 may access information and data, including sound signals and video signals, stored in or emitted by the imaging system 120, the user interface 118, the auditory system 114, the storage server(s) 112, the database(s) 110, or other data storage locations using the input 102. In some aspects, the auditory system 114 may acquire a sound signal continuously using, for example, a microphone or other audio or sound recording device to continuously or periodically record a user or the user's environment 122. In some aspects, the imaging system 120 may acquire either a single image or a continuous video signal using, for example, a camera, an infrared scanning system, or any other image capturing or video recording device that can be used to periodically image and/or scan and/or continuously record the user 122.
The system 100 may be comprised of only local electronic devices, one remote device and multiple local electronic devices, multiple remote devices and multiple local electronic devices, or any combination thereof.
The system 100 may include one or more local electronic devices that include any one or more of the components depicted in
The system 100 may include one or more remote devices, which may include any one or more of the components depicted in
As should be clear, while only one of each of the components in
Embodiments of the present invention may be described with reference to flowchart illustrations of methods and systems according to embodiments of the technology, and/or algorithms, formulae, or other computational depictions, which may also be implemented as computer program products. In this regard, each block or step of a flowchart, and combinations of blocks (and/or steps) in a flowchart, algorithm, formula, or computational depiction can be implemented by various means, such as hardware, firmware, and/or software including one or more computer program instructions embodied in computer-readable program code logic. As will be appreciated, any such computer program instructions may be loaded onto the system 100 as described with reference to
Accordingly, blocks of the flowcharts, algorithms, formulae, or computational depictions support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and computer program instructions, such as instructions embodied in computer-readable program code logic means, for performing the specified functions. It will also be understood that each block of the flowchart illustrations, algorithms, formulae, or computational depictions and combinations thereof described herein, can be implemented by special purpose hardware-based computer systems which perform the specified functions or steps, or combinations of special purpose hardware and computer-readable program code logic means.
Furthermore, these computer program instructions, such as instructions embodied in computer-readable program code logic, may also be stored in a computer-readable memory in the form of non-transitory computer-readable media, that can direct a computer or other programmable processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the block(s) of the flowchart(s). The computer program instructions may also be loaded onto a computer or other programmable processing apparatus to cause a series of operational steps to be performed on the computer or other programmable processing apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable processing apparatus provide steps for implementing the functions specified in the block(s) of the flowchart(s), algorithm(s), formula(e), or computational depiction(s).
It will further be appreciated that the terms “programming” or “program executable” as used herein refer to one or more instructions that can be executed by a processor to perform a function as described herein. The instructions can be embodied in software, in firmware, or in a combination of software and firmware. The instructions can be stored local to the device in non-transitory media, or can be stored remotely such as on a server, or all or a portion of the instructions can be stored locally and remotely. Instructions stored remotely can be downloaded (pushed) to the device by user initiation, or automatically based on one or more factors. It will further be appreciated that as used herein, that the terms processor, computer processor, central processing unit (“CPU”), and computer are used synonymously to denote a device capable of executing the instructions and communicating with input/output interfaces and/or peripheral devices.
Referring now to
It should be noted that any of the components (202, 204, 206, 208 (including 210, 212, 214, 216, 218, 220, 222, 224, 226, and 228)) may be implemented by one or more local electronic devices and/or one or more remote devices, as will be discussed in greater detail below.
It should be noted that the following description of the VHA system 200, as well as the accompanying figures, are meant to be illustrative and are in no way meant to be limiting. Other VHA systems can be arranged differently and can include different components or modules, while still being capable of performing the features disclosed in the present application.
As illustrated, the VHA system 200 can include a temporal database 202, a decisions database 204, an analysis module 206, and/or a report engine 208. In some embodiments, the temporal database 202, the decisions database 204, the analysis module 206, and/or the report engine 208 may be implemented on a local electronic device that a user may use in his or her home or care facility or a healthcare professional may use in his or her office. In other embodiments, one or more of the temporal database 202, the decisions database 204, the analysis module 206, and/or the report engine 208 may be implemented on a remote device, for example, on one or more devices associated with a VHA cloud 902, as seen in
Similar to
The VHA system 200 may include one or more local electronic devices, which may include any one or more of the components depicted in
The VHA system 200 may also include one or more remote devices, which may include any one or more of the components depicted in
As should be clear, while only one of each of the components in
Turning again to
The analysis module 206 may receive the input data 201, which may include a user selection from a predetermined list of options, a text input, an audio or sound signal, and/or a video signal, from the input 102. As will be further described below, the analysis module 206 may additionally receive patient information stored in the temporal database 202, as well as corresponding decisions stored the decisions database 204.
The analysis module 206 comprises a rules engine 210, an enrollment module 212, a personalization module 214, a speech-to-text module 216, a speaker-state detection module 218, a speaker identification module 220, an acoustic signal processing module 222, a virtual personal assistant (VPA) 224, a text-to-speech module 226, and/or a training module 228.
The analysis module 206, along with the rest of the VHA system 200, is governed by the rules engine 210. The rules engine 210 contains a plurality of rules that collectively define the system behavior. Rules are typically in the form of predicate tests (e.g., “If Condition then Action(s)”) where the condition can be tested using an explicit test (e.g., “Temperature>101”) or a fuzzy test (e.g., “Breathing fast”). Rules can be predefined based on standard practice, and new rules can be added by a health care professional or other user if the existing rules do not support a required test. If a condition is met, then the rules engine can trigger one or more actions. In some instances, the actions can be queries to the user (e.g., “You seem to be coughing a lot. Should I notify your Doctor?”). The rules engine 210 continually monitors the data in the temporal database 202 and, when a specific condition is met, takes the corresponding actions.
The system behavior defined by the rules engine 210 can include: (i) The types of sound signals the VHA system 200 is passively monitoring at any given time. (ii) How passively detected sound signals may trigger active engagement. For example, detecting coughing 5 times in a period of 30 minutes may trigger a dialogue to obtain more information about the cough. (iii) How and when to engage with the user 122 for pre-scheduled interactions. For example, 3 times a week the user 122 may engage in a dialogue relevant to his ability to perform their activities of daily living. (iv) How information collected via active engagement impacts follow-up monitoring. For example, after detecting fatigue and engaging in a fatigue related dialogue, the system switches to monitoring for sound signals indicating lack of sleep, snoring, extended physical activity, etc.
As such, the rules engine 210 controls the functionality of the VHA system 200. Rules defining the behavior can be specified manually, via the user interface 118, based on common practices from nurse caregivers, health care professionals, or other persons or customized care required for the user 122 specifically. As indicated above, the VHA system 200 may adapt/adjust the rules in a pre-defined manner based on information collected previously by the VHA system 200. The user interface 118 for defining the rules may include a graphical interface enabling activation of certain rules among multiple pre-defined choices or a natural language interface (verbal and/or text).
As will be described in detail below, the rules engine 210 can be adapted and updated using manual adjustments made by the user 122 or through autonomous adjustments made by the training module 228 based on machine learning (“ML”).
Upon initialization of the VHA system 200, the user 122 can initialize a profile using the enrollment module 212 and configure the VHA system 200 to preferences of the user 122 using the personalization module.
Using the enrollment module 212, the user 122 can register with the VHA system 200 and the user 122 or a caregiver, health care profession, or other person can define the type of sound signals that require monitoring for the user 122. The enrollment module 212 generally receives an auditory sample from the user 122 that, in one embodiment, can then be subsequently used with the speaker identification module 220 to identify the target user 122 among multiple people present in the same environment.
Additionally, if the VHA system 200 is equipped with the imaging system 120, the enrollment module 212 can also be used to register the user 122 using other biometric signatures, such as an infrared signature, or, if video monitoring is an acceptable option, a facial image.
In some instances, if the user 122 has not originally enrolled for a specific type of identification (e.g., voice recognition, facial recognition, etc.), and the rules engine 210 is set or updated to require that type of identification, then the user 122 could be asked to enroll or reenroll for that type of identification.
In some instances, the input data 201 is in the form of a continuous sound signal, which can contain both utterances (such as spoken words, grunts, sighs, etc.) by the user 122 and ambient sound conditions or ambient sound signals. The sound signal can be continuously analyzed using the speech-to-text module 216, the speaker-state detection module 218, the speaker identification module 220, and the sound signal processing module 222. The speech-to-text module 216 converts any utterances spoken by the user 122, contained within the sound signal, into corresponding text data. The speaker-state detection module 218 analyzes the sound associated with any utterances spoken by the user 122 to determine a state of the user 122. For example, the state of the user 122 can be one or more of fatigued, happy, bored, sad, angry, or any other state configured to be monitored by the VHA system 200. The speaker identification module 220 analyzes the sound signals associated with any utterances spoken by the user 122 to identify the user 122 with an associated probability. The sound signal processing module 222 analyzes sound signals to identify possible causes. For example, the sound signal processing module 222 may identify a cough, a cough epoch (e.g., a number of coughs in one or more time periods), a yawn, a fall, a door knock, audible wheezing, hoarseness, aphasia, dysarthria, stridor, repetitive statements, depression, delirium, mania or any other sound signals configured to be monitored by the VHA system 200.
After the input data 201 has been analyzed, the VPA 224 can be used to manage the interaction between the analysis module 206 and the user 122. This management may comprise the VPA 224 applying the text data, the state of the user 122, the identity of the user 122, the observed sound signals, and/or any other input data 201 received by the analysis module 206, as well as information retrieved from the decisions database 204 and the temporal database 202, to the rules engine 210 to determine an appropriate user interaction or dialogue, which can be in the form of text data.
Once the VPA 224 determines an appropriate response, the VPA 224 can then apply the text data to the text-to-speech module 226. The text-to-speech module 226 converts the text data to speech, which is outputted through the output 108 to the user 122. The output 108 can comprise a speaker or other audio or sound emitting device.
Periodically, or as a result of a user command, the analysis module 206 can instruct the report engine 208 to generate a report, as will be discussed in detail below.
Now that the general layout of the VHA system 200 has been described above, an example mode of operation will be described. It will be understood that the following description is in no way intended to be limiting.
Referring now to
In some instances, the operation may switch from passive monitoring mode 302 to active engagement and monitoring mode 304 as a result of various trigger events 306, 306′, 306″ based on observed sound signals and interactions with the user 122 who is being monitored. For example, in some instances, a trigger event may pertain to various loud noises detected by the VHA system 200, in order to switch into active engagement and monitoring mode 304 to verify whether or not the user 122 needs help. In some other instances, a trigger event may pertain to an unknown speaker being detected if the VHA system 200 detects speech, but does not recognize the speaker. In some instances, the unknown speaker trigger event may be selectively turned on or off to avoid over firing in case of, for example, a party. The trigger events can be sound signal cues that do not correspond with the words spoken by user (e.g., as part of a direct communication by the user to the system), although they may correspond with how words are spoken by the user (e.g., if the user is speaking particularly slowly, repeating or slurring words, sounding as though he or she is in pain, etc.). It should be appreciated that these examples, while illustrative of the capabilities and operation of the VHA system 200, are not meant to be limiting, as any other suitable sound signals can be selectively monitored.
Additionally, the operation may switch from passive monitoring mode 302 to active engagement and monitoring mode 304 as a result of a scheduled interaction transition 307. Once the operation switches into active engagement and monitoring mode 304, the VHA system 200 may begin an active engagement or dialogue with the user 122. During the active engagement and monitoring mode 304, the VHA system 200 continues to collect data such as sound signals from the user and from the user's environment. In addition, the VHA system 200 collects data from the user 122, thereby verifying and augmenting the passive patient information through communication with the patient to increase confidence on the produced reports. Once the VHA system 200 has finished interacting or engaging with the user 122, an engagement-completed transition 308 will occur, returning the VHA system 200 to the passive monitoring mode 302 once the engagement is completed.
Referring now specifically to
Some possible sound signals that can be monitored in passive monitoring mode 302 are, for example, coughing, snoring, sneezing, fire alarms, running water, broken glass noises, door knocks, an unknown (and potentially unauthorized) speaker in the acoustic environment, long silences, the user 122 talking to oneself, fatigue indicators, dog barking, loud noises (which may correspond to falls, dropped items, or any other sound signal desired to be monitored). Each sound signal may have a sound signature/pattern/“fingerprint” that can help identify the sound signal (with a certain confidence level). Thus, in addition to providing information for use in general health assessments, passive monitoring of a sound environment of, for example, a house may be used to infer the security and well-being of the home occupants by detecting anomalous or unexpected sound signals that could indicate potential threats and/or high-risk situations like unfamiliar speakers in the house.
Some possible scheduled interactions that the VHA system 200 could check for are, for example, reminders for daily medications, commencement of periodic health questionnaires, memory test or any other possible scheduled interaction with the user 122.
As illustrated, the flowchart 400 illustrates various flow paths for 1st through nth sound signals. A 1st flow path, a 2nd flow path, and an nth flow path are shown for the 1st, 2nd, and nth sound signals, and ellipses 406 are included to signify that the VHA system 200 can be configured to monitor for any number of sound signals prescribed by the user 122 following a similar flow path. Additionally, as these flow paths are substantially similar, the following description of the 1st flow path also pertains to the 2nd through nth flow paths. Further, steps in the 2nd flow path will be labeled similarly in the prime series (e.g., step 408 and step 408′), and steps in the nth flow path will be labeled similarly in the double-prime series (e.g., step 408 and 408″). It will be understood that the following flow path description is intended to provide an example, and is in no way intended to be limiting.
While monitoring the sound signals, at step 404, the VHA system 200 can determine, at step 408, whether a 1st sound signal has been detected. The 1st sound signal can be any of the previously described possible sound signals. Upon determining that the 1st sound signal has not been detected, at step 408, the VHA system 200 can return to monitoring the sound signals, at step 404. Upon determining that the 1st sound signal has been detected, at step 408, the VHA system 200 can update a 1st sound signal counter, at step 410. The 1st sound signal counter can be used to keep track of the number of times the 1st sound signal is detected and the frequency with which the 1st sound signal is detected over a given time period. This information can then be stored in the temporal database 202 and selectively used by the VHA system 200 or a user 122 to adapt or update the rules engine 210, as will be described below.
After updating the 1st sound signal counter, at step 410, the VHA system 200 can determine if a 1st sound signal criteria has been met, at step 412. The sound signal criteria can indicate, for example, a frequency or other condition or criterion. It should also be noted that the 1st sound signal criteria can be set and reset by the rules engine 210 periodically based on user 122 or autonomous adaptation of the rules engine 210, as will be described below.
Upon determining that the 1st sound signal criteria has not been met, at step 412, the VHA system 200 can return to monitoring the sound signals, at step 404. Upon determining that the 1st sound signal criteria has been met, at step 412, the VHA system 200 can activate a 1st trigger event 306, at step 414. Activating the 1st trigger event 306, at step 414, transitions the VHA system 200 into the active engagement and monitoring mode, at step 416.
Additionally, while monitoring the sound signals, at step 404, the VHA system 200 can determine whether there is a scheduled interaction due, at step 418. Upon determining that there is not a scheduled interaction due, at step 418, the VHA system 200 can return to monitoring the sound signals, at step 404. Upon determining that there is a scheduled interaction due, at step 418, the VHA system 200 can activate the scheduled interaction transition 307, at step 420. Once the scheduled interaction transition 307 is activated, at step 420, the VHA system 200 transitions into the active engagement and monitoring mode 304.
It should be appreciated that throughout operation in the passive monitoring mode 302, at each step described above, for each of the 1st through the nth sound signals, as well as the scheduled interactions, the VHA system 200 can collect data from the analysis module 206 and store that data in the temporal database 202 to be used in an adaptation of the rules engine 210, as will be described in detail below. In certain implementations, in the passive monitoring mode 302, the VHA system 200 may only be monitoring pre-defined sound signals of interest and does not store/save all of the input data 201, thus preserving user privacy. For example, during passive monitoring, the VHA system 200 may be collecting data pertaining to the identification of coughs in a sound stream. In this case, the VHA system 200 may be ignoring other sound signals in the input data 201 that are not medically relevant (e.g., specific words, laughter, etc.).
Generally, the VHA system 200 switches from the passive monitoring mode 302 to the active engagement and monitoring mode 304 to engage in a dialogue with the user 122 in an effort to validate detected sound signals and/or augment detected sound signals with extra information. This validation by the VHA system 200 both confirms that the VHA system 200 should take action, and also helps to self-calibrate or self-train the VHA system 200 through adaptation, as will be described below.
Referring now specifically to
As illustrated, similar to the flowchart 400, the flowchart 500 illustrates various flow paths for 1st through nth trigger events. A 1st flow path, a 2nd flow path, and an nth flow paths are shown for the 1st, 2nd and nth trigger events, and ellipses 506 are included to signify that the VHA system 200 can be configured to actively engage the user 122 in response to any number trigger events prescribed by the user 122 following a similar flow path. Additionally, as each of these flow paths are substantially similar, the following description of the 1st flow path also pertains to the 2nd through nth flow paths. Further, steps in the 2nd flow path will be labeled similarly in the prime series (e.g., step 508 and step 508′), and steps in the nth flow path will be labeled similarly in the double-prime series (e.g., step 508 and 508″). It will be understood that the following flow path description is intended to be an example, and is in no way intended to be limiting.
While checking the trigger events at step 504, the VHA system 200 can determine whether the 1st trigger event has been activated, at step 508. Upon determining that the 1st trigger event has not been activated, at step 508, the VHA system 200 can return to checking the remaining trigger events, at step 504. Upon determining that the 1st trigger event has been activated, at step 508, the VHA system 200 can perform a 1st trigger event dialogue with the user 122, at step 510. For example, in some instances, the 1st trigger event indicates that coughing has been detected and has met or exceeded a threshold coughing criteria. In these instances, the 1st trigger event dialogue may be directed at following up to confirm the detected coughing and collect more information from the user 122 relevant to the cough condition. This information can be stored in the temporal database 202 and can be used to adapt the rules engine 210 manually or autonomously, as will be described below.
While checking the trigger events and scheduled interactions, at step 504, the VHA system 200 can determine whether a scheduled interaction transition has been activated, at step 512. Similarly, upon determining that the scheduled interaction transition has not been activated, at step 512, the VHA system 200 can return to checking the remaining trigger events, at step 504. Upon determining that the scheduled interaction transition has been activated, at step 512, the VHA system 200 can perform a scheduled interaction dialogue, at step 514.
In some instances, the scheduled interaction dialogue can pertain to various prescheduled events (that correspond to particular sound signals). For example, the dialogue may comprise a questionnaire for activities of daily living, such as eating habits, or administering a cognitive assessment test. In some other instances, the scheduled interaction dialogue can pertain to a periodic reminder interaction. For example, the dialogue may comprise a daily, weekly, or monthly medication reminder.
After performing any of the various dialogues pertaining to any of the trigger events and/or the schedule interaction transition, the VHA system 200 can determine, at step 516, whether a triage dialogue is necessary. The VHA system 200 determines that the triage dialogue is necessary by interpreting the responses provided by the user 122 (alone or in combination with other acoustic or other data), and making a decision, based on rules provided by the rules engine 210, regarding whether or not the responses warrant sending an alert to a caregiver (or other system). Upon deciding that the triage dialogue is not necessary, at step 516, the VHA system 200 can return to checking the remaining trigger events and scheduled interactions, at step 504.
Upon determining that the triage dialogue is necessary, at step 516, the VHA system 200 can perform the triage dialogue, at step 518. After performing the triage dialogue, at step 518, the VHA system 200 can then determine whether or not it is necessary to notify the caretaker, at step 520. Upon determining that it is necessary to notify the caretaker, at step 520, the VHA system 200 can send an alert to the caretaker, at step 522. Upon determining that it is not necessary to notify the caretaker, at step 520, the VHA system can return to checking the remaining trigger events and scheduled interactions.
For example, in response to the coughing example provided above, during the triage dialogue, at step 518, the VHA system 200 may ask the user 122 a question such as, “Since you've been coughing for several days, and have been running a fever, I'd like to let your doctor know. Is that okay?” In this instance, if the user 122 replies that it is okay for the VHA system 200 to notify the caretaker (or does not reply, which may indicate something is wrong), the VHA system 200 can determine that it is necessary to notify the caretaker, at step 520, and can further send an alert to the caretaker, at step 522. If the user 122 alternatively replies that the caretaker should not be notified, the VHA system 200 can determine that it is not necessary to notify the caretaker, at step 520, and the VHA system 200 can return to checking the remaining trigger events and scheduled interactions. At either step 514 or step 518 (indicated by arrows 524, 526, respectively), during the interaction or triage dialogue, if it is determined by the VHA system 200 that another issue exists, the VHA system 200 may return to the steps 508, 508′, 508″ to determine if the 1st, 2nd and nth trigger events have been have been activated. In this situation, the VHA system 200 would proceed through the flow paths of
In some instances, questionnaires administered through the active engagement and monitoring mode can regularly assess a capability of the user 122 to perform activities of daily living (ADLs), nutrition, mobility, and social interaction. The active engagement and monitoring mode can also administer disease severity instruments for cognitive decline, such as the Western Aphasia Battery (WAB) and Mini-Mental State Evaluation (MMSE).
Referring to
The reports are typically requested and generated after at least some active engagement have taken place. The reports are further typically provided in either human-readable form or in data-exchange formats (e.g., JSON, XML, proprietary binary, etc.) and summarize the input data 201 from the user 122 including both the active patient information acquired from responses provided by the user 122 during active engagement and passive patient information acquired during passive monitoring of the user 122. However, although the illustrated process 602 includes a health care professional 604 requesting a health report, reports including passive patient information acquired during passive monitoring can be generated at regular intervals (e.g. twice a day) aggregating the statistics of the sound signals and/or trigger events collected (e.g. type, duration and frequency of cough epochs), and can be delivered as requested.
The VHA system 200 can further determine the type of report to be generated in a variety of methods. For example: (i) the health care professional 604 can specify the type of report to be produced, and can further specify the collected information and statistics to be included in the report, (ii) the user 122 can specify the type of report to be produced, including what information should be presented to others (e.g. family members) or him/herself, (iii) the VHA system 200 can make a decision on what kind of report should be produced based on prior examples, via machine learning algorithms, and internal rules executed by the training module 228 and the rules engine 210, as will be described below. As an example, a report intended for a doctor may be formatted differently and may contain different information than a report intended for the user or for a relative of the user. In the last case the contents of the report could be confirmed by the subject or the caregiver before the report is generated, unless the subject or the caregiver chooses to waive that option.
After the report has been generated, at step 706, the report is returned to the health care professional 604, at step 708. The health care professional 604 then reviews the report, at step 710, and makes a decision based on the report, at step 712. The decision, as well as the corresponding data can then be stored in the decisions database 204 and the temporal database 202, at step 714. Over time, the decisions database 204 and the temporal database 202 can retain records of decisions and corresponding data that led to the health care professional 604 making those decisions. These databases 202, 204 can then be used as bases for updating the rules engine 210 through the adaptation process 606, described below.
Referring now to
In some other instances, the adaptation process 606 described above may be used to adapt the active engagement and monitoring mode 304 as it collects the input data 201 from the user 122 based on prior responses from the user 122 retrieved from the temporal database 202. For example, an active engagement about coughing may also typically request information about allergies. However, if the system collected that information earlier, while engaging in an interaction about sneezing, the rules engine 210 will have received modified rules through the adaptation process 606, thereby adapting the dialogue regarding coughing, as there is no reason to repeat that portion of the dialogue again. Another example of the adaptation process 606, may include adapting the style of the interaction. For example, if fatigue is detected in a recent interaction, then the rules engine 210 will have received modified rules aimed at reducing questions when another trigger event is detected and attempting to encourage the user 122 to rest.
In yet other instances, the VHA system 200 can go through the adaptation process 606 whenever a trigger event forces the VHA system 200 into the active engagement and monitoring mode 304. In these instances, the adaptation process 606 can cause the VHA system 200 to adapt its responses and actions based on continuously analyzed detected signals and interactions with the user 122. For example, if a “too much coughing” rule is triggered, the VHA system 200 may inquire about the trigger by asking, for example, “It seems that you have been coughing a lot. May I ask a few questions?” If the user 122 responds, “Ask me in an hour,” then the VHA system 200 adapts by waiting an hour and then switching back into active engagement with the user 122. Likewise, if the VHA system 200 asks several questions based on coughing and learns that the user 122 has extensive chest pain, it could then switch to questions or tests related to a possible heart attack. That is, the VHA system 200 adapts its queries and responses to input data 201 from the user 122.
In other instances, the VHA system 200 can use the adaptation process 606 to modify sensitivity of the VHA system 200 during both passive monitoring mode 302 and active engagement and monitoring mode 304 to, for example, shift toward more falsely detected trigger events pertaining to critical conditions.
In some other instances, the VHA system 200 can use the adaptation 606 to not only monitor sounds signals and detect trigger events, such as, for example, a cough or a cough epoch, but also adaptively learn to distinguish between detected sound signals of the same type based on characteristics of the input sound signal. The adaptation process 606 can also be used to determine which sound signals are monitored during passive monitoring mode 302. For example, if fatigue was detected during an interaction with the user 122 during the day, the passive monitoring mode 302 may be adapted to monitor for various indications of poor sleep quality, such as, for example, snoring, sleep apnea, or any other suitable sleep characteristics.
Referring now to
As such, the decisions databases and temporal databases of the various modules 904, 906, 908 can be manually or automatically uploaded or sent to the VHA cloud 902 to form a universal decisions database and universal temporal databases. Each of the various modules 904, 906, 908 can then selectively retrieve information from the universal decisions database and the universal temporal database. Therefore, when new information is gathered by any VHA module 904, 906, 908, the VHA cloud 902 and/or an individual VHA module 904, 906, 908 can draw upon additional information gathered by other modules 904, 906, 908 to aid in analysis and decision making.
Additionally, each of the various modules 904, 906, 908 may be governed by a common universal rules engine, which can be stored and adapted within the VHA cloud 902. The universal rules engine can be adapted through machine learning, as described above, using the adaptation process 606, but has access to significantly more information. That is, the universal rules engine can be adapted through machine learning using information gathered from multiple modules 904, 906, 908 monitoring different users in varying settings. This vast amount of additional information thereby provides a more robust or larger data set for the VHA system 900 to learn from.
In some illustrative embodiments, a universal decisions database, a universal analysis module (including one or more of its sub-modules), a universal report engine, and a universal temporal database (see elements 202, 204, 206, and 208) may be stored and adapted within the VHA cloud 902. In such an embodiment, each of the various modules 904, 906, 908 may have basic functionality thereon to communicate with the VHA cloud 902 and the VHA cloud may analyze data from the respective modules 904, 906, 908 and make decisions based on the analyzed data, wherein the decisions are communicated to the respective modules 904, 906, 908 for implementation.
The mobile VHA modules 906 also monitor sound signals, detect trigger events, and interact with various users, as described above, with reference to the VHA system 200, but may be programmed to operate using a mobile device, such as, for example, a smartphone, a tablet, or any other suitable device (which may provide processing capabilities and inputs such as one or more microphones or cameras).
The health care VHA modules 908 are used mainly by health care professionals and family members and provide interfaces to the VHA cloud 902. Through the VHA cloud 902, this allows health care professionals and family members to interface with appropriate home-based and mobile VHA modules 904, 906. The health care VHA modules 908 retrieve data and reports from the cloud, generate alerts, allow approved personnel to modify rules, and interface to other health care IT systems (e.g., electronic medical-record systems).
Referring now to
Each of the continuously-monitored sound signal detection values 1012 is further tracked against a corresponding sound signal detection threshold value 1014, corresponding to a threshold detection value for the given sound signal. The sound signal detection threshold value 1014 may be an upper limit, for example, when monitoring coughing of the user 122, the sound signal detection threshold value 1014 may be, for example, five coughs detected within ten minutes. In this instance, if the VHA system 100, 200, 900 or VHA module 904, 906, 908 detects five or more coughs within ten minutes, it triggers a trigger event. Alternatively, the threshold value 1014 may be a lower limit, for example, when monitoring fatigue of the user 122, the sound signal detection threshold value 1014 may be a detected amount of sound energy output from the user 122. In this instances, when the VHA system 100, 200, 900 or VHA module 904, 906, 908 detects an amount of sound energy output lower than the sound signal detection threshold value 1014, it triggers a trigger event. The sound signal detection threshold value 1014 can be selected manually, through input from the user 122, or autonomously, based on machine learning, using the adaptation process 606, as described above.
The monitoring pane 1004 further indicates sound signal occurrences 1016. These sound signal occurrences 1016 indicate that the continuously-monitored sound signal detection value 1012 has met or exceeded (or, in the lower limit instances, dropped below) the sound signal detection threshold value 1014, and that a trigger event has been triggered.
It should be noted that any of the VHA systems 100, 200, 900, any of the VHA modules 904, 906, 908, any of their corresponding features, or any other various features, such as the user interface 1000, disclosed herein can be used in various combination with each other, as desired for a given situation. These combinations have been considered and are well within the scope of the present disclosure.
As such, the present invention provides VHA systems that ensure safety of the user by continuous passive monitoring for anomalous sound signals (falls, fire alarm or broken glass noises) and unfamiliar speakers at home. If a concerning sound signal is detected, the system would engage VPA (active engagement and monitoring mode), to verify with user and reduce reported false alarms. The system would verify if there is an action to take (like call the emergency services), summarize the necessary information and produce a report to present to care-givers, health care professionals, emergency assistants, or other persons. For example, if a fall is detected and verified (as shown in
The VHA systems disclosed further enable frequent, easily administered, reliable and cost effective wellness screenings.
The VHA systems can additionally monitor for indicators of illness or decline in health over time, and interact with caregivers, so that a health care professional can follow up promptly. A VPA interaction can collect information that can be reported in follow up calls, as needed, with advising nurses, physicians, or other healthcare providers. In aggregate, summarized semantic (e.g., the “digest” of a symptom evaluation) and acoustic data can be made available to health care givers through SMS or email alerts and an intuitive graphical user interface to ensure that telephone follow-up, home care visits, office evaluations, or, if necessary, emergency evacuation to an ER, take place in an efficient, timely and appropriate fashion based on available data.
The VHA systems can further provide assistance and reminders for everyday activities. The VPA systems can serve as a medication reminder, and source of simple solutions for common problems that may affect activities of daily living, risks in the home, or nutritional status. Such an assistant would complement the more thorough assistance from care-givers and clinicians.
The VHA systems can additionally provide high-accuracy automated speech recognition. From an anatomical point of view, studies have shown age-related degeneration with atrophy of vocal cords, calcification of laryngeal cartilage, and changes in muscles of the larynx. Such changes, result in specific features of elderly voices, such as imprecise production of consonants, tremors and slower articulation, making ASR systems trained on regular adult population not very accurate on elderly speakers. The VHA systems can address this problem by applying a high accuracy ASR system for the elderly through robust features and modeling informed by existing data collected from the elderly population.
The present invention has been described in terms of one or more preferred embodiments, and it should be appreciated that many equivalents, alternatives, variations, additions, and modifications, aside from those expressly stated, and apart from combining the different features of the foregoing embodiments in varying ways, can be made and are within the scope of the invention. In the above description, a number of specific details, examples, and scenarios are set forth in order to provide a better understanding of the present disclosure. These examples and scenarios are provided for illustration, and are not intended to limit the disclosure in any way. Those of ordinary skill in the art, with the included descriptions, should be able to implement appropriate functionality without undue experimentation. References in the specification to an “embodiment,” an “example,” a “version,” an “implementation,” a “configuration,” an “instance,” etc., indicate that the embodiment, example, version, etc. described may include a particular feature, structure, or characteristic, but every embodiment, example, version, etc. may not necessarily include the particular feature, structure, or characteristic. Such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed to be within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly indicated.
The computerized functionality described above may be implemented in hardware, firmware, software, single integrated devices, multiple devices in wired or wireless communication, or any combination thereof. Computerized functions may be implemented as instructions stored using one or more machine-readable media, which may be read and executed by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine. For example, a machine-readable medium may include any suitable form of volatile or non-volatile memory. In the drawings, specific arrangements or orderings of schematic elements may be shown for ease of description. However, the specific ordering or arrangement of such elements is not meant to imply that a particular order or sequence of processing, or separation of processes, is required in all embodiments. Further, some connections or relationships between elements may be simplified or not shown in the drawings so as not to obscure the disclosure. This disclosure is to be considered as exemplary and not restrictive in character, and all changes and modifications that come within the spirit of the disclosure are desired to be protected.
This application is a U.S. National Stage Patent Application filed under 35 U.S.C. § 371 of International Patent Application No. PCT/US17/35859 filed Jun. 3, 2017, which claims priority to U.S. Provisional Application Ser. No. 62/345,610, filed Jun. 3, 2016, the entire contents of all of which are hereby incorporated by reference as if fully set forth herein for all purposes. N/A
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2017/035859 | 6/3/2017 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2017/210661 | 12/7/2017 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6231500 | Kehoe | May 2001 | B1 |
| 6416471 | Kumar | Jul 2002 | B1 |
| 8821415 | Al-Ali | Sep 2014 | B2 |
| 8905927 | Cheung Hyen | Dec 2014 | B2 |
| 20030101078 | Voegeli | May 2003 | A1 |
| 20040059205 | Carlson | Mar 2004 | A1 |
| 20070208558 | de Matos | Sep 2007 | A1 |
| 20080249376 | Zaleski | Oct 2008 | A1 |
| 20100286490 | Koverzin | Nov 2010 | A1 |
| 20110015496 | Sherman | Jan 2011 | A1 |
| 20120191476 | Reid et al. | Jul 2012 | A1 |
| 20120265024 | Shrivastav | Oct 2012 | A1 |
| 20130085758 | Csoma et al. | Apr 2013 | A1 |
| 20130090567 | Lee | Apr 2013 | A1 |
| 20190272725 | Viklund | Sep 2019 | A1 |
| Entry |
|---|
| Al Hemairy, M., Amin, S., Hijji, M., Serhani, M., & Al Ahmed, M. (Aug. 2016). Integrated and Scalable Architecture for Providing Cost-Effective Remote Health Monitoring. In 2016 9th International Conference on Developments in eSystems Engineering (DeSE) (pp. 74-80). IEEE. |
| The International Bureau of WIPO, “International Preliminary Report on Patentability”, in application No. PCT/US2017/035859, dated Dec. 4, 2018, 11. |
| Current Claims in application No. PCT/US2017/035859, dated Dec. 2018, 9 pages. |
| International Searching Authority, “Written Opinion” in application No. PCT/US2017/035859, dated Aug. 16, 2017, 10 pages. |
| International Searching Authority, “Search Report” in application No. PCT/US2017/035859, dated Aug. 16, 2017, 2 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20190108841 A1 | Apr 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62345610 | Jun 2016 | US |
