Patent Grant
11170484
The subject matter described in this application generally relates to patient compliance to medication administration protocol scenarios including ingestion of medication, and more particularly to an apparatus and method for the identification of suspicious or purposefully malicious activity on the part of the patient, where the patient is purposefully not properly administering their medication.
Traditionally, participants attend introductions and follow ups for clinical trials in-person. Other patients outside of the clinical trial setting attempting to adhere to a particular medication protocol similarly are given a prescription and a particular set of instructions from a prescribing medical provider or prescribing doctor. Compliance is then measured at a next visit with that prescribing professional through traditional methods of counting unused medication, and patient interviews. Thus, data collection is similarly limited to patient visits, rather than on a daily basis. These old methods such as patient questioning and medication counting have been proven to be inadequate measures of adherence and offer no information on dose timing and drug holidays (omission of medication for three or more sequential days).
U.S. patent application Ser. No. 13/831,555, filed Mar. 14, 2013, titled Apparatus and Method for Recognition of Suspicious Activities to Hanina et al., the contents of this application being incorporated herein by reference in its entirety, describes a system, method and apparatus that allow for determination of suspicious activity by a patient administering medication following a predetermined protocol or prescription.
The subject matter described in this application is directed to a system and method for detecting suspicious activity of a user (hereafter also referred to as a patient), and to thwart users who purposefully do not take their medication. The system presents an advanced mechanism for teaching a system to acquire and learn what to look for related to suspicious activity, and further includes a mechanism for analyzing input captured video to determine suspicious activity. The system may also interact in real-time with the patient when something is identified as incorrect, wrong or suspicious and either prompt a different action or raise an alert and recommend an intervention by a healthcare provider.
In accordance with an embodiment of the subject matter described in the present application, a motion capture procedure for capturing patient motion data related to the administration of pill or film based oral medications, or injectable, inhaler-based, other non-pill based medication, or any other form of patient administration task that may be performed, may be utilized. Other capture sensors may further be utilized, including range (depth) sensors, hyperspectral sensors, such as infrared or near infrared sensor, and the like. Such sensors may help to localize and segment the face and hand, improving the ability to discern motion and other attributes of a video and other data capture. Use of such other sensors may also help to identify material properties of an item, such as a medication pill, thus assisting in confirming the pill's authenticity. Therefore, in accordance with an embodiment of the present application, a method and apparatus may be provided for analyzing the captured patient motion data to provide feedback to the user and/or to determine a number of times the user performs a suspicious action. The system can detect suspicious action and provide feedback to the user in near real-time, and importantly, in time sufficient to allow for the feedback to be useful to the subject while the subject is still performing the activity. Additionally, the patient performance information may be analyzed asynchronously to determine other features of data that may suggest a malicious intent on the part of the patient.
Still other objects and advantages of the subject matter described in this application will in part be obvious and will in part be apparent from the specification and drawings.
The subject matter described in this application accordingly comprises the several steps and the relation of one or more of such steps with respect to each of the others, and the apparatus embodying features of construction, combinations of elements and arrangement of parts that are adapted to affect such steps, all as exemplified in the following detailed disclosure, and the scope of the subject matter described herein will be indicated in the claims.
For a more complete understanding of the subject matter described in this application, reference is made to the following description and accompanying drawings, in which:
Compliance and medication adherence technologies can increase the statistical power of clinical trials. Through the use of such technology, clinical events can be precisely linked to medication use history. Captured data can be linked to other sources such as EDC, patient diaries and data collected by the physician. Technologies can create many possibilities for remote visits and data capture. While smart packaging technologies exist such as RFID-enabled computer chip technology, smart blister packs and MEMS caps (microprocessor in a bottle cap), they are: a) invasive and need to be physically attached to the medications; b) are non-conclusive regarding compliance—a patient may activate the technology without ingestion of the medication; c) remain largely un-adopted in clinical trials by the pharmaceutical and biotech companies due to their high cost; and d) take a longer time to implement. Further, electronic patient diaries allow for ease of entry of data by a patient. These diaries, however, are still subject to issues related to compliance with medication adherence. Thus, even if a patient is meticulous about entering information into the diary, and thus complying with the requirements for data entry, there is still no guarantee that they are properly taking medication at prescribed times. This problem is even more acute when a participant is performing a suspicious action, or is otherwise maliciously avoiding taking their medication.
The subject matter of the present disclosure aims to address the drawbacks of the prior systems. The subject matter of the present disclosure will now be described making reference to the following drawings in which like reference numbers denote like structure or steps. Referring to
Once confirmed, a medication administration and ingestion or other medication regimen in accordance with the clinical trial or other prescription requirements such as in a public health, medical practice environment or the like may be prescribed and entered into the system of the invention at 120. Such medication administration regimen may include one or more of ingesting or swallowing a pill, using an inhaler device, using an injectable medication device, employing a patch, sublingual administration, a cheek or other skin located medication administration device or the like. Of course, the present invention may be applied to any patient administered procedure. Once entered into the system, a particular prescription regimen may cause a set of user instructions, various training sequences and the like 125 to be generated and transmitted to an apparatus (e.g., the remote information capture apparatus 1000 of
Such an apparatus may comprise a user device, such as a smart phone, a netbook, a laptop computer, a desktop computer, a tablet device, a free standing or built into a mirror or cabinet, or other computing appliance. The apparatus includes a video and audio capture apparatus (e.g., a camera, a voice recorder, and/or a video capture device), and a video and audio analysis and transmission apparatus for transmitting the captured video/audio to another system or device. The apparatus allows for the display of the user instructions to a patient, and allowing for the eventual capture, analysis and transmission of video, audio and other analysis information. The apparatus can be considered as the patient facing portion of the system (e.g., the information capture and analysis system of
When installing software on a user's own hardware system, it is preferred that the software detect and otherwise test or determine that the hardware attempting to be utilized by the patient is sufficient to implement the system and method described in this disclosure and is sufficient to run a software package provided in accordance with the system and method described herein. Thus, the software may check that a camera includes sufficient resolution, that a memory of the device is of sufficient size to allow for sufficient captured video storage, that audio may be properly captured, and that the transmission system includes sufficient bandwidth to transmit and receive captured video, audio, video instructions and the like. Processing may also be performed at a remote location, thus allowing the user to include a lighter application or the like on their local device. Alternatively, the user may employ the local device as a gateway only, all data being transmitted to a remote system (e.g., the remote data and computing device 3000 of
Such user instructions and training sequences may include general instructions about the particular medication subject to the current trial or medication administration protocol, methods for administration, warnings about side effects, and concerns about drug interactions with common substances or medications, or other medications prescribed to the patient by the system or by another medical service provider. It is contemplated in accordance with an embodiment of the invention that such set of user instructions may be interactive, allowing a user to view additional information about such instructions or prescriptions as desired. These instructions may comprise written, audio or video instructions provided to the user on a display of the user apparatus. It is further contemplated that such instructions may indicate one or more movement sequences to be associated with a corresponding one or more medication administration sequences. These instructions indicate proper and improper motions that may be taken by a patient for ingesting a pill, using an inhaler, using an injectable medication, and the like, and may indicate various motions that may or may not be performed by a user. In such a manner, the patient may be instructed to properly perform all requested actions, and avoid actions that may be indicative of a patient trying to trick the system.
In an alternative embodiment, it may be possible to particularly not describe to the patient the various suspicious behaviors that will be tracked in an attempt to monitor without notification of the unsuspecting malicious patient. These traps may be maintained and running in a background of the system. Thus, as will be described below, detection of one or more errors may generate real time displayed video and/or audio feedback to the patient in order to correct actions during the session during which they are taking the medication so that the current administration session is not lost and may still be a successful administration, while one or more other errors may be logged by the system but preferably provide no feedback to the patient, and thus may be accumulated and used to analyze patient actions without providing instructions and help to avoid being recognized by the system. After being recognized as performing undesirable actions a predefined number of times (for example, after three, five, ten, twelve, or twenty times), the user may be identified as one who should have heightened security applied. Thus, a user may be first started in a low level of observation. After a predetermined number of potentially suspicious actions are recognized, the user may be labeled as a user that is attempting to trick the system, or that is performing one or more suspicious acts, thus warranting a heightened level of security. Additional suspicion will surround review of all actions of the patient. After properly using the system correctly for another predetermined period of time, the heightened level of security may be removed. If on the other hand, suspicious activity continues, the user may be recommended to be removed from a clinical trial, or taken off a particular medication, for example.
In accordance with one or more embodiments of the present disclosure, one or more of these sets of motions or actions may include confirmation that a user has placed a pill in their mouth and has properly swallowed the pill correctly and has therefore ingested the medication properly through visual confirmation of location, confirmation that the user has properly used an inhaler device through visual confirmation of position and/or audio confirmation of actuation, confirmation that the user has properly used an injectable device through visual confirmation of position and/or audio confirmation of actuation, or the like.
Referring to the lower portion of
In some implementations, the system shall be able to learn the pattern of suspicious patients based on all data collected (such as number of misses, number of skips, manual ratios, time on tasks, number of usability errors, number of suspicious errors, study coordinator/sponsors notes, flags, video reviewing results.). The system can learn the pattern using machine learning techniques such as unsupervised learning or supervised learning techniques, and therefore the system learns as data is processed during use. Based on prior patient data, the system is able to recognize patterns and classify the patient based upon these patterns. For example, if a patient is showing erratic administration time and times on task, coupled with some administration misses, and one or more suspicious activity errors, the patient may be classified as one likely to be lost to follow up, and to purposefully not take medication. A patient can therefore be easily classified as suspicious level 1, level 2, or not by checking whether the patient fits in certain pattern. The system will keep learning each day when more data are gathered and more patients are enrolled. Thus, a more accurate placement of the user in a particular category or risk classification, as will be described below, can be more accurately determined. This risk category can then be used to define a level of observance and appropriate follow up for the particular category of patient.
Once identified and notified of a type of medication to be administered, the patient may show the system a medication (e.g., a pill or dissolvable film), administration apparatus (e.g., an inhaler, injectable apparatus, or other medication form such as a pill bottle) to confirm that the medication is correct. As shown in
In addition to recognizing the pill or other medication, the system may track how patient holds the pill or other medication, and further may track continuous motion from hand to mouth to confirm that no suspicious action has taken place. Thus, the system may also check consistent hand usage between identification of the pill and placing the pill in the mouth of the user. In addition to tracking the motion of the hand, if the user is to take the pill out of a blister pack, the user may track a motion sequence for tracking complete gestures on screen from taking out of the pill bottle or blister pack all the way to placing in mouth/ingesting, drinking water, showing and tracking water go down, swallowing, and even showing empty mouth so that any deviation from the desired sequence may be identified as a potentially suspicious activity. Furthermore, the system may view the blister pack to determine if the correct number of pills have been removed, that the correct number of pills are remaining, and whether other pills or the like have been removed that should not have been removed. Such suspicious activity may be used to automatically flag potentially suspicious activity.
Thereafter, the system instructs the patient to administer the medication in the prescribed manner. One or more of these administration sequences may include confirmation that a user has placed a pill in their mouth (ingested the pill) through a visual confirmation of the location of the pill in the user's mouth, confirmation that the user has properly used an inhaler device through visual confirmation of position and/or audio confirmation of actuation, and/or confirmation that the user has properly used an injectable device through visual confirmation of position and/or audio confirmation of actuation.
The system (for example, the information capture and analysis system of
During this administration, the system monitors potentially suspicious activities performed by the patient (step 140 of
If a medication is being used to treat a certain symptom, therefore in accordance with one or more embodiments of the subject matter of this disclosure, the system may be employed to measure visually if that symptom is improving. For example, concentration, speed to complete tasks, motor control etc. If symptoms or behavior (eye movement still erratic for example) fail to improve, this may suggest that the user is not taking their medication. Mood may also be measured and monitored as a signal of whether the user is taking their medication. The system may perform various actions to classify a user in one or more medication administration risk category, thus suggesting a level of review and follow up that may be required by the system to follow up with the user.
Additionally, the system may determine a movement of the imaging device, through analysis of information provided in the field of view thereof, or through the use of one or more gyroscopes or accelerometers thereon. Thus, if the user tampers with the position of the imaging device or camera, the system can detect the user's suspicious action and automatically notifies a monitor or a healthcare provider. Other clues, such as changes in a background setting at one or more critical times of medication administration, or other indications of movement of the device, or the like.
After confirmation or failure of confirmation of such administration, the patient may be provided with a progress report regarding how they have performed over time, and further providing encouragement for future adherence. The progress report may include adherence to the medication protocol over time, adherence changes recently, percentage of medication protocol completed, etc. Additionally, notice of a next administration time may be provided, along with one or more messages from a healthcare provider regarding protocol changes, or other desired information. Furthermore, the system may report various detected instances of potentially suspicious behavior to a healthcare provider (e.g., a doctor or a nurse) via a dashboard, and preferably a browser-based dashboard that the healthcare provider can access over the Internet with a user name and password specific to the provider. The provider dashboard allows for the review of near real-time adherence and other information about each patient using the system. The dashboard reports general adherence information along with the noted suspicious activity information. The system may cause a display of a warning message to the patient. The warning message indicates that unusual activity has been detected. The warning message may inform the patient that incorrect use of the medication or protocol may result in their study coordinator/healthcare provider being contacted, their being removed from the trial, or there may be an additional level of monitoring applied to the patient.
Therefore, in accordance with the present disclosure, confirmation of patient adherence to the prescribed administration schedule for the medication as prescribed by the clinical trial or other prescription regimen may be determined by the system, which includes hardware component, while suspicious or malicious behavior may be identified by the system. While such confirmation may take a number of forms, in some embodiments, a method for such confirmation may include capturing a video and audio sequence of the patient actually administering the medication. In a further preferred method, such a sequence for such confirmation may include employing a facial recognition sequence or other biometric confirmation that a particular patient is in fact receiving treatment, but may also provide for the ability to obscure the face or other identifying feature of a user, simplify a displayed image through rotoscoping or the like, or otherwise encrypt such information to allow for the storage and use of such images while protecting the identity of the patient, a technique that may be beneficial when a medication administration manager is providing a general report about a clinical trial, and not trying to remedy a situation with a particular patient, or in particular in a public health or disease management scenario. Activity recognition, gesture recognition, computer vision processing, deep learning processing, or other feature for determining whether a particular subject movement meets a predefined movement sequence may be performed by a computer system to be sure that the patient is properly taking prescribed medication. This same gesture recognition may also be employed to determine suspicious or malicious behavior on the part of the user, as described above. Computerized techniques for performing activity recognition, gesture recognition are known in the art, for example, as described in Niebles, J. C., Wang, H., and Fei-Fei, L. Unsupervised Learning of Human Action Categories Using Spatial-temporal Words. Int J Comput Vis. 16 Mar. 2007. Audio recognition may also be employed by the system, which includes hardware components, to determine suspicious noises such as coughing noises or the like at one or more critical key moments associated with medication administration. For example, if coughing consistently occurs routinely after the user has pace a medication pill in their mouth, then the patient may be escalated to a risk category providing for a higher level of scrutiny of review during the administration process over time. Finally, measurement of time on task, indicative of an amount of time required for the user to perform one or more prescribed steps may also be employed by the system to aid in determining suspicious behavior. This time on task may be measured across many different devices and instances of use of the apparatus to determine one or more trends that may be interesting, and may be used to determine suspicious behavior. For example, time on task may be employed to determine miming of an activity that the user I not really performing the medication administration steps as required.
Referring next to
If the system determines that administration of the medication did not take place properly, processing may return to step 225 and the user may be once again prompted to perform the action. (Of course, if this process involves actual administration of pill, film, inhaler, injectable medication, or any other medication, it may not be proper to request re-performance of the action, unless it can be determined that the user did not actually administer the medication.) If the action has been properly performed and is able to be analyzed, processing ends at step 250 where the various captured video sequences are stored. These stored sequences may also be made available for human review and involvement, when it is determined that this would be beneficial, and may further preferably be employed to determine suspicious or malicious activity. Areas of interest for review may be highlighted and marked as potentially including suspicious or incorrect activity. Percentage estimates of risk may be automatically provided by the system based on risk algorithms that are generated, and based upon a category into which the user may have been placed based upon their prior activity, and one or more learned activities from a plurality of users.
Referring next to
Thus, as is shown in
Elements monitored for suspicious activity include: 1) Hand to Mouth (HTM) during or after the Pill in Mouth step; 2) Mouth not Visible (MNV) during or after the Pill in Mouth (PIM) Step, or when checking for an Empty space under the Tongue when not drinking; 3) Mouth not Visible during or after the Pill in Mouth Step, or when checking for an Empty space under the Tongue and the phone is rotating; 4) When the patient is not drinking and the pill is physically large, and thus hard to dry-swallow. More specifically, if the pill is very large/hard to swallow without liquid, and if the system determines that the patient nonetheless does not have any liquid, the system can determine that this is a suspicious activity; 5) if the patient is found to be spitting out the pill; 6) if the patient is found not to be swallowing; 7) if the patient is found to have a closed mouth between the empty mouth step and the empty under tongue check step; 8) if there is a very short time (e.g., one second or two seconds) between the pill in mouth and the empty mouth step (e.g., the time between the pill in mouth and the empty mouth is less than a predefined time); 9) if there is considerable motion from the muscles of mastication and buccinators (jaw and cheek). Considerable motion refers to motion that is sufficient for the system to determine that the user has placed the pill in their cheek. The system can learn to determine consideration motion over time when trained with examples of patients placing the pill in their cheek.
Elements monitored for cheating activity include: 1) Pill detection after empty mouth; 2) Pill detection after pill in mouth and spitting of the pill; 3) Pill detection after pill in mouth and detection of not swallowing; 4) Miming of the pill in hand step; 5) Generic pill or object in hand is confirmed, while the specific pill of interest in hand is not confirmed; 6) Generic pill or object in mouth is confirmed, while the specific pill of interest in mouth is not confirmed; 7) wrong person is recognized; 8) double enrollment is recognized; and 9) the pill recognized for the pill in hand and the pill in mouth steps do not match.
Finally, elements monitored to confirm missing information include: 1) mouth not visible in either of the pill in mouth or the empty under tongue check; 2) system recognizes it is too bright or too dark, and 3) there is corrupt video or missing frames.
The system may provide responses to the user in real time or near real time in the event that one or more of the elements to be monitored are recognized, thus providing immediate and helpful feedback to the user to properly perform the desired sequence of actions. As noted above, it may be desirable to not report some of the monitoring results to the user, so that a consistent review of action may be provided without alerting the user to adjust their behavior.
As is next shown in
The left column and the bottom row of
The top row of
During the “pill in hand” and “pill in mouth” steps (when a pill is visible), the system can also check whether the pill is still the same as originally imaged (to check for substitution).
In accordance with a further embodiment of the subject matter described in this disclosure, monitored elements may be combined in accordance with an equation of the Bayesian form as follows.
p[cheating data|]∝p[data cheating|]p[cheating]
The equation notes that the probability of cheating being detected is proportional to the likelihood that the data will support an inference of cheating and a prior determination of the probability of cheating. The likelihood that the data will support an inference of cheating is preferably a concatenation of a number of elements, as shown in
R(x/θ) is a regression model parameterized by theta which infers a continuous likelihood value based on the evidence x (concatenation of the features). Θ are the parameters of the model which can be modified over time. In particular, θ can be learned from supervised data. The training data, in accordance with an embodiment of the present disclosure, comprises videos of doses and their corresponding features x. Supervision of the training system and process comes from one or more other computer systems or human annotators and from PK data whenever that is available. Thus, confirmation of status of particular actions from one or more other computer systems or human annotators may be provided to the learning system at the time of acquisition or at a later date in order to teach the system preferred and proper evaluation, allowing the system to in the future also make such accurate and proper evaluations. Examples of implementations of R can be, Support Vector Regression, Random Forest, Logistic Regression, Neural networks, Recurrent neural networks, LSTMs, etc.
Calculation of the prior determination of the probability of cheating is preferably related to one or more of Age, Gender, Race, on any additional attribute found to be related to likelihood of cheating. These additional attributes may be determined in accordance with analysis of prior data related to medication administration and determined cheating. Data may furthermore be collected from one or more patient populations related to therapeutic area, response to medications, co-morbidities, other demographic information, and the like. Such data may also be provided by one or more healthcare providers, sites in clinical trials, or other available databases of information.
The system may provide one or more prompts to a patient related to measurement of any of the elements shown in
Mechanisms for measuring the elements in
The system may further present one or more questions, or other information to the patient in response to a recognition of potentially suspicious activity, and gauge a response. The response can be considered for not only content, but also for status of the user when responding. Thus, it is possible to judge whether the user appears to be nervous and stressed or calm when answering the question, for example. Other issues, such a slurring speech, speeding up or slowing down when speaking may also be correlated with the responses from the user and ultimately correlating with an actual determination of whether the user is properly administering the medication, or is cheating/performing suspicious activity.
Users may be associated with a score indicative of medication adherence, likelihood of future proper medication adherence or the like. Through such a scoring system, it is possible to easily categorize and manage users so that similarly-scored users can be provided with similar intervention, follow up and scrutiny.
Deep learning and other supervised or unsupervised learning techniques may be employed to improve the responsiveness of the system, and to extract patterns that may be predictive of future action on the part of users. Thus, such expert systems may review input data in an automated fashion, and provide correlations between the data that are predictive in the future moving forward.
In a further embodiment, it is possible to provide a unique serialization of medication so that the system may confirm that each and every dose is correct. Thus, each bottle, or each dose, may be imprinted with a unique identifying barcode, QR code, data matrix code, or other code that allows for a reader to determine an identity based upon a visual capture of the code. Codes may be assigned to define categories of pills, individual groups of doses, or individual pills, and may further be customized to allow for the assignment of a particular pill to be correlated with a single individual. Thus, facial recognition results may be combined with QR code results from a bottle or pill to confirm that the correct pill is being taken by the corresponding correct person.
According to another aspect, once the analyses noted above have been performed, in order to encourage the system to learn more about human behavior, measurements of actual drug levels, for example, may be determined and fed back into the system. Therefore, by way of example, if during monitoring, the system finds a particular patient who have caused one or more of the elements for suspicious activity to have been recognized, subsequent to this process, the system may determine actual drug levels in the blood of the patients. This is an accurate depiction of whether the patient has ingested the medication, at least on the few days before the blood was drawn (depending on half-life of drug). Thus, it is possible to correlate recognition of suspicious activity elements (or other elements) with actual drug blood levels. The system may recognize patterns or groups of elements that correlate with lack of drug in the blood, and then use the patterns or groups of elements in the future to better predict the likely failure to properly administer medication. Furthermore, as additional information becomes available, such as in a clinical trial where it might be years before knowledge of actual medication blood levels may be determined, data may be reprocessed to determine any correlations between any one or more of the monitored elements and actual drug levels.
Therefore, in accordance with the present disclosure, a method and apparatus are provided that allow for the automated confirmation of adherence to administration protocol for medication, and provide for a most sophisticated method for confirming and studying methods of administration of such prescription medication. The system described in this disclosure can monitor suspicious activity, cheating, and other elements in order to insure proper administration, and can learn to improve its performance based upon actual use of the system by patients.
The remote data and computing device 3000 comprises system management functions 3030, and a transmission and reception system 3050 and 3060 for communicating with apparatus 1000. Transmission and reception system 3050 and 3060 may further comprise various GPS modules so that a location of the device can be determined at any time, and may further allow for a message to be sent to one or more individual apparatuses, broadcast to all apparatuses in a particular trial, or being used for administration of a particular prescription regimen, of broadcast to all available apparatuses.
In accordance with an embodiment of the disclosure, apparatus 1000 is adapted to be part of a system that monitors user (patient) medication administration, and determined that the correct user is administering the correct medication at the correct time. Users of apparatus 1000 in accordance with this system give administrators a tangible and concrete manner in which to review activities and collected information related to proper medication administration. Apparatus 1000 of the disclosure is adapted to receive medication protocol and instructions for medication administration, or the like for patients from remote data and computing device 3000 and provide these instructions to patients to guide them in administering their medication. Such instructions may comprise written, audio or audio/video instructions for guiding a user to perform one or more medication administration activities, such as showing a face of the user, showing a pill to be ingested, or confirming that the medication has been properly ingested, and therefore confirming whether a user is adhering to a prescribed medication protocol. The video instructions can be provided either by a real person or by an animated cartoon character (avatar), or the like, and are designed to be interactive in nature.
It will thus be seen that the objects set forth above, among those made apparent from the preceding description, are efficiently attained and, because certain changes may be made in carrying out the above method and in the construction(s) set forth without departing from the spirit and scope of the subject matter described in this disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
It is also to be understood that this description is intended to cover all of the generic and specific features of the subject matter described in this disclosure herein described and all statements of the scope of the subject matter described in this disclosure which, as a matter of language, might be said to fall there between.
Embodiments of the subject matter and the functional operations, for example, the computer system in
The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
A computer program, which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser.
Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the user device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received from the user device at the server.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any subject matter described in this disclosure or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of the subject matter described in this disclosure. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.
This application claims priority to U.S. Provisional Application Ser. No. 62/560,528, filed on Sep. 19, 2017. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3814845 | Hurlbrink et al. | Jun 1974 | A |
| 5065447 | Barnsley et al. | Nov 1991 | A |
| 5441047 | David et al. | Aug 1995 | A |
| 5544649 | David et al. | Aug 1996 | A |
| 5619991 | Sloane | Apr 1997 | A |
| 5646912 | Cousin | Jul 1997 | A |
| 5752621 | Passamante | May 1998 | A |
| 5764296 | Shin | Jun 1998 | A |
| 5810747 | Brudny et al. | Sep 1998 | A |
| 5911132 | Sloane | Jun 1999 | A |
| 5961446 | Beller et al. | Oct 1999 | A |
| 6151521 | Guo et al. | Nov 2000 | A |
| 6233428 | Fryer | May 2001 | B1 |
| 6283761 | Joao | Sep 2001 | B1 |
| 6380858 | Yarin et al. | Apr 2002 | B1 |
| 6409661 | Murphy | Jun 2002 | B1 |
| 6421650 | Goetz et al. | Jul 2002 | B1 |
| 6483993 | Misumi et al. | Nov 2002 | B1 |
| 6484144 | Martin et al. | Nov 2002 | B2 |
| 6535637 | Wootton et al. | Mar 2003 | B1 |
| 6611206 | Milanski et al. | Aug 2003 | B2 |
| 6705991 | Bardy | Mar 2004 | B2 |
| 6879970 | Shiffman et al. | Nov 2005 | B2 |
| 6988075 | Hacker | Jan 2006 | B1 |
| 7184047 | Crampton | Feb 2007 | B1 |
| 7184075 | Reiffel | Feb 2007 | B2 |
| 7256708 | Rosenfeld et al. | Aug 2007 | B2 |
| 7277752 | Matos | Oct 2007 | B2 |
| 7304228 | Bryden et al. | Dec 2007 | B2 |
| 7307543 | Rosenfeld et al. | Dec 2007 | B2 |
| 7317967 | DiGianfilippo et al. | Jan 2008 | B2 |
| 7340077 | Gokturk | Mar 2008 | B2 |
| 7369919 | Vonk et al. | May 2008 | B2 |
| 7395214 | Shillingburg | Jul 2008 | B2 |
| 7415447 | Shiffman et al. | Nov 2008 | B2 |
| 7447333 | Masticola et al. | Nov 2008 | B1 |
| 7448544 | Louie et al. | Nov 2008 | B1 |
| 7562121 | Berisford et al. | Jul 2009 | B2 |
| 7627142 | Kurzweil et al. | Dec 2009 | B2 |
| 7657443 | Crass et al. | Feb 2010 | B2 |
| 7692625 | Morrison et al. | Apr 2010 | B2 |
| 7740013 | Ishizaki et al. | Jun 2010 | B2 |
| 7747454 | Bartfeld et al. | Jun 2010 | B2 |
| 7761311 | Clements et al. | Jul 2010 | B2 |
| 7769465 | Matos | Aug 2010 | B2 |
| 7774075 | Lin et al. | Aug 2010 | B2 |
| 7874984 | Elsayed et al. | Jan 2011 | B2 |
| 7881537 | Ma et al. | Feb 2011 | B2 |
| 7908155 | Fuerst et al. | Mar 2011 | B2 |
| 7912733 | Clements et al. | Mar 2011 | B2 |
| 7956727 | Loncar | Jun 2011 | B2 |
| 7983933 | Karkanias et al. | Jul 2011 | B2 |
| 8321284 | Clements et al. | Nov 2012 | B2 |
| 9679113 | Hanina | Jun 2017 | B2 |
| 20010049673 | Dulong et al. | Dec 2001 | A1 |
| 20010056358 | Dulong et al. | Dec 2001 | A1 |
| 20020026330 | Klein | Feb 2002 | A1 |
| 20020027507 | Yarin et al. | Mar 2002 | A1 |
| 20020093429 | Matsushita et al. | Jul 2002 | A1 |
| 20020143563 | Hufford et al. | Oct 2002 | A1 |
| 20030164172 | Chumas et al. | Sep 2003 | A1 |
| 20030190076 | Delean | Oct 2003 | A1 |
| 20030225325 | Kagermeier et al. | Dec 2003 | A1 |
| 20040100572 | Kim | May 2004 | A1 |
| 20040107116 | Brown | Jun 2004 | A1 |
| 20040155780 | Rapchak | Aug 2004 | A1 |
| 20040168951 | Mackie | Sep 2004 | A1 |
| 20050144150 | Ramamurthy et al. | Jun 2005 | A1 |
| 20050149361 | Saus et al. | Jul 2005 | A1 |
| 20050180610 | Kato et al. | Aug 2005 | A1 |
| 20050182664 | Abraham-Fuchs et al. | Aug 2005 | A1 |
| 20050234381 | Niemetz et al. | Oct 2005 | A1 |
| 20050267356 | Ramasubramanian et al. | Dec 2005 | A1 |
| 20060066584 | Barkan | Mar 2006 | A1 |
| 20060218011 | Walker et al. | Sep 2006 | A1 |
| 20060238549 | Marks | Oct 2006 | A1 |
| 20070008112 | Covannon et al. | Jan 2007 | A1 |
| 20070008113 | Spoonhower et al. | Jan 2007 | A1 |
| 20070016443 | Wachman et al. | Jan 2007 | A1 |
| 20070030363 | Cheatle et al. | Feb 2007 | A1 |
| 20070118054 | Pinhas et al. | May 2007 | A1 |
| 20070118389 | Shipon | May 2007 | A1 |
| 20070194034 | Vasiadis | Aug 2007 | A1 |
| 20070233035 | Wehba et al. | Oct 2007 | A1 |
| 20070233049 | Wehba et al. | Oct 2007 | A1 |
| 20070233050 | Wehba et al. | Oct 2007 | A1 |
| 20070233281 | Wehba et al. | Oct 2007 | A1 |
| 20070233520 | Wehba et al. | Oct 2007 | A1 |
| 20070233521 | Wehba et al. | Oct 2007 | A1 |
| 20070273504 | Tran | Nov 2007 | A1 |
| 20070276270 | Tran | Nov 2007 | A1 |
| 20080000979 | Poisner | Jan 2008 | A1 |
| 20080086533 | Neuhauser et al. | Apr 2008 | A1 |
| 20080093447 | Johnson et al. | Apr 2008 | A1 |
| 20080114226 | Music et al. | May 2008 | A1 |
| 20080114490 | Jean-Pierre | May 2008 | A1 |
| 20080138604 | Kenney et al. | Jun 2008 | A1 |
| 20080140444 | Karkanias et al. | Jun 2008 | A1 |
| 20080162192 | Vonk et al. | Jul 2008 | A1 |
| 20080178126 | Beeck et al. | Jul 2008 | A1 |
| 20080201174 | Ramasubramanian | Aug 2008 | A1 |
| 20080219493 | Tadmor | Sep 2008 | A1 |
| 20080275738 | Shillingburg | Nov 2008 | A1 |
| 20080279420 | Masticola et al. | Nov 2008 | A1 |
| 20080281630 | Sekura | Nov 2008 | A1 |
| 20080290168 | Sullivan et al. | Nov 2008 | A1 |
| 20080297589 | Kurtz et al. | Dec 2008 | A1 |
| 20080303638 | Nguyen et al. | Dec 2008 | A1 |
| 20090012818 | Rodgers | Jan 2009 | A1 |
| 20090018867 | Reiner | Jan 2009 | A1 |
| 20090043610 | Nadas et al. | Feb 2009 | A1 |
| 20090048871 | Skomra | Feb 2009 | A1 |
| 20090062624 | Neville | Mar 2009 | A1 |
| 20090095837 | Lindgren | Apr 2009 | A1 |
| 20090128330 | Monroe | May 2009 | A1 |
| 20090149721 | Yang | Jun 2009 | A1 |
| 20090159714 | Coyne, III et al. | Jun 2009 | A1 |
| 20090217194 | Martin et al. | Aug 2009 | A1 |
| 20090245655 | Matsuzaka | Oct 2009 | A1 |
| 20100042430 | Bartfeld | Feb 2010 | A1 |
| 20100050134 | Clarkson | Feb 2010 | A1 |
| 20100057646 | Martin et al. | Mar 2010 | A1 |
| 20100092093 | Akatsuka et al. | Apr 2010 | A1 |
| 20100136509 | Mejer et al. | Jun 2010 | A1 |
| 20100138154 | Kon | Jun 2010 | A1 |
| 20100255598 | Melker | Oct 2010 | A1 |
| 20100262436 | Chen et al. | Oct 2010 | A1 |
| 20100316979 | Von Bismarck | Dec 2010 | A1 |
| 20110021952 | Vallone | Jan 2011 | A1 |
| 20110082712 | Eberhardt | Apr 2011 | A1 |
| 20110119073 | Hanina et al. | May 2011 | A1 |
| 20110153360 | Hanina et al. | Jun 2011 | A1 |
| 20110161109 | Pinsonneault et al. | Jun 2011 | A1 |
| 20110195520 | Leider et al. | Aug 2011 | A1 |
| 20110275051 | Hanina et al. | Nov 2011 | A1 |
| 20120009555 | Hanina | Jan 2012 | A1 |
| 20120075464 | Derenne et al. | Mar 2012 | A1 |
| 20120121729 | Paterson | May 2012 | A1 |
| 20120316897 | Hanina | Dec 2012 | A1 |
| 20130044196 | Gualn et al. | Feb 2013 | A1 |
| 20140184772 | Hanina | Jul 2014 | A1 |
| 20140376876 | Bentley | Dec 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 102612703 | Jul 2012 | CN |
| WO 2004103232 | Dec 2004 | WO |
| WO 2011062934 | May 2011 | WO |
| WO 2012177524 | Dec 2012 | WO |
| WO 2014152828 | Sep 2014 | WO |
| Entry |
|---|
| International Search Report and Written Opinion for Int. App No. PCT/US2018/051669, dated Nov. 21, 2018 (8 pages). |
| Ammouri and Biloduau, Face and Hands Detection and Tracking Applied to the Monitoring of Medication Intake, Computer and Robot Vision, 2008. CRV '08. Canadian Conference pp. 147-154, (May 28-30, 2008). |
| Batz, et al. A computer Vision System for Monitoring Medication Intake, in Proc. IEEE 2nd Canadian Conf. on Computer and Robot Vision, Victoria, BC, Canada, (2005) (8 pages). |
| Bilodeau et al. Monitoring of Medication Intake Using a Camera System, Journal of Medical Systems, [retrieved on Feb. 18, 2013] Retrieved from ProQuest Technology Collection (2011) (13 pages). |
| Chen, Pauline W., Texting as a Health Tool for Teenagers, The New York Times, Nov. 5, 2009, http://www.nytimes.com/2009/11/05/health/05chen.html?_r=1&emc (3 pages). |
| CN Office Action for CN App No. 201480014372.8 dated Aug. 2, 2018 (with English translation) (21 pages). |
| CN Office Action for CN App. No. 2014800143728, dated Jul. 31, 2017 (9 pages). |
| Danya International Inc., Pilot Study Using Cell Phones for Mobile Direct Observation Treatment to Monitor Medication Compliance of TB Patients, Mar. 20, 2009, www.danya.com/MDOT.asp (2 pages). |
| EP Supplementary European Search Report for EP 14770974, dated Aug. 22, 2016 (2 pages). |
| Huynh et al., Real time detection, tracking and recognition of medication intake World Academy of Science, Engineering and Technology 60: 280-287 (2009). |
| International Search Report in PCT/US14/27901, dated Aug. 19, 2014, p. 1-9. |
| Mintchell, Exploring the Limits of Machine Vision, Automating World, (Oct. 1, 2011) (6 pages). |
| Niebles, et al., Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words, Int J Comput Vis. (Mar. 16, 2007) (20 pages). |
| Ostergerg and Blaschke, Adherence to Medication, New England Journal of Medicine 353:487-97 (Aug. 4, 2005). |
| University of Texas, GuideView: When the physician is really far away, Mar. 15, 2007, http://www.sahs.uth.tmc.edu/MSriram/GuideView (3 pages). |
| Valin, et al. Video Surveillance of Medication intake, Int. Conf. of the IEEE Engineering in Medicine and Biology Society, New York City, USA, Aug. 2006 (4 pages). |
| Wang et al. Recent Developments in human motion analysis Pattern Recognition 36: 585-601 (2003). |
| Whitecup, Morris S., 2008 Patient Adherence Update: New Approaches for Success, Guideline Trend Report, Oct. 2008 (17 pages). |
| Wikipedia, Super-Resolution, (Oct. 5, 2010) (4 pages). |
| World Health Organization, Global Tuberculosis Control: A short update to the 2009 report, (2009) (48 pages). |
| Extended European Search Report in EP Appln. No. 18858751, dated Sep. 9, 2020, 6 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20190087951 A1 | Mar 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62560528 | Sep 2017 | US |
