Personalized medicine has the potential to detect various possible diseases, disorders, or conditions that are personalized to the individual patient based on molecular profiling. However, many challenges remain in generating therapeutically useful diagnoses.
Described herein are platforms, systems, media, and methods for assessing an individual for one or more diseases, disorders, or conditions. A machine learning algorithm can be used to provide the assessment based on personalized data derived from the individual. The personalized data can include metabolite data from a specimen or biological sample of the individual.
In one aspect, described herein is a system for assessing an individual, comprising: (a) a processor; (b) a non-transitory computer readable medium encoded with software comprising one or more machine learning algorithms together with instructions configured to cause the processor to: (i) receive data related to a specimen taken from the individual; and (ii) provide the data as input to the one or more machine learning algorithms, wherein the one or more machine learning algorithms use the data to generate a classification of the individual relative to a plurality of related classifications. In some instances, the one or more machine learning algorithms comprise an ensemble of machine learning algorithms. Sometimes, the ensemble comprises at least three machine learning algorithms. In certain cases, the ensemble of machine learning algorithms comprises a Generalized Linear algorithm, a Random Forests algorithm, a Partial Least Squares algorithm, and Extreme Gradient Boosting algorithm, a Support Vector Machines with Linear Basis Function Kernel algorithm, a Support Vector Machines with Radial Basis Function Kernel, and a Neural Networks algorithm. In some cases, each machine learning algorithm of the ensemble of machine learning algorithms produces an output that is averaged by the software. Sometimes, each machine learning algorithm of the ensemble of machine learning algorithms produces an output and wherein at least one output is an input for at least one of the machine learning algorithms. In certain instances, the at least one machine learning algorithm is trained using data relating to specimens from other individuals. Oftentimes, the specimen comprises a biological sample. In some cases, the specimen comprises at least one of a sputum sample, a urine sample, a blood sample, a cerebrospinal fluid sample, a stool sample, a hair sample, and a biopsy. The data often relates to a metabolite. In certain instances, the metabolite comprises at least one of oleamide, creatine, and 4-methyl-2-oxopentanoate. Sometimes, the instructions are further configured to cause the processor to receive a parameter related to the individual and wherein the one or more machine learning algorithms use the parameter together with the data to generate the classification of the individual relative to the plurality of related classifications. The parameter often comprises at least one of an age, a gender, a race, a weight, a BMI, a height, a waist size, a blood pressure, a heart rate, and a temperature. Sometimes, the classification comprises a disease. In various instances, the disease comprises at least one of multiple sclerosis, amyotrophic lateral sclerosis, systemic lupus erythematosus, fibromyalgia, and gastrointestinal reflux disease. In certain cases, the plurality of related classifications comprise a spectrum of severity of a single disease. Sometimes, the plurality of related classifications comprise a spectrum of prognoses of a single disease. In certain instances, the plurality of related classifications comprise a spectrum of related diseases. The spectrum of related diseases comprise a plurality of neurological diseases that share at least one common feature, in various cases.
In another aspect, disclosed herein is a computer implemented method for assessing an individual, comprising: (a) receiving data relating to a specimen taken from the individual; (b) providing the data as input to one or more machine learning algorithms; and (c) generating, using the one or more machine learning algorithms, a classification of the individual relative to a plurality of related classifications based on the data. In some instances, the one or more machine learning algorithms comprise an ensemble of machine learning algorithms. Sometimes, the ensemble comprises at least three machine learning algorithms. In certain cases, the ensemble of machine learning algorithms comprises a Generalized Linear algorithm, a Random Forests algorithm, a Partial Least Squares algorithm, and Extreme Gradient Boosting algorithm, a Support Vector Machines with Linear Basis Function Kernel algorithm, a Support Vector Machines with Radial Basis Function Kernel, and a Neural Networks algorithm. In some cases, each machine learning algorithm of the ensemble of machine learning algorithms produces an output that is averaged by the software. Sometimes, each machine learning algorithm of the ensemble of machine learning algorithms produces an output and wherein at least one output is an input for at least one of the machine learning algorithms. In certain instances, the at least one machine learning algorithm is trained using data relating to specimens from other individuals. Oftentimes, the specimen comprises a biological sample. In some cases, the specimen comprises at least one of a sputum sample, a urine sample, a blood sample, a cerebrospinal fluid sample, a stool sample, a hair sample, and a biopsy. The data often relates to a metabolite. In certain instances, the metabolite comprises at least one of oleamide, creatine, and 4-methyl-2-oxopentanoate. Sometimes, the instructions are further configured to cause the processor to receive a parameter related to the individual and wherein the one or more machine learning algorithms use the parameter together with the data to generate the classification of the individual relative to the plurality of related classifications. The parameter often comprises at least one of an age, a gender, a race, a weight, a BMI, a height, a waist size, a blood pressure, a heart rate, and a temperature. Sometimes, the classification comprises a disease. In various instances, the disease comprises at least one of multiple sclerosis, amyotrophic lateral sclerosis, systemic lupus erythematosus, fibromyalgia, and gastrointestinal reflux disease. In certain cases, the plurality of related classifications comprise a spectrum of severity of a single disease. Sometimes, the plurality of related classifications comprise a spectrum of prognoses of a single disease. In certain instances, the plurality of related classifications comprise a spectrum of related diseases. The spectrum of related diseases comprise a plurality of neurological diseases that share at least one common feature, in various cases.
In another aspect, disclosed herein is a system for assessing an individual, comprising: (a) a processor; (b) a non-transitory computer readable medium encoded with software comprising one or more machine learning algorithms together with instructions configured to cause the processor to: (i) receive data related to a specimen taken from the individual; and (ii) provide the data as input to the one or more machine learning algorithms, wherein the one or more machine learning algorithms use the data to generate an assessment of one or more traits of the individual. In some cases, the one or more machine learning algorithms comprise an ensemble of machine learning algorithms. Sometimes, the ensemble comprises at least three machine learning algorithms. In some aspects, the ensemble of machine learning algorithms comprises a Generalized Linear algorithm, a Random Forests algorithm, a Partial Least Squares algorithm, and Extreme Gradient Boosting algorithm, a Support Vector Machines with Linear Basis Function Kernel algorithm, a Support Vector Machines with Radial Basis Function Kernel, and a Neural Networks algorithm. In certain instances, each machine learning algorithm of the ensemble of machine learning algorithms produces an output that is averaged by the software. In various aspects, each machine learning algorithm of the ensemble of machine learning algorithms produces an output and wherein at least one output is an input for at least one of the machine learning algorithms. In certain cases, at least one machine learning algorithm is trained using data relating to specimens from other individuals. Sometimes, the specimen comprises a biological sample. In some instances, the specimen comprises at least one of a sputum sample, a urine sample, a blood sample, a cerebrospinal fluid sample, a stool sample, a hair sample, and a biopsy. In certain aspects, the data relates to a metabolite, a protein, a nucleic acid, or any combination thereof. In various cases, the metabolite comprises at least one of oleamide, creatine, and 4-methyl-2-oxopentanoate. Sometimes, the instructions are further configured to cause the processor to receive a parameter related to the individual and wherein the one or more machine learning algorithms use the parameter together with the data to generate the assessment of the individual. In some cases, the parameter comprises at least one of an age, a gender, a race, a weight, a BMI, a height, a waist size, a blood pressure, a heart rate, and a temperature. In certain aspects, the assessment comprises at least one trait selected from a category that is personal characteristics, general health, mental health, health behaviors, interventions, organ systems, environmental, or conditions. In some instances, the one or more traits comprises at least one of sex, age, BMI, race, ethnicity, personality, traits, family history, current, conditions, acute infection, allergies, perceived health, circadian cycle, menstrual cycle, genetic predisposition, thrive, cognition, energy, depression, anxiety, stress, coping ability, feels good or bad, fitness, substances, sleep, diet, sun exposure, sex drive, vaccines, treatment, procedures, supplement, circulatory, dental, digestive, endocrine, lymph or immune system, metabolism, musculoskeletal system, nervous system, renal system, reproductive system, respiratory system, skin, life events including trauma, living environment, work environment, chemical, exposures, social functioning, diagnostic history, disease severity, symptoms and signs, potential, complications, and, comorbidities, monitoring labs and tests, or treatment.
In another aspect, disclosed herein is a computer implemented method for assessing an individual, comprising: (a) receiving data relating to a specimen taken from the individual; (b) providing the data as input to one or more machine learning algorithms; and (c) generating, using the one or more machine learning algorithms, an assessment of one or more traits of the individual. In some cases, the one or more machine learning algorithms comprise an ensemble of machine learning algorithms. Sometimes, the ensemble comprises at least three machine learning algorithms. In some aspects, the ensemble of machine learning algorithms comprises a Generalized Linear algorithm, a Random Forests algorithm, a Partial Least Squares algorithm, and Extreme Gradient Boosting algorithm, a Support Vector Machines with Linear Basis Function Kernel algorithm, a Support Vector Machines with Radial Basis Function Kernel, and a Neural Networks algorithm. In certain instances, each machine learning algorithm of the ensemble of machine learning algorithms produces an output that is averaged by the software. In various aspects, each machine learning algorithm of the ensemble of machine learning algorithms produces an output and wherein at least one output is an input for at least one of the machine learning algorithms. In certain cases, at least one machine learning algorithm is trained using data relating to specimens from other individuals. Sometimes, the specimen comprises a biological sample. In some instances, the specimen comprises at least one of a sputum sample, a urine sample, a blood sample, a cerebrospinal fluid sample, a stool sample, a hair sample, and a biopsy. In certain aspects, the data relates to a metabolite, a protein, a nucleic acid, or any combination thereof. In various cases, the metabolite comprises at least one of oleamide, creatine, and 4-methyl-2-oxopentanoate. Sometimes, the method further comprises receiving a parameter related to the individual and wherein the one or more machine learning algorithms use the parameter together with the data to generate the assessment of the individual. In some cases, the parameter comprises at least one of an age, a gender, a race, a weight, a BMI, a height, a waist size, a blood pressure, a heart rate, and a temperature. In certain aspects, the assessment comprises at least one trait selected from a category that is personal characteristics, general health, mental health, health behaviors, interventions, organ systems, environmental, or conditions. In some instances, the one or more traits comprises at least one of sex, age, BMI, race, ethnicity, personality, traits, family history, current, conditions, acute infection, allergies, perceived health, circadian cycle, menstrual cycle, genetic predisposition, thrive, cognition, energy, depression, anxiety, stress, coping ability, feels good or bad, fitness, substances, sleep, diet, sun exposure, sex drive, vaccines, treatment, procedures, supplement, circulatory, dental, digestive, endocrine, lymph or immune system, metabolism, musculoskeletal system, nervous system, renal system, reproductive system, respiratory system, skin, life events including trauma, living environment, work environment, chemical, exposures, social functioning, diagnostic history, disease severity, symptoms and signs, potential, complications, and, comorbidities, monitoring labs and tests, or treatment.
The novel features of the invention are set forth with particularity in the appended claims. The file of this patent contains at least one drawing/photograph executed in color. Copies of this patent with color drawing(s)/photograph(s) will be provided by the Office upon request and payment of the necessary fee. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which:
Described herein are devices, software, systems, and methods for assessing an individual for a disease, disorder, or condition by generating a classification relative to a plurality of related classifications based on data obtained from the individual. More specifically, the data comprises metabolite data suitable for detecting at least one of the presence, severity, duration, or status of a disease, disorder, or condition. In some instances, the metabolite data is obtained from a biological sample of the individual and evaluated to determine the presence and/or quantitation of one or more metabolites in the sample. The metabolite data may be obtained multiple times from the individual to enable monitoring over time. The data can also include non-metabolite data such as nucleic acid sequencing and/or expression data. In some cases, the data includes protein or polypeptide data such as expression or quantitation level for a panel of proteins or polypeptides. Machine learning algorithms can be trained to generate classifiers or models that comprise a panel or list of features such as discriminating metabolites or other biomarkers. Multiple machine learning algorithms may be utilized to assess the sample. In some cases, an Ensemble classifier that consolidates two or more machine learning algorithms is used to generate the classification. The classification can include a grade, severity, or class of a particular disease, disorder, or condition. In some embodiments, the systems, devices, software, and methods described herein are configured to identify a diagnostic modality that should be used as an additional step in evaluating an individual who is found by the systems, devices, software, and methods to have a particular disease, disorder, or condition. In some embodiments, the systems, devices, software, and methods described herein are configured to identify a therapy for an individual based on the results of the classification.
Disease Scoring or Classification
In some aspects, described herein are devices, software, systems, and methods for providing disease scoring or classification for an individual based on data such as a molecular profile. An individual's molecular profile can be compared to a broad spectrum of disease, disorder, or condition-associated profiles to generate one or more scores or matches using a classifier or model. The molecular profile can be a metabolite profile comprising one or more metabolites. The metabolites can be associated with one or more metabolic pathways such as, for example, lipid, carbohydrate, or protein metabolism. In some embodiments, the molecular profile comprises a metabolite profile, a protein/polypeptide profile, a gene expression profile, or any combination thereof. In some embodiments, the protein/polypeptide profile comprises quantification or abundance data for one or more proteins or polypeptides. In some embodiments, the gene expression profile comprises RNA sequencing data for one or more biomarkers.
The disease, disorder, or condition-associated profiles can correspond to a plurality of related classifications. In some cases, the related classifications share at least one common feature. In certain aspects, the algorithms described herein provide a classification that stratifies a disease, disorder, or condition. The stratification can be based on severity, grade, class, prognosis, or treatment of a particular disease, disorder, or condition, and/or other relevant factors. In some cases, a subject can be classified for a spectrum of a plurality of diseases, disorders, or conditions, which are optionally further classified into subcategories of the diseases, disorders, or conditions (e.g., subtypes or varying degrees of severity of a disease). For example, autoimmune diseases may be further subcategorized based on biomarkers such as one or more of the metabolite biomarkers disclosed herein.
An individual specimen such as a biological sample can be evaluated to generate a metabolite profile. The metabolite profile can be classified on a spectrum of a plurality of diseases, disorders, or conditions. In some cases, the classification is generated using classifiers trained using one or more machine learning algorithms. Sometimes, the classification comprises a score and/or indicator of the accuracy or confidence of the classification. In certain instances, the score is produced by ensemble machine learning methods, trained to a variety of complex patterns that are tightly associated with disease conditions reported by other individuals or patients. The classification can include a probability that a new sample belongs to a previously learned class of patient-reported outcomes.
The score can be used to evaluate individual disease states and track signs of progress or decline associated with given conditions and interventions, over periods of time. In some cases, a spectrum of multiple classifications are generated for an individual using one or more machine learning algorithms or classifiers. The spectrum of multiple classifications can comprise a plurality of classifications that are related, for example, sharing one or more common predictive features. As an example, MS and ALS share common features, which can lead to misclassification between MS and ALS positive cases. Thus, the generation of a spectrum of multiple classifications can help identify, resolve, and/or mitigate misclassifications between related diseases, disorders, or conditions. In some cases, a spectrum classification comprises a classification between two or more related classifications with a score and/or confidence or likelihood that the individual is positive for one or more of the related classifications. For example, the spectrum classification can be a score indicating a relative likelihood the individual has MS vs ALS (e.g., 35% MS score vs. 65% ALS score). In some instances, the spectrum classification comprises two or more of gastroesophageal reflux disease, bipolar disorder, amyotrophic lateral sclerosis, osteoarthritis, multiple sclerosis, fibromyalgia, systemic lupus erythematosus, generalized anxiety disorder, rheumatoid arthritis, major depressive disorder, high blood pressure hypertension, hypothyroidism, or post-traumatic stress disorder (see
In some cases, the classifications for the plurality of multiple classification is output as a spectrum of various diseases, disorders, or conditions corresponding to the classifications. The output can be shown as a diagram indicating the score (e.g., as a percentage) of the individual overlaid over the “normal” score range corresponding to non-positive individuals (see
Metabolite profiles can be generated for two or more specimens obtained from an individual over a period of time. The metabolite profiles can be evaluated using the methods described herein to generate a classification or a spectrum of related classifications. The classification or spectrum of related classifications can be compared between specimens to assess an individual over a period of time. The period of time can be at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60 minutes and/or no more than 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60 minutes. In some cases, the period of time is at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, or 24 hours and/or no more than 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, or 24 hours. In certain instances, the period of time is at least 1, 2, 3, 4, 5, 6, or 7 days and/or no more than 1, 2, 3, 4, 5, 6, or 7 days. The period of time can be at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, or 52 weeks and/or no more than 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, or 52 weeks. Sometimes, the period of time is at least 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 years and/or no more than 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10 years.
The accuracy, specificity, sensitivity, positive predictive value, negative predictive value, or any combination thereof may be determined for a classifier by testing it against a set of independent samples. True positive (TP) is a positive test result that detects the condition when the condition is present. True negative (TN) is a negative test result that does not detect the condition when the condition is absent. False positive (FP) is a test result that detects the condition when the condition is absent. False negative (FN) is a test result that does not detect the condition when the condition is present. Accuracy is defined by the formula: accuracy=(TP+TN)/(TP+FP+FN+TN). Specificity (“true negative rate”) is defined by the formula: specificity=TN/(TN+FP). Sensitivity (“true positive rate”) is defined by the formula: sensitivity=TP/(TP+FN). Positive predictive value (PPV or “precision”) is defined by the formula: PPV=TP/(TP+FP). Negative predictive value (NPV) is defined by the formula: NPV=TN/(TN+FN).
In some cases, an individual or sample is classified with respect to one or more diseases, disorders, or conditions with an accuracy of at least about 80%, 85%, 90%, 95%, 96%, 97%, 98%, or 99% for at least 100, 150, or 200 independent samples. In some cases, an individual or sample is classified with respect to one or more diseases, disorders, or conditions with an specificity of at least about 80%, 85%, 90%, 95%, 96%, 97%, 98%, or 99% for at least 100, 150, or 200 independent samples. In some cases, an individual or sample is classified with respect to one or more diseases, disorders, or conditions with a sensitivity of at least about 80%, 85%, 90%, 95%, 96%, 97%, 98%, or 99% for at least 100, 150, or 200 independent samples. In some cases, an individual or sample is classified with respect to one or more diseases, disorders, or conditions with an AUC of at least about 0.80, 0.85 0.90, 0.95, 0.96, 0.97, 0.98, or 0.99 for at least 100, 150, or 200 independent samples.
Health data of an individual presented to a healthcare provider by the systems, media, or methods as described herein may include the diagnoses or classification, treatment regimen, and/or outcome of the individual. Non-limiting examples of health data presented to a healthcare provider may include metabolite data, classification, other data such as non-molecular/metabolite data, and therapeutic options.
In some embodiments an insight generated by the systems, media, or methods as described herein comprises one or more treatment regimens. For example, the system may present a treatment regimen to a healthcare provider that was deemed successful for subjects having metabolite profiles similar to that of the individual evaluated according to the systems, media, and methods described herein. A treatment regimen may be deemed successful, in some embodiments, when a goal of the patient is achieved through the application of the treatment regimen. An example of a treatment is diet, exercise, and statins for an individual determined to be overweight or obese and having atherosclerosis or heart disease.
Diseases, Disorders, and Conditions
Disclosed herein are algorithms, classifiers, or models that generate classifications of individuals based on input data. The classifications can correspond or relate to one or more diseases, disorders, or conditions. A disease can be identified as abnormalities or dysfunctions in systemic functions. A disorder can be identified as a disruption of the normal functions of the body. Accordingly, a disorder can be the resulting disruption caused by a disease in certain cases. A condition can be identified as an abnormal state of health, including states that interfere with normal activities or well-being of the individual. These categories may exhibit some overlap.
In some cases, the classification corresponds or relates to a neurological and/or autoimmune disease such as, for example, multiple sclerosis, systemic lupus erythematosus, or amyotrophic lateral sclerosis.
In some cases, the systems, media and methods disclosed herein provide a prediction or recommendation for treatment based on the classification or evaluation of one or more diseases, disorders, or conditions. In some cases, a report is generated comprising one or more findings such as the results of the classification or evaluation. In some cases, the report comprises or more diagnoses. In some cases, the report comprises one or more treatments or treatment recommendations. In some cases, the methods disclosed herein comprise providing treatment to the subject. In some instances, treatment is provided based at least on the classification or evaluation. The treatment can be a particular treatment for the one or more diseases, disorders, or conditions, for example, autoimmune diseases or disorders may be treated using an anti-inflammatory medication (e.g., acetaminophen, NSAIDs such as ibuprofen), corticosteroids (e.g., hydrocortisone, dexamethasone, prednisone, methylprednisolone, betamethasone), antimalarial drugs for skin and joint problems (e.g., hydroxychloroquine), immunosuppressants (e.g., azathioprine, mycophenolate mofetil, methotrexate), biologics such as antibodies (e.g., belimumab for treating SLE, and rituximab for treating MS, SLE, etc.), or any combination thereof. In some cases, the treatment comprises a drug treatment. Alternatively or in combination, the treatment comprises lifestyle changes such as to diet and/or exercise, or other non-pharmaceutical therapies. In some cases, the treatment comprises a drug treatment or therapy for a single disease, disorder, or condition identified according to the present disclosure. In some cases, the treatment comprises one or more drug treatments or therapies for multiple related diseases, disorders, or conditions.
In some cases, the treatment or treatment recommendation is generated for a subject who has not undergone treatment. In some cases, the subject is undergoing treatment and/or has previously been treated. In some cases, the classification or evaluation of a subject for one or more diseases, disorders, or conditions is used to monitor responsiveness to the treatment. For example, an algorithm or model that generates a result indicating severity or severity category for a disease, disorder, or condition may be used to evaluate a subject over time to determine whether the subject is responding to ongoing treatment as indicated by a decrease in severity over time. Accordingly, the systems and methods disclosed herein can include recommendations or steps to continue current treatment or therapy, cease treatment or therapy, or change/modify the current treatment or therapy (e.g., by changing a dose or adding another treatment).
In some instances, the systems and methods disclosed herein provide treatment(s) or treatment recommendation(s) for one or more diseases, disorders, or conditions selected from gastroesophageal reflux disease (e.g., antacids, H-2 receptor blockers such as cimetidine, famotidine, nizatidine, ranitidine, proton pump inhibitors such as esomeprazole, lansoprazole, omeprazole, pantoprazole, surgery), bipolar disorder (e.g., mood stabilizers such as valproic acid, antipsychotics such as Haldol decanoate, aripiprazole, olanzapine, risperidone, antidepressants such as selective serotonin reuptake inhibitors, citalopram, fluoxetine, paroxetine), amyotrophic lateral sclerosis (e.g., riluzole, edavarone, physical therapy, speech therapy), osteoarthritis (e.g., acetaminophen, NSAIDs, duloxetine, corticosteroids, surgery, physical therapy), multiple sclerosis (e.g., corticosteroids, such as prednisone and methylpredinisone, plasmapheresis, beta interferons, glatiramer acetate, fingolimod, teriflunomide, biologics such as ocrelizumab, natalizumab, alemtuzumab), fibromyalgia (acetaminophen, NSAIDs, antidepressants such as duloxetine, milnacipran, cyclobenzaprine, anti-seizure drugs such as gabapentin, pregabalin, physical therapy), systemic lupus erythematosus (acetaminophen, NSAIDs, steroid creams, corticosteroids, antimalarial drugs, immunotherapies), generalized anxiety disorder (e.g., antidepressants such as escitalopram, duloxetine, venlafaxine, paroxetine), rheumatoid arthritis (e.g., acetaminophen, NSAIDs, corticosteroids such as prednisone, disease modifying antirheumatic drugs (DMARMs) such as methotrexate, biologics such as adalimumab, certolizumab, etanercept, golimumab, occupational therapy, surgery), major depressive disorder (e.g., SSRIs, SNRIs, antidepressants, MAOIs), high blood pressure hypertension (e.g., angiotensin converting enzyme (ACE) inhibitors, beta-blockers, calcium channel blockers, alpha-blockers, alpha-agonists, renin inhibitors diuretics), hypothyroidism (e.g., synthetic levothyroxine), or post-traumatic stress disorder (e.g., antidepressants, anti-anxiety medication, prazosin, cognitive therapy, exposure therapy). Although an exhaustive list of pharmaceutical and non-pharmaceutical treatments is not provided for every disease, disorder, or condition described herein, the present disclosure contemplates any treatment known in the field for the diseases, disorders, or conditions including but not limited to treatments for those diseases, disorders, or conditions recited in Table 1.
The classifications can comprise conditions that are not necessarily associated with disease states. For example, a classification can include obesity, sleep deprivation, lack of exercise, oral health status, sleep apnea, and other health status indicators.
C. diff (Clostridium difficile)
E. coli infection
H. pylori
Staphylococcus aureus
Biological Samples
In some aspects, the algorithms, models, or classifiers described herein utilize data derived from biological samples. Biological samples include any biological material from which biomolecules such as metabolites can be prepared and examined. Non-limiting examples include whole blood, plasma, saliva, cheek swab, fecal material, urine, cell mass, biopsy, or any other bodily fluid or tissue.
Metabolites
In some aspects, the algorithms, models, or classifiers described herein are configured to generate a classification or a spectrum of related classifications based on data such as metabolite data. The metabolite data can be obtained from a biological sample of an individual using various molecular detection techniques described herein. The metabolites can be implicated in one or more metabolic pathways. Metabolites include small molecules present in the cells, tissues, organs, and/or fluids that are involved in metabolism. A metabolite can be an intermediate end product of a metabolic pathway or process. Metabolites can have various functions, including use as a source of energy (e.g., ATP), a metabolic building block (e.g., acetyl coenzyme A), signaling, and other molecular pathways.
Metabolites can include components of biochemical classes of molecules such as amino acids, monosaccharides, nucleotides, and fatty acids/glycerol and other building blocks of proteins, carbohydrates, nucleic acids, and lipids, respectively. Metabolites can include coenzymes such as adenosine triphosphate (ATP) and nicotinamide adenine dinucleotide (NADH or NADPH) which play roles in various biochemical anabolic and catabolic reactions. Table 2 shows a non-limiting list of metabolites that can be evaluated by the algorithms described herein to generate one or more classifications of diseases, disorders, or conditions. In some cases, the panel of biomarkers used to classify or evaluate the status of a disease, disorder, or condition as disclosed herein comprises one or more metabolites selected from Table 2. In some cases, the panel of biomarkers comprises at least 1, 2, 3, 4, 5, 6, 7, 8, 19, 10, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, or 950 or more metabolites from Table 2. In some cases, the panel of biomarkers comprises no more than 1, 2, 3, 4, 5, 6, 7, 8, 19, 10, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, or 950 or more metabolites from Table 2. In some embodiments, the panel of biomarkers comprises a subset of metabolites selected from Table 2 that satisfy a threshold or performance metric as disclosed herein, for example, a correlation or association with one or more diseases, disorders, or conditions of interest having a certain p-value or metric such as PPV or AUC.
Protein or Polypeptide Data
Disclosed herein are algorithms, classifiers, or models that generate classifications of individuals based on input data including protein or polypeptide data. Protein or polypeptide data can include information regarding the identity and/or quantity of one or more proteins or polypeptides obtained from a biological sample. In some embodiments, the data is obtained using proteomics techniques such as ELISA, proximity extension assay (PEA), mass spectrometry. In some embodiments, the data is obtained using antibodies that recognize the one or more proteins or polypeptides. Various techniques allow for multiplex analysis of a plurality of proteins or polypeptides in a single sample such as, for example, multiple reaction monitoring (MRM) mass spectrometry, ELISA, proximity extension assay, Western Blot, and protein detection techniques used in the field. In some embodiments, the protein or polypeptide data comprises information for a protein panel. The protein panel can be configured to address specific inquiries such as, for example, having protein biomarkers linked to cardiovascular health for purposes of assessing a heart condition.
In some embodiments, the protein panel comprises a list of proteins such as the ones provided by Olink Proteomics. In some embodiments, the protein panel comprises a cardiometabolic panel. In some embodiments, the protein panel comprises a cell regulation panel. In some embodiments, the protein panel comprises a cardiovascular panel. In some embodiments, the protein panel comprises a development panel. In some embodiments, the protein panel comprises an immune response panel. In some embodiments, the protein panel comprises an immune-oncology panel. In some embodiments, the protein panel comprises an inflammation panel. In some embodiments, the protein panel comprises a metabolism panel. In some embodiments, the protein panel comprises a neurology panel. In some embodiments, the protein panel comprises an oncology panel. In some embodiments, the protein panel comprises an organ damage panel.
In some embodiments, the protein panel comprises a plurality of proteins or polypeptide biomarkers. In some embodiments, the protein panel comprises at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, or more proteins or polypeptides. In some embodiments, the protein panel comprises a shortened or filtered group of proteins or polypeptides. In some embodiments, the protein panel comprises a reduced group of proteins or polypeptides generated by curating an initial group of proteins or polypeptides for targeted properties or associations. For example, an initial group of proteins linked to ALS may be curated to generate a filtered list of proteins that has more robust experimental support for a causative role in ALS. Accordingly, in some embodiments, the protein panel (e.g., a reduced or filtered panel) has no more than 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, or no more than 200 proteins or polypeptides.
Gene Expression Data
Disclosed herein are algorithms, classifiers, or models that generate classifications of individuals based on input data including RNA sequencing and/or expression data. In some embodiments, input data is generated by performing RNA sequencing on a biological sample obtained from a subject. The input data can be generated using any number of available laboratory techniques including reverse transcriptase quantitative PCR (RT-qPCR) and various RNA sequencing technologies. Examples of RNA sequencing include mRNA sequencing, small RNA sequencing, whole RNA sequencing, targeted RNA sequencing, RNA exome targeted sequencing, and single-cell RNA sequencing. Small RNA sequencing targets small RNA molecules such as microRNA. Whole RNA sequencing targets the RNA transcripts in the transcriptome, and includes both coding and noncoding RNA. Targeted RNA sequencing allows for the selecting and sequencing of specific transcripts of interest using targeted enrichment or targeted amplicon. RNA exome capture sequencing enriches for the coding regions of the transcriptome. In some embodiments, the RNA data comprises information for a genetic panel. The genetic panel can be configured to address specific inquiries such as, for example, having genetic biomarkers linked to cardiovascular health for purposes of assessing a heart condition.
In some embodiments, the genetic panel comprises a list of genes or transcripts having some link or association with one or more health conditions or traits. In some embodiments, the genetic panel comprises RNA sequencing information for a plurality of genes or transcripts. In some embodiments, the genetic panel comprises a cardiometabolic panel. In some embodiments, the genetic panel comprises a cell regulation panel. In some embodiments, the genetic panel comprises a cardiovascular panel. In some embodiments, the genetic panel comprises a development panel. In some embodiments, the genetic panel comprises an immune response panel. In some embodiments, the genetic panel comprises an immune-oncology panel. In some embodiments, the genetic panel comprises an inflammation panel. In some embodiments, the genetic panel comprises a metabolism panel. In some embodiments, the genetic panel comprises a neurology panel. In some embodiments, the genetic panel comprises an oncology panel. In some embodiments, the genetic panel comprises an organ damage panel.
In some embodiments, the genetic panel comprises a plurality of genetic biomarkers. In some embodiments, the genetic panel comprises at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, or more genes or transcripts. In some embodiments, the genetic panel comprises a shortened or filtered group of genes or transcripts. In some embodiments, the genetic panel comprises a reduced group of genes generated by curating an initial group of genes or transcripts for targeted properties or associations. For example, an initial group of genes or transcripts linked to ALS may be curated to generate a filtered list of genes or transcripts that has more robust experimental support for a causative role in ALS. Accordingly, in some embodiments, the genetic panel (e.g., a reduced or filtered panel) has no more than 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, or no more than 200 genes or transcripts.
Feature Selection
Disclosed herein are algorithms, classifiers, or models that generate classifications of individuals based on input data. The algorithms, classifiers, or models incorporate various features such as, for example, one or more biomarkers selected from protein levels, RNA transcript levels, and/or metabolite levels obtained from a biological sample of a subject. The features can be selected through analyzing data obtained for an initial feature set to identify the features that are significantly associated with the biological trait of interest. Using prior knowledge and/or data-driven methods, such features can be selected prior to the application of machine learning algorithms to generate trained models. In some embodiments, features are using certain data transformations such as, for example, principal component analysis.
In some embodiments, an initial feature set is generated by selecting or screening for all biomarkers known to have some association with a particular biological trait or combination of traits. In some embodiments, the initial feature set is generated by mining scientific publications or studies. For example, loose statistical associations from large-scale screening may generate statistical noise that makes it difficult to identify relevant biomarkers as features for the model. Thus, in some cases, prior knowledge from scientific publications is used to screen for relevant features. As an example, diabetes type 2, ALS, and MS models can be constructed based on publications that identified key biomarkers as indicators of these particular diseases or disorders. In some embodiments, feature selection comprises screening for or identifying features from scientific publications. In some embodiments, feature selection comprises screening for or identifying features from one or more databases. In some embodiments, the one or more databases comprises annotation(s) of one or more biomarkers. In some embodiments, feature selection comprises screening for or identifying features based on annotation(s) from one or more databases.
The annotation(s) can be a gene ontology of a particular biomarker such as an RNA transcript or a gene. The annotation or gene ontology for a biomarker can provide information about its function or relationship with other markers. As an example, the Gene Ontology (GO) resource provides a comprehensive knowledge base on genes and their products. In some embodiments, the gene ontology is represented as a keyword(s) or an identifier (a number, letter, or other unique combination of characters). The gene ontology can refer to a cellular component such as the parts of the cell or its extracellular environment associated with the biomarker. Illustrative and non-limiting examples of cellular components include cell junction, chromatin, membrane, organelle, or other component of the cell. In some embodiments, the gene ontology refers to a molecular function, which refers to the activities of the biomarker at the molecular level such as, for example, a chemical reaction catalyzed by the enzyme protein product of a gene or a binding activity of a transcription factor. Illustrative and non-limiting examples of molecular functions include antioxidant activity, protein folding chaperone, transcription regulator activity, and various other functions. In some embodiments, the gene ontology refers to a biological process, which refers to a set or sequence of one or more molecular activities that together play some role in the functioning of a living organism or component thereof. Illustrative and non-limiting examples of biological processes include actin filament polymerization, detoxification, hematopoiesis, phosphorus utilization, signaling, and various other processes. The annotation or gene ontology can be tiered or structured to provide more general information and/or more detailed information. For example, some biomarkers may be annotated with a cellular component gene ontology tag for cell junction, but a subset of these biomarkers may be further tagged with adherens junction while other biomarkers are instead tagged with desmosome depending on the specific cellular structure these individual biomarkers are associated with.
In some embodiments, the annotation for a biomarker is obtained from a database comprising biological pathway information. The database can include pathway maps of the relationships between genes or gene products and other biological molecules such as, for example, metabolites. As an example, the KEGG pathway database provides biological pathways that model molecular interactions between biological components. The pathway maps can include various types of information for biomarkers including genes, proteins, RNAs, chemical compounds, glycans, and chemical reactions. In some embodiments, additional information such as mutations associated with diseases and drug targets are included. In some embodiments, the pathway maps are classified into the sub-sections such as metabolism, genetic information processing (e.g., transcription, translation, replication, etc.), environmental information processing (e.g., signal transduction), cellular processes (e.g., cell proliferation), organismal systems (e.g., immune system), human diseases, and drug development.
In some embodiments, the systems, methods, and software disclosed herein utilize an automated or semi-automated feature selection process by which features are selected based on third party annotations such as gene ontology tags. In some embodiments, the features are at least partially selected or screened based on one or more annotations. In some embodiments, the biomarker(s) are annotated or associated with a gene ontology based on one or more tags or labels. These tags or labels can be standardized and formatted to facilitate automated or semi-automated analysis such as, for example, extraction and/or processing. In some embodiments, one or more biomarkers are extracted from a third party database based on one or more annotations. In some embodiments, the biomarkers are further processed or selected based on scientific publications to arrive at an initial feature set. This feature set can be trained using labeled data by a machine learning algorithm to generate a model and/or select for the most significantly associated features for a disease, disorder, or condition, or other trait. In some embodiments, feature selection comprises screening for biomarkers based on KEGG and/or GO annotations.
In some embodiments, the systems, methods, and software disclosed herein comprise a feature selection or feature transformation process. A goal of feature selection is to reduce the size of the feature set while retaining as much useful information as possible. In some embodiments, feature selection comprises filtering out or removing features based on variance. Such techniques include principal component analysis (PCA), partial least squares (PLS) regression, and independent component analysis (ICA).
Non-Molecular Data
Although various algorithms described herein utilize molecular information such as metabolite data to generate classifications of individuals, non-assayed information can also be used. The combination of molecular data and non-molecular data can be useful in enhancing classifier performance. For example, age and sex can serve as important discriminatory features for accurately classifying an individual. Non-molecular data can include patient information such as demographic information. In some cases, classifiers or machine learning models utilize data comprising non-molecular data such as, for example, age or age range, race, ethnicity, nationality, sex, smoking status, weight, body mass index (BMI), exercise (e.g., frequency, duration, and/or intensity), hobbies, household income, geographic location, disabilities, education, employment status, health status (e.g. a confirmed cancer diagnosis), children, marital status, or any combination thereof.
Non-molecular data can include measurable health parameters. Examples of health parameters include heart rate, blood pressure, body temperature, body fat percentage, height, waistline, VO2 max, and other relevant parameters.
Traits
Disclosed herein are algorithms, classifiers, or models that generate classifications or predictions pertaining to one or more traits. Traits are non-molecular information about a subject that can be related to the subject's general well-being or health status. In some embodiments, traits are not directed to a particular disease or disease spectrum. Examples of traits include non-molecular data such as age, sex, body mass index (BMI), race, ethnicity, personality traits, family history, and other measurable health parameters or demographics. Traits can be selected from or organized into various categories including personal characteristics, general health, mental health, health behaviors, interventions (e.g., treatments and therapies), systems (e.g., organ systems), environmental (e.g., work environment), conditions (e.g., diagnostic history), and other categories related to general health and well-being. In some embodiments, the algorithms, classifiers, or models disclosed herein are trained on data pertaining to one or more traits.
Accordingly, in some embodiments, predictions are generated for individuals that provide an assessment (e.g., a regression score) of one or more traits. In some embodiments, the prediction is an assessment of a composite well-being for an individual that incorporates multiple traits. In some embodiments, the prediction incorporates information about one or more traits to provide an assessment of one or more other traits. As an example, trait information or data for sleep, diet, and sun exposure may be included in a data set along with the “thrive” trait (e.g., a general assessment of health and well-being) that is used to train a model to predict a “thrive” assessment or score based at least in part on the trait information. In some embodiments, the model is trained to assess one or more traits using molecular data and/or trait information. As an example, a model can be trained to incorporate protein levels and RNA sequencing data in providing an assessment of an individual for a particular trait such as anxiety.
Alternatively, in some embodiments, predictions or classifications of a disease, disorder, or condition is generated based on input data incorporating trait information. In some embodiments, the systems, methods, and software disclosed herein identify certain traits that are significantly associated with or predictive of some mental health conditions such as depression. As an illustrative example, trait information for fitness and sleep may be identified as being associated with depression. Trait information and other data types such as molecular data can be combined as features in a single model or multiple models. In some embodiments, the model(s) undergoes machine learning using training data that incorporates trait information and/or molecular data such as RNA sequencing data and/or protein quantification. As a result, predictions can be generated that provide an assessment or evaluation of one or more traits and/or one or more diseases, disorders, or conditions. As an example, certain trait information may be associated with a particular disease or disorder that the subject is unaware of such as ALS.
In some embodiments, the systems, methods, and software disclosed herein incorporate input data including molecular data to generate predictions or evaluations of one or more traits. As an illustrative example, a model or algorithm undergoes machine learning using training data that includes metabolite data for individuals along with trait information relating to smoking, past smoking, alcohol load, amount of sleep, hours awake, or acute infection(s). Accordingly, certain metabolite levels can be identified as relevant to certain traits which can, for example, provide a metabolite signature for smokers. In some embodiments, the trait-related predictions or evaluations provided by the algorithms, models, or classifiers disclosed herein comprise a regression (e.g., a numerical or continuous output) instead of a classification (e.g., a categorical output such as yes/no).
In some embodiments, the systems, methods, and software disclosed herein incorporate input data such as patient-generated health data alone or in combination with other types of data (e.g., molecular data). In some embodiments, training data pertaining to one or more traits include a subject's self-assessment of a trait such as responses to questions. Trait information can include patient-generated health data. In some cases, trait information comprises yes/no responses to questions. In some cases, trait information comprises a response that is a number or score (e.g., an acute pain self-assessment from 1 to 10 with 10 being the highest possible level of pain). Examples of trait categories, traits, questions, and responses are provided in Table 3.
Metabolite Detection Techniques
Metabolites in a specimen can be determined using various molecular detection techniques such as mass spectrometry, nuclear magnetic resonance, chromatography, or other methods. Oftentimes, mass spectrometry is used in combination with a chromatography technique in order to separate metabolites of interest prior to mass spectrometry analysis in order to provide enhanced sensitivity of detection and/or quantitation of metabolites in complex samples. For example, high performance liquid chromatography (HPLC), gas chromatography (GC), and capillary electrophoresis (CE) may be coupled to mass spectrometric analysis to evaluate metabolites in a biological sample.
A cohort sample set can be processed in sample groups with subject samples and pooled plasma samples for QC/normalization purposes. Each sample group is then analyzed on the LC-MS platform shortly after processing, for example the day following the completion of sample processing. Consistent with the specification, alternative numbers of subject and normalization samples are employed in certain examples.
In some cases, LC-MS data from each sample is collected on an appropriate instrument with an appropriate ionization source, for example a quadrupole time-of-flight (Q-TOF) mass spectrometer coupled to ultra-high performance liquid chromatography (UHPLC) instrument, with an electrospray ionization (ESI) source. LC flow rates can be optimized based on sample conditions and pressures.
The biological sample can be assessed by analysis of a number of injections from a single pooled source. For example, a collection of blood samples is assessed by LC-MS using multiple injections from a single pooled source. Data is collected in MS1/MS2 mode so that feature identifications can be made concurrently with the quantitative MSI data. Tandem mass spectrometry data is collected via a second fragmentation method, such as collision induced dissociation (CID), in which an MSI survey scan is followed by fragmentation of other precursor ions, such as the three most abundant precursor ions.
Algorithms
Disclosed herein are algorithms for analyzing input data for one or more biomarkers to generate output relating to differential classifications or associations such as the presence or likelihood of a disease, disorder, or condition or trait. In some embodiments, the input data comprises one or more data types such as metabolite data, genetic data, protein data, or any combination thereof. Analyses of input data such as metabolite data, and the differential classifications derived therefrom are typically performed using various algorithms and programs. The levels of individual metabolites can make up a metabolite pattern, signature, or profile that corresponds to a particular individual. The machine learning algorithms described herein can generate classifications that account for the complex interrelationships between different metabolites and the pathways that impact those metabolites. Metabolite signatures can provide insight into the health status and/or therapeutic options for the individual. In some embodiments, non-metabolite data such as gene expression data and/or protein quantification data is analyzed alone or in combination with each or with metabolite data using any of the algorithms or methods described herein. Accordingly, genetic signatures and/or protein signatures can also provide insight into the health status or other traits for the individual. In some cases, the algorithms disclosed herein allow for detection, evaluation, assessment, and/or diagnosis of two or more diseases, disorders, or conditions or traits. The two or more diseases, disorders, or conditions or traits may be related, for example, falling within a common category such as autoimmune disorder or immune-related disorder. In some cases, diseases, disorders, or conditions or traits are related if they share one or more common features that are predictive of their status such as in the case of overlapping feature sets of biomarker panels.
Metabolites displaying differential signaling patterns, i.e., discriminating metabolites, between samples obtained from reference subjects (e.g., healthy subjects or subjects with a different disease) can be identified using known statistical tests such as a Student's T-test or ANOVA. The statistical analyses can be applied to select the discriminating metabolites that distinguish the different conditions at predetermined stringency levels. In some cases, metabolites are evaluated for feature importance within one or more models such as shown in
In some cases, a metabolite biomarker panel as used herein comprises at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, or 24 of the metabolites listed in
In some cases, a metabolite biomarker panel as used herein comprises at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, or 25 of the metabolites listed in
In some cases, a metabolite biomarker panel as used herein comprises at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, or 20 of the metabolites listed in
In some instances, the systems, media, and methods disclosed herein apply machine learning models or algorithms that use one or more biomarker panels to identify, classify, evaluate, or differentiate between related diseases, disorders, or conditions. Related diseases, disorders, or conditions can include autoimmune or immune-related diseases, disorders, or conditions. In some cases, the systems, media, and methods disclosed herein comprise monitoring or evaluating biomarkers such as metabolites for an individual over time (optionally with or without treatment) and generating a recommendation for a treatment.
In some cases, information of the discriminating metabolites selected can be subsequently imported into a machine learning algorithm to obtain a statistical or mathematical model (e.g., a classifier) that classifies the metabolic data with accuracy, sensitivity, and/or specificity. Any one of the many computational algorithms can be utilized for the classification purposes. Likewise, information for discriminating genes or proteins can also be imported into a machine learning algorithm to generate a model that classifies data or generates a risk prediction based on the data including metabolic data, gene expression data, protein quantification data, or any combination thereof.
The classifiers can be rule-based or machine learning algorithms. The machine learning classification algorithms can be supervised or unsupervised. A basic classification algorithm, Linear Discriminant Analysis (LDA) may be used in analyzing biomedical data in order to classify two or more disease classes. LDA can be, for example, a classification algorithm. A more complex classification method, Support Vector Machines (SVM), uses mathematical kernels to project the original predictors to higher-dimensional spaces, then identifies the hyperplane that optimally separates the samples according to their class. Some common kernels include linear, polynomial, sigmoid or radial basis functions. Other algorithms for data analysis and predictive modeling based on metabolite data can include but are not limited to Naive Bayes Classifiers, Logistic Regression, Quadratic Discriminant Analysis, K-Nearest Neighbors (KNN), K Star, Attribute Selected Classifier (ACS), Classification via clustering, Classification via Regression, Hyper Pipes, Voting Feature Interval Classifier, Decision Trees, Random Forest, and Neural Networks, including Deep Learning approaches.
In some embodiments, a machine learning algorithm (or software module) of a platform or system as described herein utilizes one or more neural networks. A neural network is a type of computational system that can learn the relationships between an input data set and a target data set. A neural network is a software representation of a human neural system (e.g., cognitive system), intended to capture “learning” and “generalization” abilities as used by a human. In some embodiments machine learning algorithm (or software module), the machine learning algorithm (or software module) comprises a neural network comprising a convolutional neural network. Non-limiting examples of structural components of embodiments of the machine learning software described herein include: convolutional neural networks, recurrent neural networks, dilated convolutional neural networks, fully connected neural networks, deep generative models, and Boltzmann machines.
In some embodiments, a neural network is comprised of a series of layers termed “neurons.” In some embodiments, a neural networks comprises an input layer, to which data is presented; one or more internal, and/or “hidden,” layers; and an output layer. A neuron may be connected to neurons in other layers via connections that have weights, which are parameters that control the strength of the connection. The number of neurons in each layer may be related to the complexity of the problem to be solved. The minimum number of neurons required in a layer may be determined by the problem complexity, and the maximum number may be limited by the ability of the neural network to generalize. The input neurons may receive data from data being presented and then transmit that data to the first hidden layer through connections' weights, which are modified during training. The first hidden layer may process the data and transmit its result to the next layer through a second set of weighted connections. Each subsequent layer may “pool” the results from the previous layers into more complex relationships. In addition, whereas conventional software programs require writing specific instructions to perform a function, neural networks are programmed by training them with a known sample set and allowing them to modify themselves during (and after) training so as to provide a desired output such as an output value. After training, when a neural network is presented with new input data, it is configured to generalize what was “learned” during training and apply what was learned from training to the new previously unseen input data in order to generate an output associated with that input.
In some embodiments, metabolite profiles are obtained from a training set of samples, which are used to identify the most discriminative combination of metabolites. In some cases, the most discriminative combination of metabolites is identified by applying an elimination algorithm based on SVM analysis. The accuracy of the algorithm using various numbers of input metabolites ranked by level of statistical significance can be determined by cross-validation. To generate and evaluate metabolite profiles of a feasible number of discriminating metabolites, multiple models can be built using a plurality of discriminating metabolites to identify the best performing model(s). In some cases, an Ensemble model is generated that incorporates a plurality of models. The Ensemble model can provide classification of samples that is subject to less variation than individual models or classifiers that are incorporated into the Ensemble model.
In some instances, specific metabolite(s) are excluded from inclusion in the training and/or testing of machine learning algorithms. Metabolites can be excluded based on certain rules designed to reduce sample-to-sample variation. For example, certain metabolites undergo significant variation over time and may correspond to certain activities such as, for example, consumption of food or liquids, physical activity, sleep, or other factors. Accordingly, failure to account for these factors can result in considerable variation of corresponding metabolites that consequently reduce the predictive performance of classifiers trained using data for these metabolites. Thus, in some cases, the methods described herein comprise removing or excluding one or more metabolites from inclusion in the classifier(s) in order to enhance predictive performance. In some embodiments, a feature list or panel of features (e.g., biomarkers) comprises at least 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 350, 400, 450, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 10000, 20000, 30000, 40000, or 50000 metabolites and/or no more than 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 350, 400, 450, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 10000, 20000, 30000, 40000, or 50000 metabolites that are used in a specific machine learning algorithm or classifier.
Multiple models comprising different numbers of discriminating metabolites can be generated, and the performance of each model can be evaluated by a cross-validation process. An SVM classifier can be trained and cross-validated by assigning each sample of a training set of samples to one of a plurality of cross-validation groups. For example, for a five-fold cross-validation, each sample is assigned to one of four cross-validation groups such that each group comprises test and control or reference samples. One of the cross-validation groups is held-out, and a classifier model is trained using the samples in the remaining groups 2-4. Metabolites that discriminate test cases and reference samples in the training group can be analyzed and ranked, for example by statistical p-value. The top k metabolites can be used as predictors for the model. To evaluate the relationship between the number of input predictors and model performance, and to guard against overfitting, the sub-loop is repeated for a range of k such as 10, 25, 50 top metabolites or more. Predictions or classification of samples in group 1 are made using the model generated using groups 2-4. Models for each of the four groups are generated, and the performance (AUC, sensitivity and/or specificity) can be calculated using all the predictions from the 4 models using data from true disease samples. The cross-validation steps can be repeated at least 100 times, and the average performance is calculated relative to a confidence interval such as, for example, 95%.
Alternatively, unsupervised learning can be used to train a classifier or model without using labeled cases or samples. A common example of unsupervised training entails cluster analysis. Non-limiting examples of clustering algorithms include hierarchical clustering, k-means clustering, Gaussian mixture models, and Hidden Markov models.
In some cases, a plurality of machine learning algorithms are utilized to generate a final Ensemble model. The plurality of machine learning algorithms can comprise two or more of: Generalized Linear Model (glmnet), Random Forests (if), Partial Least Squares (pls), Extreme Gradient Boosting (xgbDART), Support Vector Machines with Linear Basis Function Kernel (svmLinear), Support Vector Machines with Radial Basis Function Kernel (svmRadial), or Neural Networks (nnet). Two or more of these 7 algorithms can be run with various different random seed train/test splits.
The classifier used to generate predictions includes one or more selected feature spaces such as metabolite, gene expression, protein quantity, or any combination thereof. The values for these features obtained from a sample can be fed into the classifier or trained algorithm to generate one or more predictions. In some cases, the methods disclosed herein select for the variables that are of predictive value, for example, by culling the features to generate a feature subset used for generating predictions in the final classifier or model. Methods that reduce the number of variables or features can be selected from a non-limiting group of algorithms including principal component analysis (PCA), partial least squares (PLS) regression, and independent component analysis (ICA). In some cases, the methods disclosed herein analyze numerous variables directly and are selected from a non-limiting group of algorithms including methods based on machine learning processes. Machine learning processes can include random forest algorithms, bagging techniques, boosting methods, or any combination thereof. Methods may be statistical methods. Statistical methods can include penalized logistic regression, prediction analysis of microarrays, methods based on shrunken centroids, support vector machine analysis, or regularized linear discriminant analysis.
A feature space can comprise a panel of metabolites, genes, proteins, or any combination thereof within a sample. In some cases, the classifier or trained algorithm comprises a metabolite panel comprising at least 1, at least 2, at least 3, at least 4, at least 5, at least 6, at least 7, at least 8, at least 9, at least 10, at least 11, at least 12, at least 13, at least 14, at least 15, at least 16, at least 17, at least 18, at least 19, at least 20, at least 20, at least 25, at least 30, at least 35, at least 40, at least 45, or at least 50 or more metabolites (e.g., metabolite levels). In some cases, the classifier or trained algorithm comprises a genetic panel comprising at least 1, at least 2, at least 3, at least 4, at least 5, at least 6, at least 7, at least 8, at least 9, at least 10, at least 11, at least 12, at least 13, at least 14, at least 15, at least 16, at least 17, at least 18, at least 19, at least 20, at least 20, at least 25, at least 30, at least 35, at least 40, at least 45, or at least 50 or more genes (e.g., gene expression levels). In some cases, the classifier or trained algorithm comprises a protein panel comprising at least 1, at least 2, at least 3, at least 4, at least 5, at least 6, at least 7, at least 8, at least 9, at least 10, at least 11, at least 12, at least 13, at least 14, at least 15, at least 16, at least 17, at least 18, at least 19, at least 20, at least 20, at least 25, at least 30, at least 35, at least 40, at least 45, or at least 50 or more proteins (e.g., protein levels).
An optimal model/classifier based on metabolite data can be selected and used to classify a test set. The performance of different classifiers is determined using a validation set and/or using a test set of samples. Accordingly, performance characteristics such as accuracy, sensitivity, specificity, and Area Under the Curve (AUC) of the Receiver Operating Characteristic (AUC) curve can be obtained from a given model. In some embodiments, different sets of discriminating metabolites are identified to distinguish different diseases, disorders, or conditions. Accordingly, an optimal model/classifier based on a set of the most discriminating input metabolites is established for each of the diseases, disorders, or conditions to provide a differential diagnosis.
In some instances, a plurality of models is combined or consolidated into an Ensemble classifier or model. The plurality of models can include two, three, four, five, six, seven, or more models. In some cases, the Ensemble model is an average of the plurality of models. One challenge that can arise in the classification of a particular disease, disorder, or condition is that some such diseases, disorders, or conditions are closely related and may share one or more common features used to train the classifier or model. For example,
Systems for Classifying an Individual
In some aspects, a system as described herein is configured to generate a classification of an individual relative to one or more related classifications. The system as described herein can comprise a network element for communicating with a server. Sometimes, the system comprises a server. The system can be configured to upload to and/or download data from the server. In some cases, the server is configured to store metabolite data and/or other information for the subject. The server can be configured to store historical data (e.g., past metabolite data) for the subject. In some instances, the server is configured to backup data from the system. In certain cases, the system is configured to perform any of the methods described herein.
In some aspects, a system as described herein is configured to generate a classification of an individual for one or more diseases, disorders, or conditions. The system can comprise a network element communicating with a server on a network and a device, the device comprising: a processor; and a non-transitory computer-readable medium including instructions executable by the processor and configured to cause the processor to: (a) receiving data relating to a specimen taken from the individual; (b) providing the data as input to one or more machine learning algorithms; and (c) generating, using the one or more machine learning algorithms, a classification of the individual relative to a plurality of related classifications based on the data.
In some cases, the system is configured to encrypt data. In some embodiments, data on the server is encrypted. The system or apparatus can comprise a data storage unit or memory for storing data. In certain instances, data encryption is carried out using Advanced Encryption Standard (AES). Data encryption is often carried out using 128-bit or 256-bit AES encryption. Data encryption can include full-disk encryption of the data storage unit. In some instances, data encryption comprises virtual disk encryption (e.g., encrypting a folder containing sensor data files for a subject). In various aspects, data encryption comprises file encryption (e.g., encrypting sensor data files for an individual). Sometimes, data that is transmitted or otherwise communicated between the system or apparatus and other devices or servers is encrypted during transit. Wireless communications between the system and other devices or servers can be encrypted. Data in transit can be encrypted using a Secure Sockets Layer (SSL).
A system as described herein can comprise a digital processing device that includes one or more hardware central processing units (CPUs) or general purpose graphics processing units (GPGPUs) that carry out the device's functions. The digital processing device further comprises an operating system configured to perform executable instructions. The digital processing device is optionally connected to a computer network. The digital processing device is optionally connected to the Internet such that it accesses the World Wide Web. The digital processing device is optionally connected to a cloud computing infrastructure. Suitable digital processing devices include, by way of non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, sub-notebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Those of skill in the art will recognize that many smartphones are suitable for use in the system described herein.
Typically, a digital processing device includes an operating system configured to perform executable instructions. The operating system is, for example, software, including programs and data, which manages the device's hardware and provides services for execution of applications. Those of skill in the art will recognize that suitable server operating systems include, by way of non-limiting examples, FreeBSD, OpenBSD, NetBSD®, Linux, Apple® Mac OS X Server®, Oracle® Solaris®, Windows Server®, and Novell® NetWare®. Those of skill in the art will recognize that suitable personal computer operating systems include, by way of non-limiting examples, Microsoft® Windows®, Apple® Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU/Linux®. In some embodiments, the operating system is provided by cloud computing.
A digital processing device as described herein either includes or is operatively coupled to a storage and/or memory device. The storage and/or memory device is one or more physical apparatuses used to store data or programs on a temporary or permanent basis. In some embodiments, the device is volatile memory and requires power to maintain stored information. In some embodiments, the device is non-volatile memory and retains stored information when the digital processing device is not powered. In further embodiments, the non-volatile memory comprises flash memory. In some embodiments, the non-volatile memory comprises dynamic random-access memory (DRAM). In some embodiments, the non-volatile memory comprises ferroelectric random access memory (FRAM). In some embodiments, the non-volatile memory comprises phase-change random access memory (PRAM). In other embodiments, the device is a storage device including, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, magnetic disk drives, magnetic tapes drives, optical disk drives, and cloud computing based storage. In further embodiments, the storage and/or memory device is a combination of devices such as those disclosed herein.
A system or method as described herein can be used to generate a classification based on data such as metabolite data which may then be used to determine whether an individual has or is at risk of having a disease, disorder, or condition. In addition, in some embodiments, a system or method as described herein generates a database as containing or comprising past and/or present metabolite data and/or classifications.
Some embodiments of the systems described herein are computer based systems. These embodiments include a CPU including a processor and memory which may be in the form of a non-transitory computer-readable storage medium. These system embodiments further include software that is typically stored in memory (such as in the form of a non-transitory computer-readable storage medium) where the software is configured to cause the processor to carry out a function. Software embodiments incorporated into the systems described herein contain one or more modules.
In various embodiments, an apparatus comprises a computing device or component such as a digital processing device. In some of the embodiments described herein, a digital processing device includes a display to send visual information to a user. Non-limiting examples of displays suitable for use with the systems and methods described herein include a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic light emitting diode (OLED) display, an OLED display, an active-matrix OLED (AMOLED) display, or a plasma display.
A digital processing device, in some of the embodiments described herein includes an input device to receive information from a user. Non-limiting examples of input devices suitable for use with the systems and methods described herein include a keyboard, a mouse, trackball, track pad, or stylus. In some embodiments, the input device is a touch screen or a multi-touch screen.
The systems and methods described herein typically include one or more non-transitory computer-readable storage media encoded with a program including instructions executable by the operating system of an optionally networked digital processing device. In some embodiments of the systems and methods described herein, the non-transitory storage medium is a component of a digital processing device that is a component of a system or is utilized in a method. In still further embodiments, a computer-readable storage medium is optionally removable from a digital processing device. In some embodiments, a computer-readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives, cloud computing systems and services, and the like. In some cases, the program and instructions are permanently, substantially permanently, semi-permanently, or non-transitorily encoded on the media.
Typically the systems and methods described herein include at least one computer program, or use of the same. A computer program includes a sequence of instructions, executable in the digital processing device's CPU, written to perform a specified task. Computer-readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, those of skill in the art will recognize that a computer program may be written in various versions of various languages. The functionality of the computer-readable instructions may be combined or distributed as desired in various environments. In some embodiments, a computer program comprises one sequence of instructions. In some embodiments, a computer program comprises a plurality of sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from a plurality of locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, or combinations thereof. In further various embodiments, a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, and a standalone application. In some embodiments, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on cloud computing platforms. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.
Typically, the systems and methods described herein include and/or utilize one or more databases. In view of the disclosure provided herein, those of skill in the art will recognize that many databases are suitable for storage and retrieval of baseline datasets, files, file systems, objects, systems of objects, as well as data structures and other types of information described herein. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity-relationship model databases, associative databases, and XML databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, and Sybase. In some embodiments, a database is internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloud computing-based. In other embodiments, a database is based on one or more local computer storage devices.
The digital processing device 3201 includes input device(s) 3245 to receive information from a user, the input device(s) in communication with other elements of the device via an input interface 3250. The digital processing device 3201 can include output device(s) 3255 that communicates to other elements of the device via an output interface 3260.
The CPU 3205 is configured to execute machine-readable instructions embodied in a software application or module. The instructions may be stored in a memory location, such as the memory 3210. The memory 3210 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., RAM) (e.g., a static RAM “SRAM”, a dynamic RAM “DRAM, etc.), or a read-only component (e.g., ROM). The memory 3210 can also include a basic input/output system (BIOS), including basic routines that help to transfer information between elements within the digital processing device, such as during device start-up, may be stored in the memory 3210.
The storage unit 3215 can be configured to store files, such as patient information, e.g., metabolite data and non-molecular data. The storage unit 3215 can also be used to store operating system, application programs, and the like. Optionally, storage unit 3215 may be removably interfaced with the digital processing device (e.g., via an external port connector (not shown)) and/or via a storage unit interface. Software may reside, completely or partially, within a computer-readable storage medium within or outside of the storage unit 3215. In another example, software may reside, completely or partially, within processor(s) 3205.
Information and data can be displayed to a user through a display 3235. The display is connected to the bus 3225 via an interface 3240, and transport of data between the display other elements of the device 3201 can be controlled via the interface 3240.
Methods as described herein can be implemented by way of machine (e.g., computer processor) executable code stored on an electronic storage location of the digital processing device 3201, such as, for example, on the memory 3210 or electronic storage unit 3215. The machine executable or machine readable code can be provided in the form of a software application or software module. During use, the code can be executed by the processor 3205. In some cases, the code can be retrieved from the storage unit 3215 and stored on the memory 3210 for ready access by the processor 3205. In some situations, the electronic storage unit 3215 can be precluded, and machine-executable instructions are stored on memory 3210.
In some embodiments, a remote device 3202 is configured to communicate with the digital processing device 3201, and may comprise any mobile computing device, non-limiting examples of which include a tablet computer, laptop computer, smartphone, or smartwatch. For example, in some embodiments, the remote device 3202 is a smartphone of the user that is configured to receive information from the digital processing device 3201 of the apparatus or system described herein in which the information can include a summary, classifications or predictions, or other data. In some embodiments, the remote device 3202 is a server on the network configured to send and/or receive data from the system described herein.
Some embodiments of the systems and methods described herein are configured to generate a database containing or comprising patient information such as metabolite data. A database, as described herein, is configured to function as, for example, a lookup table for healthcare providers, other medical industry professionals and/or other end users. In these embodiments of the systems and methods described herein, metabolite data and/or classifications or diagnoses are presented in a database so that a user is able to, for example, identify whether a specific individual is at risk of certain diseases, disorders, or conditions. In some embodiments, the database is stored on a server on the network. In some embodiments the database is stored locally on the apparatus (e.g., the monitor component of the apparatus). In some embodiments, the database is stored locally with data backup provided by a server.
Certain Terminology
As used herein, the terms “patient,” “individual,” and “subject” encompasses mammals. Examples of mammals include, but are not limited to, any member of the mammalian class: humans, non-human primates such as chimpanzees, and other apes and monkey species; farm animals such as cattle, horses, sheep, goats, swine; domestic animals such as rabbits, dogs, and cats; laboratory animals including rodents, such as rats, mice and guinea pigs, and the like. In one aspect, the mammal is a human. The term “animal” as used herein comprises human beings and non-human animals. In one embodiment, a “non-human animal” is a mammal, for example a rodent such as rat or a mouse.
As used herein, the term “classify” or “classification” refers to the output of the model or algorithm being a categorical output, for example, positive identification of a disease, disorder, or condition. As used herein, the term “regression” refers to the output of the model or algorithm being a non-categorical output, for example, a number or continuous variable. As classification and regression can both fall under supervised machine learning, a regression output is also contemplated wherever classification is described within the present disclosure. Therefore, disclosure of “a classifier” configured to evaluate the status of a disease, disorder, or condition is to be interpreted as also disclosing a regression model or algorithm.
In some cases, a plurality of machine learning algorithms are utilized to generate a final Ensemble model. The plurality of machine learning algorithms can comprise two or more of: Generalized Linear Model (glmnet), Random Forests (rf), Partial Least Squares (pls), Extreme Gradient Boosting (xgbDART), Support Vector Machines with Linear Basis Function Kernel (svmLinear), Support Vector Machines with Radial Basis Function Kernel (svmRadial), or Neural Networks (nnet). Two or more of these 7 algorithms can be run with various different random seed train/test splits.
For example, 3 different random seed train/test splits were run for all 7 algorithms for 21 models total with respect to MS, ALS, and SLE. Models were run for all conditions with metabolite data for those classifications having more than 35 participants (see
While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.
This application claims the benefit of U.S. Provisional Application No. 62/774,788, filed Dec. 3, 2018, and U.S. Provisional Application No. 62/818,310, filed Mar. 14, 2019, the contents of each of which is hereby incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
3934226 | Stone et al. | Jan 1976 | A |
4712562 | Ohayon et al. | Dec 1987 | A |
4899839 | Dessertine et al. | Feb 1990 | A |
5014798 | Glynn | May 1991 | A |
5025374 | Roizen et al. | Jun 1991 | A |
5435324 | Brill | Jul 1995 | A |
5549117 | Tacklind et al. | Aug 1996 | A |
5594637 | Eisenberg et al. | Jan 1997 | A |
5594638 | Iliff | Jan 1997 | A |
5626144 | Tacklind et al. | May 1997 | A |
5653739 | Maurer et al. | Aug 1997 | A |
5692215 | Kutzik et al. | Nov 1997 | A |
5692500 | Gaston-Johansson | Dec 1997 | A |
5704366 | Tacklind et al. | Jan 1998 | A |
5713350 | Yokota et al. | Feb 1998 | A |
5720502 | Cain | Feb 1998 | A |
5732709 | Tacklind et al. | Mar 1998 | A |
5778882 | Raymond et al. | Jul 1998 | A |
5812983 | Kumagai | Sep 1998 | A |
5830149 | Oka | Nov 1998 | A |
5838313 | Hou et al. | Nov 1998 | A |
5937387 | Summerell et al. | Aug 1999 | A |
5950168 | Simborg et al. | Sep 1999 | A |
5984368 | Cain | Nov 1999 | A |
5991729 | Barry et al. | Nov 1999 | A |
6024699 | Surwit et al. | Feb 2000 | A |
6032119 | Brown et al. | Feb 2000 | A |
6067523 | Bair et al. | May 2000 | A |
6081786 | Barry et al. | Jun 2000 | A |
6102874 | Stone et al. | Aug 2000 | A |
6108635 | Herren et al. | Aug 2000 | A |
6108685 | Kutzik et al. | Aug 2000 | A |
6113552 | Shimazu et al. | Sep 2000 | A |
6168569 | McEwen et al. | Jan 2001 | B1 |
6186145 | Brown | Feb 2001 | B1 |
6231560 | Bui et al. | May 2001 | B1 |
6234964 | Iliff | May 2001 | B1 |
6235964 | Kadash et al. | May 2001 | B1 |
6236983 | Hofmann et al. | May 2001 | B1 |
6246992 | Brown | Jun 2001 | B1 |
6277072 | Bardy | Aug 2001 | B1 |
6282441 | Raymond et al. | Aug 2001 | B1 |
6283923 | Finkelstein et al. | Sep 2001 | B1 |
6294999 | Yarin et al. | Sep 2001 | B1 |
6314405 | Richardson | Nov 2001 | B1 |
6322504 | Kirshner | Nov 2001 | B1 |
6334192 | Karpf | Dec 2001 | B1 |
6335907 | Momich et al. | Jan 2002 | B1 |
6339410 | Milner et al. | Jan 2002 | B1 |
6380858 | Yarin et al. | Apr 2002 | B1 |
6400996 | Hoffberg et al. | Jun 2002 | B1 |
6405034 | Tijerino | Jun 2002 | B1 |
6484144 | Martin et al. | Nov 2002 | B2 |
6529195 | Eberlein | Mar 2003 | B1 |
6560541 | Singh | May 2003 | B1 |
6587829 | Camarda et al. | Jul 2003 | B1 |
6589169 | Surwit et al. | Jul 2003 | B1 |
6611846 | Stoodley | Aug 2003 | B1 |
6612985 | Eiffert et al. | Sep 2003 | B2 |
6690397 | Daignault, Jr. | Feb 2004 | B1 |
6723045 | Cosentino et al. | Apr 2004 | B2 |
6770029 | Iliff | Aug 2004 | B2 |
6789091 | Gogolak | Sep 2004 | B2 |
6802810 | Ciarniello et al. | Oct 2004 | B2 |
6856315 | Eberlein | Feb 2005 | B2 |
6915170 | Engleson et al. | Jul 2005 | B2 |
6929607 | Lipman | Aug 2005 | B2 |
6956572 | Zaleski | Oct 2005 | B2 |
6970742 | Mann et al. | Nov 2005 | B2 |
6980958 | Surwit et al. | Dec 2005 | B1 |
6988075 | Hacker | Jan 2006 | B1 |
6999890 | Kai | Feb 2006 | B2 |
7029441 | Dodds | Apr 2006 | B2 |
7039878 | Auer et al. | May 2006 | B2 |
7054758 | Gill-Garrison et al. | May 2006 | B2 |
7066883 | Schmidt et al. | Jun 2006 | B2 |
7107547 | Cule et al. | Sep 2006 | B2 |
7137951 | Pilarski | Nov 2006 | B2 |
7165221 | Monteleone et al. | Jan 2007 | B2 |
7169085 | Killin et al. | Jan 2007 | B1 |
7251609 | McAlindon et al. | Jul 2007 | B1 |
7286997 | Spector et al. | Oct 2007 | B2 |
7287031 | Karpf et al. | Oct 2007 | B1 |
7302398 | Ban et al. | Nov 2007 | B2 |
7330818 | Ladocsi et al. | Feb 2008 | B1 |
7337121 | Beinat et al. | Feb 2008 | B1 |
7428494 | Hasan et al. | Sep 2008 | B2 |
7725328 | Sumner, II et al. | May 2010 | B1 |
7761311 | Clements et al. | Jul 2010 | B2 |
8160901 | Heywood et al. | Apr 2012 | B2 |
8190451 | Lloyd et al. | May 2012 | B2 |
8214224 | Rao et al. | Jul 2012 | B2 |
8313433 | Cohen et al. | Nov 2012 | B2 |
8643648 | Heywood et al. | Feb 2014 | B2 |
8652039 | Rosales et al. | Feb 2014 | B2 |
8930224 | Heywood et al. | Jan 2015 | B2 |
9270632 | Heywood et al. | Feb 2016 | B2 |
9589104 | Heywood et al. | Mar 2017 | B2 |
9589251 | Heywood et al. | Mar 2017 | B2 |
9638723 | Takagi et al. | May 2017 | B2 |
10402916 | Heywood et al. | Sep 2019 | B2 |
10664572 | Bitran et al. | May 2020 | B2 |
10665344 | Heywood et al. | May 2020 | B2 |
10832816 | Heywood et al. | Nov 2020 | B2 |
11010843 | Heywood et al. | May 2021 | B2 |
20010034639 | Jacoby et al. | Oct 2001 | A1 |
20020010595 | Kapp | Jan 2002 | A1 |
20020026103 | Norris et al. | Feb 2002 | A1 |
20020072933 | Vonk et al. | Jun 2002 | A1 |
20020150872 | Glenn et al. | Oct 2002 | A1 |
20020177757 | Britton | Nov 2002 | A1 |
20020184094 | Calloway | Dec 2002 | A1 |
20030014006 | Alexandre et al. | Jan 2003 | A1 |
20030023461 | Quintanilla et al. | Jan 2003 | A1 |
20030065535 | Karlov et al. | Apr 2003 | A1 |
20030101076 | Zaleski | May 2003 | A1 |
20030125609 | Becker | Jul 2003 | A1 |
20030135128 | Suffin et al. | Jul 2003 | A1 |
20030140063 | Pizzorno et al. | Jul 2003 | A1 |
20030163353 | Luce et al. | Aug 2003 | A1 |
20030187683 | Kirchhoff et al. | Oct 2003 | A1 |
20030233197 | Padilla et al. | Dec 2003 | A1 |
20040006444 | Kang et al. | Jan 2004 | A1 |
20040015337 | Thomas et al. | Jan 2004 | A1 |
20040030741 | Wolton et al. | Feb 2004 | A1 |
20040064447 | Simske et al. | Apr 2004 | A1 |
20040078237 | Kaafarani et al. | Apr 2004 | A1 |
20040103001 | Mazar et al. | May 2004 | A1 |
20040122707 | Sabol et al. | Jun 2004 | A1 |
20040132633 | Carter et al. | Jul 2004 | A1 |
20040161143 | Dietz et al. | Aug 2004 | A1 |
20040193448 | Woodbridge et al. | Sep 2004 | A1 |
20040210458 | Evans et al. | Oct 2004 | A1 |
20040225529 | Snyder et al. | Nov 2004 | A1 |
20040267570 | Becker | Dec 2004 | A1 |
20050080462 | Jenkins et al. | Apr 2005 | A1 |
20050085866 | Tehrani | Apr 2005 | A1 |
20050102160 | Brown | May 2005 | A1 |
20050102162 | Blumenfeld | May 2005 | A1 |
20050108051 | Weinstein | May 2005 | A1 |
20050119534 | Trost et al. | Jun 2005 | A1 |
20050144042 | Joffe et al. | Jun 2005 | A1 |
20050187866 | Lee | Aug 2005 | A1 |
20050191716 | Surwit et al. | Sep 2005 | A1 |
20050197545 | Hoggle | Sep 2005 | A1 |
20050197553 | Cooper | Sep 2005 | A1 |
20050216307 | Clements et al. | Sep 2005 | A1 |
20050251025 | Hancu | Nov 2005 | A1 |
20050272640 | Doyle et al. | Dec 2005 | A1 |
20050283384 | Hunkeler et al. | Dec 2005 | A1 |
20050283385 | Hunkeler et al. | Dec 2005 | A1 |
20050285746 | Sengupta et al. | Dec 2005 | A1 |
20060009810 | Mann et al. | Jan 2006 | A1 |
20060010098 | Goodnow et al. | Jan 2006 | A1 |
20060015369 | Bachus et al. | Jan 2006 | A1 |
20060020175 | Berry et al. | Jan 2006 | A1 |
20060030890 | Cosentino et al. | Feb 2006 | A1 |
20060031101 | Ross | Feb 2006 | A1 |
20060036134 | Tarassenko et al. | Feb 2006 | A1 |
20060036294 | Tehrani | Feb 2006 | A1 |
20060036619 | Fuerst et al. | Feb 2006 | A1 |
20060052945 | Rabinowitz et al. | Mar 2006 | A1 |
20060059160 | Smola et al. | Mar 2006 | A1 |
20060064030 | Cosentino et al. | Mar 2006 | A1 |
20060085217 | Grace | Apr 2006 | A1 |
20060089540 | Meissner | Apr 2006 | A1 |
20060122468 | Tavor | Jun 2006 | A1 |
20060249423 | Reijonen | Nov 2006 | A1 |
20060287885 | Frick | Dec 2006 | A1 |
20070005393 | Cole et al. | Jan 2007 | A1 |
20070015974 | Higgins et al. | Jan 2007 | A1 |
20070021979 | Cosentino et al. | Jan 2007 | A1 |
20070021984 | Brown | Jan 2007 | A1 |
20070048691 | Brown | Mar 2007 | A1 |
20070060803 | Liljeryd et al. | Mar 2007 | A1 |
20070061166 | Ramasubramanian et al. | Mar 2007 | A1 |
20070061487 | Moore et al. | Mar 2007 | A1 |
20070115282 | Turner et al. | May 2007 | A1 |
20070118348 | Brown | May 2007 | A1 |
20070168461 | Moore | Jul 2007 | A1 |
20070168501 | Cobb et al. | Jul 2007 | A1 |
20070203423 | Bardy | Aug 2007 | A1 |
20070214015 | Christian | Sep 2007 | A1 |
20070239416 | Saito et al. | Oct 2007 | A1 |
20070244372 | Merkle | Oct 2007 | A1 |
20070250134 | Miesel et al. | Oct 2007 | A1 |
20070288266 | Sysko et al. | Dec 2007 | A1 |
20080010089 | DiMaggio et al. | Jan 2008 | A1 |
20080015891 | Lee | Jan 2008 | A1 |
20080020877 | Bogner | Jan 2008 | A1 |
20080059232 | Iliff | Mar 2008 | A1 |
20080076976 | Sakurai et al. | Mar 2008 | A1 |
20080077489 | Gilley et al. | Mar 2008 | A1 |
20080091084 | Yudkovitch et al. | Apr 2008 | A1 |
20080109412 | Drayer et al. | May 2008 | A1 |
20080119958 | Bear et al. | May 2008 | A1 |
20080133269 | Ching | Jun 2008 | A1 |
20080133716 | Rao et al. | Jun 2008 | A1 |
20080140449 | Hayes | Jun 2008 | A1 |
20080147440 | Kil | Jun 2008 | A1 |
20080147441 | Kil | Jun 2008 | A1 |
20080147688 | Beekmann et al. | Jun 2008 | A1 |
20080200771 | Brown | Aug 2008 | A1 |
20080201174 | Ramasubramanian et al. | Aug 2008 | A1 |
20080208777 | Stephens | Aug 2008 | A1 |
20080229213 | Hamilton et al. | Sep 2008 | A1 |
20080238666 | Loncar | Oct 2008 | A1 |
20080249806 | Dlugos et al. | Oct 2008 | A1 |
20080312510 | Ross | Dec 2008 | A1 |
20080313256 | Kanazawa et al. | Dec 2008 | A1 |
20090018862 | Sanger et al. | Jan 2009 | A1 |
20090037470 | Schmidt | Feb 2009 | A1 |
20090048865 | Breazeale, Jr. | Feb 2009 | A1 |
20090055150 | Prior et al. | Feb 2009 | A1 |
20090131758 | Heywood | May 2009 | A1 |
20090144089 | Heywood et al. | Jun 2009 | A1 |
20090150180 | Cohen et al. | Jun 2009 | A1 |
20090172773 | Moore | Jul 2009 | A1 |
20090222284 | McEachern | Sep 2009 | A1 |
20090234755 | Sidoruk | Sep 2009 | A1 |
20100070306 | Dvorak et al. | Mar 2010 | A1 |
20100131860 | DeHaan et al. | May 2010 | A1 |
20100286490 | Koverzin | Nov 2010 | A1 |
20110029895 | Ternouth | Feb 2011 | A1 |
20110184747 | Bozic et al. | Jul 2011 | A1 |
20120116685 | Alaupovic | May 2012 | A1 |
20120129139 | Partovi | May 2012 | A1 |
20120265552 | Rabinowitz et al. | Oct 2012 | A1 |
20130024207 | Anderson et al. | Jan 2013 | A1 |
20130253940 | Zziwa | Sep 2013 | A1 |
20140100885 | Stern | Apr 2014 | A1 |
20140257856 | Sasai | Sep 2014 | A1 |
20140350954 | Ellis et al. | Nov 2014 | A1 |
20150112710 | Haber | Apr 2015 | A1 |
20150161331 | Oleynik | Jun 2015 | A1 |
20160103973 | Singal et al. | Apr 2016 | A1 |
20160135706 | Sullivan | May 2016 | A1 |
20160188807 | Heywood et al. | Jun 2016 | A1 |
20160203281 | Zalis et al. | Jul 2016 | A1 |
20160228003 | Apte | Aug 2016 | A1 |
20160232312 | Apte | Aug 2016 | A1 |
20160300015 | Natarajan et al. | Oct 2016 | A1 |
20170206327 | Heywood et al. | Jul 2017 | A1 |
20170249434 | Brunner | Aug 2017 | A1 |
20170262604 | Francois | Sep 2017 | A1 |
20170281020 | Mulligan | Oct 2017 | A1 |
20170308671 | Bahrami | Oct 2017 | A1 |
20180308569 | Luellen | Oct 2018 | A1 |
20190043610 | Vaughan | Feb 2019 | A1 |
20190139641 | Itu | May 2019 | A1 |
20190211378 | Apte | Jul 2019 | A1 |
20190339291 | Edmonds | Nov 2019 | A1 |
20190347744 | Heywood et al. | Nov 2019 | A1 |
20200003762 | Brown | Jan 2020 | A1 |
20200185063 | Narain | Jun 2020 | A1 |
20200279622 | Heywood et al. | Sep 2020 | A1 |
20210057046 | Liu | Feb 2021 | A1 |
20210247403 | Arvey | Aug 2021 | A1 |
20210256630 | Heywood et al. | Aug 2021 | A1 |
Number | Date | Country |
---|---|---|
2884613 | Mar 2014 | CA |
3703404 | Aug 1988 | DE |
0912957 | Dec 2004 | EP |
3287530 | Feb 2018 | EP |
H07271857 | Oct 1995 | JP |
H08140944 | Jun 1996 | JP |
2001175761 | Jun 2001 | JP |
2001331581 | Nov 2001 | JP |
2001331585 | Nov 2001 | JP |
2002011057 | Jan 2002 | JP |
2002041670 | Feb 2002 | JP |
2002056099 | Feb 2002 | JP |
2002095641 | Apr 2002 | JP |
2002512712 | Apr 2002 | JP |
2002245172 | Aug 2002 | JP |
2002245180 | Aug 2002 | JP |
2002539561 | Nov 2002 | JP |
2002366662 | Dec 2002 | JP |
2003010288 | Jan 2003 | JP |
2003108679 | Apr 2003 | JP |
2003175005 | Jun 2003 | JP |
2003186995 | Jul 2003 | JP |
2003256573 | Sep 2003 | JP |
2003337864 | Nov 2003 | JP |
2004178264 | Jun 2004 | JP |
2005004398 | Jan 2005 | JP |
2005506601 | Mar 2005 | JP |
2005326943 | Nov 2005 | JP |
2006053628 | Feb 2006 | JP |
2006155071 | Jun 2006 | JP |
2006155411 | Jun 2006 | JP |
2006163489 | Jun 2006 | JP |
2006185396 | Jul 2006 | JP |
2006221471 | Aug 2006 | JP |
2006350992 | Dec 2006 | JP |
2007052774 | Mar 2007 | JP |
2007514207 | May 2007 | JP |
2007140905 | Jun 2007 | JP |
2007200093 | Aug 2007 | JP |
2007525154 | Sep 2007 | JP |
2007265347 | Oct 2007 | JP |
2008513884 | May 2008 | JP |
2008177713 | Jul 2008 | JP |
2010500648 | Jan 2010 | JP |
2011501276 | Jan 2011 | JP |
2011501845 | Jan 2011 | JP |
WO 2000029983 | May 2000 | WO |
WO 2000055751 | Sep 2000 | WO |
WO 2001050950 | Jul 2001 | WO |
WO 2004080312 | Sep 2004 | WO |
WO 2007019504 | Feb 2007 | WO |
WO 2007023818 | Mar 2007 | WO |
WO 2007111910 | Oct 2007 | WO |
WO 2009049277 | Apr 2009 | WO |
WO 2009049278 | Apr 2009 | WO |
WO 2010126577 | Nov 2010 | WO |
WO 2010148365 | Dec 2010 | WO |
WO 2012154594 | Nov 2012 | WO |
WO 2014036312 | Mar 2014 | WO |
WO 2014039718 | Mar 2014 | WO |
WO 2014062981 | Apr 2014 | WO |
WO 2014144383 | Sep 2014 | WO |
WO 2014145123 | Sep 2014 | WO |
2015095343 | Jun 2015 | WO |
WO-2016141127 | Sep 2016 | WO |
WO-2017181147 | Oct 2017 | WO |
WO 20170173365 | Oct 2017 | WO |
WO-2017201540 | Nov 2017 | WO |
WO-2018079840 | May 2018 | WO |
WO-2018140014 | Aug 2018 | WO |
WO-2019103738 | May 2019 | WO |
Entry |
---|
Piri et al., “A data analytics approach to building a clinical decision support system for diabetic retinopathy: Developing and deploying a model ensemble,” Decision Support Systems 101 (2017) 12-27 (Year: 2017). |
Eom et al., “AptaCDSS-E: A classifier ensemble-based clinical decision support system for cardiovascular disease level prediction,” Expert Systems with Applications 34 (2008) 2465-2479 (Year: 2008). |
Aljaaf et al., “Early Prediction of Chronic Kidney Disease Using Machine Learning Supported by Predictive Analytics,” 2018 IEEE Congress on Evolutionary Computation (CEC) (Year: 2018). |
Margineanu et al., “Machine Learning Approach for Classifying Multiple Sclerosis Courses by Combining Clinical Data with Lesion Loads and Magnetic,” Frontiers in Neuroscience | www.frontiersin.org Jul. 1, 2017 | vol. 11 | Article 398 (Year: 2017). |
Bratic et al., “Machine Learning for Predicting Cognitive Diseases: Methods, Data Sources and Risk Factors,” J Med Syst (2018) 42: 243 (Year: 2018). |
Ko et al., “Machine learning to detect signatures of disease in liquid biopsies—a user's guide,” Lab Chip, 2018, 18, 395-405 (Year: 2018). |
Alves et al., “Progression of Motor Impairment and Disability in Parkinson Disease: A Population-Based Study,” Neurology, vol. 65, No. 9, pp. 1436-1441, Nov. 2005. |
Banerjee et al., “R-U-In?: Doing What You Like, with People Whom You Like” In Proceedings of the 17th international conference on World Wide Web, pp. 1239-1240, Apr. 2008. |
Baum et al., “A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains,” Annals of Mathematical Statistics, 1970, vol. 41, No. 1, pp. 164-171. |
Beenen et al., “Using Social Psychology to Motivate Contributions to Online Communities” Conference on Computer Supported Cooperative Work, Chicago, IL, Nov. 6-10, 2004. |
Cedarbaum et al., “Performance of the Amyotrophic Lateral Sclerosis Functional Rating Scale (ALSFRS) in multicenter clinical trials,” Journal of the Neurological Sciences, Oct. 1997, vol. 152, Supplement 1, pp. S1-S9. |
Cedarbaum et al., “The ALSFRS-R: a revised ALS functional rating scale that incorporates assessments of respiratory function,” Journal of the Neurological Sciences, 1999, vol. 169, pp. 13-21. |
Cosley et al., “SuggestBot: Using Intelligent Task Routing to Help People Find Work in Wikipedia”, IUI '07, Jan. 28-31, 2007, Honolulu, Hawaii, pp. 32-41. |
Cudkowicz et al., “Measures & Markers in Amyotrophic Lateral Sclerosis,” NeuroRx, Apr. 2004, vol. 1, pp. 273-283. |
Dayhoff et al., “Providing a Complete Online Multimedia Patient Record,” AMIA, Inc., 1999, pp. 241-245. |
Deneault et al., “An Integrative Display for Patient Monitoring,” IEEE International Conference on Systems, Man & Cybernetics Conference Proceedings, 1990, pp. 515-517. |
Emmons et al., “Counting Blessings Versus Burdens: An Experimental Investigation of Gratitude and Subjective Well-Being in Daily Life,” Journal of Personality and Social Psychology, 2003, vol. 84, No. 2, pp. 377-389. |
Eton et al., “Harmonizing and Consolidating the Measurement of Patient-Reported Information at Health Care Institutions: a Position Statement of the Mayo Clinic,” Patient Related Outcome Measures, pp. 7-15, 2014. |
Foraker et al., “Her-based Visualization Tool: Adoption Rates, Satisfaction, and Patient Outcomes,” eGEMS (Generating Evidence & Methods to Improve Patient Outcomes), vol. 3, Iss. 2, pp. 1-14, 2015. |
Fornai et al., “Lithium delays progression of amyotrophic lateral sclerosis,” Proceedings of the National Academy of Sciences, Feb. 12, 2008, vol. 105, No. 6, pp. 2052-2057. |
Frankowski et al., “Recommenders Everywhere: the WikiLens Community-Maintained Recommender System”, WikiSym '07, Oct. 21-23, 2007, Montreal, Quebec, Canada, pp. 47-59. |
Frost et al., “Social Uses of Personal Health Information Within PatientsLikeMe, an Online Patient Community: What Can Happen When Patients Have Access to One Another's Data,” Journal of Medical Internet Research, May 27, 2008, vol. 10, No. 3, p. e15. |
Geyer et al., “Recommending Topics for Self-Descriptions in Online User Profiles”, RecSys '08, Oct. 23-25, 2008, Lausanne, Switzerland, pp. 59-66. |
Goetz, “Practicing Patients,” The New York Times Magazine, Mar. 23, 2008. |
Gordon, “Advances in Clinical Trials for Amyotrophic Lateral Sclerosis,” Current Neurology & Neuroscience Reports, 2005, vol. 5, pp. 48-54. |
Gordon et al., “Progression rate of ALSFRS-R at time of diagnosis predicts survival time in ALS,” Neurology, Oct. 2006, vol. 67, pp. 1314-1315. |
Harper et al., “Talk Amongst Yourselves: Inviting Users to Participate in Online Conversations”, IUI '07, Jan. 28-31, 2007, Honolulu, Hawaii, pp. 62-71. |
Ikemoto et al., “Diabetes nursing support system using mobile phones,” Information Processing Society of Japan Research Report 2006-DBS-138 Database System 2006-GN-58 Groupware and Network IPSJ, Information Processing Society of Japan, Jan. 27, 2006, vol. 2006, No. 9, pp. 197-202. |
Iriberri et al., “A Life-Cycle Perspective on Online Community Success”, ACM Computing Surveys, vol. 41, No. 2, Article 11, Feb. 2009. |
Johnson et al., “Prediction of the Rate of Decline in Cognitive Function in Alzheimer's Disease: A Model based on Simple Demographic Data and Widely Used Rating Scales” Dementia and Geriatric Disorders, vol. 15, No. 4, pp. 276-282, Sep. 2003. |
Kasarskis et al., “Rating the severity of ALS by caregivers over the phone using ALSFRS-R,” Amyotrophic Lateral Sclerosis & Other Motor Neuron Disorders, 2004, vol. 5, Supplement 2, p. 12. |
Kasarskis et al., “Rating the severity of ALS by caregivers over the phone using ALSFRS-R,” Amyotrophic Lateral Sclerosis, Mar. 2005, vol. 6, Iss. 1, pp. 50-54. |
Koh et al., “Encouraging Participating in Virtual Communities”, Communications of the ACM, col. 50, No. 2, Feb. 2007, pp. 69-73. |
Litt et al., “Graphical Representation of Medical Information in the Visual Chart,” Seventh Annual IEEE Symposium on Computer-Based Medical Systems, 1994, pp. 252-257. |
Liu et al., “Towards a Rich-Context Participatory Cyberenvironment”, International Workshops on Grid Computing Environments 2007, Nov. 11-12, 2007, Reno, Nevada. |
Long et al., “Web Interface for the Heart Disease Program,” Proceedings: AMIA Symposium, 1996, pp. 762-766. |
Marquardt, “An Algorithm for Least-Squares Estimation of Nonlinear Parameters,” Journal of the Society for Industrial and Applied Mathematics, Jun. 1963, vol. 11, Iss. 2, pp. 431-441. |
Miller et al., “Is the ALSFRS-R Rate of Decline Linear Over Time?,” Amyotrophic Lateral Sclerosis, Jan. 2007, vol. 8, Supplement 1, pp. 140-155. |
Montes et al., “Development & Evaluation of self-administered version of the ALSFRS-R,” Neurology, Oct. 2006, vol. 67, pp. 1294-1296. |
Mould, “Developing Models of Disease Progression”, Pharmacometrics: the Science of Quantitative Pharmacology Wiley, Hoboken, pp. 547-581, 2007. |
Nielsen, “Participation Inequality: Encouraging More Users to Contribute”, Participation Inequality in Social Design, Oct. 2006. |
Ogura et al., “Clinical Trial of Risedronate in Japanese Volunteers: A Study on the Effects of Timing of Dosing on Absorption,” Journal of Bone and Mineral Metabolism, 2004, vol. 22, pp. 120-126. |
Orimo et al., “Graphical Output of Health Testing Data,” Medical Informatics, 1990, vol. 15, Iss. 2, pp. 141-149. |
“Pediatric Research Program Issue APS-SPR,” San Diego Convention Center, San Diego, CA, May 7-11, 1995, vol. 37, No. 4 Part 2, p. 139A (Apr. 1995). |
Rashid et al., “Motivating Participating by Displaying the Value of Contribution”, CHI 2006, Apr. 22-27, 2006, Montreal, Quebec, Canada, pp. 955-958. |
Seligman et al., “Positive Psychology Progress,” American Psychologist, Jul.-Aug. 2005, pp. 410-421. |
U.S. FDA, “Guideline for Industry: Dose Response Information to Support Drug Registration,” U.S. FDA; Federal Register, Nov. 9, 1994, vol. 59, No. 216. |
Wicks et al., “Accelerated clinical discovery using self-reported patient data collected online and a patient-matching algorithm,” Nature Biotechnology, May 2011, vol. 29, No. 5, pp. 411-414. |
Number | Date | Country | |
---|---|---|---|
62818310 | Mar 2019 | US | |
62774788 | Dec 2018 | US |