Patent Grant
11380425
The present disclosure relates generally rehabilitative techniques and, more particularly, to a computer-assisted method for assessing a patient.
An “outcomes measure,” also known as an “outcomes assessment tool,” is a series of items used to determine varying medical conditions or functional status of a patient. One outcomes measure is the Functional Independence Measure (FIM®), which provides a method of measuring functional status. The assessment contains eighteen items composed of motor tasks (13 items) and cognitive tasks (5 items). Tasks are rated by a clinician on a seven-point ordinal scale that ranges from total assistance to complete independence. Scores range from 7 (lowest) to 91 (highest) for motor skills and 7 to 35 for cognition skills. Items include eating, grooming, bathing, upper body dressing, lower body dressing, toileting, bladder management, bowel management, bed to chair transfer, toilet transfer, shower transfer, locomotion (ambulatory or wheelchair level), stairs, cognitive comprehension, expression, social interaction, problem solving, and memory.
The FIM measure uses a scoring criteria that ranges from a score of 1 (which reflects total assistance) to a score of 7 (which reflects complete independence). A score of 7 is intended to reflect that a patient has complete independence. A score of 1 is intended to reflect that a patient can perform less than 25% of the task or requires more than one person to assist. As a result of this scoring system, many patients who make improvements in a free-standing inpatient rehabilitation facility or an inpatient rehabilitation unit within a hospital do not necessarily register gains in their outcomes score during their rehabilitation. For instance, a spinal cord injury patient may make significant improvements to fine finger skill motor skills during rehabilitation, allowing the patient to use a computer or a smart phone. However, his or her FIM score in this situation would not improve.
An outcomes measure is needed that more accurately captures assessment of a patient's medical condition or functional status. Additionally, an outcomes measure is needed that helps to better identify areas in which patients, such as rehabilitation patients, can improve.
An “item” is a question or other kind of assessment used in an outcomes measure. For example, one item on an outcomes measure known as the Berg Balance Scale instructs a patient as follows: “Please stand up. Try not to use your hands for support.” A “rating” is a score outcome or other evaluation in response to an item assessment. For example, the ratings for the Berg Balance Scale item are as follows: a rating of 4, which reflects that the patient is able to stand without using her hands and stabilize independently; a rating of 3, which reflects that the patient is able to stand independently using her hands; a rating of 2, which reflects that the patient is able to stand using her hands after several tries; a rating of 1, which reflects that the patient needs minimal aid from another to stand or to stabilize; and a rating of 0, which reflects that the patient needs moderate or maximal assistance from another to stand.
Classical test theory is a body of related psychometric theory that predicts outcomes of educational assessment and psychological testing such as the difficulty of items or the ability of test-takers. It is a theory of testing based on the idea that a person's observed or obtained score on a test is the sum of a true score (error-free score) and an error score. Classical test theory assumes that each person has a true score, T, that would be obtained if there were no errors in measurement. A person's true score is defined as the expected number-correct score over an infinite number of independent administrations of the test. Unfortunately, test users never observe a person's true score, only an observed score, X. It is assumed that observed score=true score plus some error, or X=T+E, where X is the observed score, T is the true score, and E is the error. The reliability, i.e., the overall consistency of a measure, of the observed test score X is defined as the ratio of true score variance to the observed score variance. Because the variance of the observed scores can be shown to equal the sum of the variance of true scores and the variance of error scores, this formulates a signal-to-noise ratio wherein reliability of test scores becomes higher as the proportion of error variance in the test scores becomes lower and vice versa. The reliability is equal to the proportion of the variance in the test scores that could be explained if the true scores were known. The square root of the reliability is the correlation between true and observed scores. Estimates of reliability can be obtained by various means, such as the parallel test or a measure of internal consistency known as Cronbach's coefficient α. Cronbach's α can be shown to provide a lower bound for reliability, and thus, the reliability of test scores in a population is always higher than the value of Cronbach's α in that population.
The problem of accurately measuring improvements in rehabilitation patients is solved by developing an outcomes assessment that incorporates factor analysis and item response theory.
The problem of measuring improvements in rehabilitation patients is solved by asking a series of questions to the patient and returning a domain-specific and/or composite score.
The problem of improving care in rehabilitation patients is solved by predicting the domain-specific and/or composite score on an outcomes measurement of the patient and providing a clinical intervention if the domain-specific and/or composite score falls below the predicted score.
While the appended claims set forth the features of the present techniques with particularity, these techniques, together with their objects and advantages, may be best understood from the following detailed description taken in conjunction with the accompanying drawings of which:
A “bifactor model” is a structural model wherein items cluster onto a specific factors while at the same time loading onto a general factor.
The term “categorical” is used to describe response options for which there is no explicit or implied order or ranking.
A “Comparative Fit Index” (CFI) compares the performance of the constructed structural model against the performance of a model that postulates no relationships between variables. A good-fitting model generally has a CFI greater than 0.95.
A “complex structure” is a CFA structural model where at least one item loads onto more than one factor.
“Confirmatory factor analysis” (CFA) is a form of factor analysis utilized where a psychometrician has an understanding of how the latent traits and items should be grouped and related. A structural model is developed, and this is fit to the data. A goal of the model is to achieve a good fit with the data.
A “constraint” is a restriction imposed on a model for the sake of mathematical stability or the application of content area theory. For example, if two factors in a confirmatory factor analysis are not expected to have any relationship between them, a constraint on that correlation (requiring that it equal 0.00) can be added to the model.
A “continuous” variable is a variable that is measured without categories, like time, height, weight, etc.
A “covariate” is a variable in a model that is not on a measure, but may still have some explanatory power. For example, in rehabilitation research, it may occasionally be useful to include covariates for age, sex, length of stay, diagnostic group, etc.
“Dichotomous” describes response options that are ordinal with two categories (e.g., low versus high). Alternatively, it may refer to items that are scored correct vs. incorrect, which are conceptually also ordinal responses with two categories.
“Differential item functioning” (DIF), in item response theory, is a measure of how the parameter estimates may behave differently from group to group (in different samples) or from observation to observation (over time).
“Difficulty,” in item response theory, is the required minimum level of a latent trait that is necessary to respond in a certain way. On a measure with dichotomous responses, there is a single difficulty (e.g., the minimum level of the latent trait that will raise the probability of answering correctly to 50% or greater). On a measure with polytomous responses, “difficulty” is better described as “severity,” as there is not usually a correct or incorrect answer. On a measure with polytomous responses, the number of difficulties estimated is k−1, where k is the number of response options. These difficulties describe the level of latent trait necessary to endorse the next highest category. Sometimes also referred to as a threshold.
“Dimension” refers to the number of latent traits a measure addresses. A measure that records one trait is said to be unidimensional, while a measure recording more than one trait is referred to as multidimensional.
“Discrimination” is the ability of a test to differentiate between people with high versus low ability of the latent trait. Similarly, it describes the magnitude of the relationship between an item and a latent trait. Conceptually, it is very similar to a factor loading and mathematically, it can be converted into a factor loading.
“Endorse” means to select a response option.
“Equality constraint” in item response theory and confirmatory factor analysis, is a mathematical requirement to constrain the discriminations or factor loadings to be equal when only two items load onto a factor.
“Equating” refers to the use of item response theory to draw similarities between the scores on different measures that record the level of the same latent trait(s). Equating may also be used to compare alternate forms of the same measure.
“Error” refers to a term to describe the amount of uncertainty surrounding a model. A model with parameter estimates that are very close to the observed data will have low amounts of error, while one that is quite different would have a large amount of error. Error may also indicate the amount of uncertainty surrounding a specific parameter estimate itself.
“Estimation” refers to the statistical process of deriving parameter estimates from the data. These procedures may be performed using specialized psychometric software known in the art.
“Exploratory factor analysis” is a form of factor analysis that clusters items according to their correlations. This is often done without any direction from the analyst other than how many factors should be extracted. The groupings are then “rotated.” Rotation methods attempt to find factor loadings that are indicative of simple structure by making sure that factor loadings are pushed towards −1.00, 0.00, or 1.00.
“Factor” in factor analysis describes a latent trait. Unlike a latent trait in item response theory, factors do not normally have scores associated with them.
“Factor analysis” is a statistical method for determining the strength and direction of the relationships between factors and items. The data on which factor analysis is based are the correlations between items. Factor analysis can accommodate either ordinal or continuous data, but not unordered categorical. It is possible to compute scores from factor analysis, but IRT scores are more reliable. May either be exploratory or confirmatory.
“Factor correlation” refers to a correlation between two factors. A CFA model with correlated factors is called “oblique.”
“Factor loading,” in factor analysis, describes the magnitude of the relationship between an item and a factor. It is not mathematically the same as a correlation, though its scale and interpretation are similar. That is, values (usually) range from −1.00 to 1.00. A strong negative factor loading indicates a strong inverse relationship between an item and a latent trait, while a strong positive loading has the opposite interpretation. A factor loading of 0.00 indicates no relationship whatsoever.
“Fit statistics” or “fit index” refer to metrics used to quantify how well the model performs. Popular fit metrics confirmatory factor analysis and structural equation modeling include the root mean square error of approximation (RMSEA), the comparative fit index (CFI), the Tucker-Lewis Index (TLI), and the weighted root mean-square residual/standardized root mean-square residual (WRMR/SRMR).
“General factor,” in a bifactor model, refers to the factor onto which all items load.
“Graded response model” (GRM) is an extension of the two-parameter logistic model that allows for ordinal responses. Instead of only one difficulty, the graded response model yields k−1 difficulties, where k is the number of response categories.
“Hierarchical model” is a structural model where latent traits load onto other latent traits, forming a hierarchy.
“Higher-/lower-order factor,” in a hierarchical model, is a higher-order factor is a type of latent variable onto which lower-order factors load.
“Index” is a term used to refer to a fit index/statistic (e.g., comparative fit index) or as a synonym for “measure.”
“Item” refers to the questions, tasks, or ratings on a measure that are addressed by a respondent or a respondent's representative (such as a clinician).
“Item characteristic curve (ICC)” is a graph that plots the probability of selecting different response options given the level of a latent trait. Sometimes it also is called a “trace line.”
“Item response theory” (IRT) is a collection of statistical models used to obtain scores and determine item behavior according to a structural model. In one used form, IRT uses the response pattern of every person in the sample in order to get these item and score estimates. IRT uses data that are ordinal or categorical. Mathematically speaking, item response theory uses item and person characteristics in order to predict the probability that a person selects a certain response option on a given item.
“IRT score” is a score specific to IRT analysis that is given on a standardized scale. It is similar to a z-score. In one IRT scoring system, a score of 0.00 implies someone has an average level of a latent trait, a large negative score implies a low level of the latent trait, and a large positive value implies a large level of a latent trait.
“Latent trait” is similar to a factor in factor analysis, but used more often in item response theory. A latent trait is what a related set of items purports to measure. It may be used interchangeably with factor, domain, or dimension.
“Latent variable” is a term for a variable that is not measured directly. It includes latent traits.
“Linking” is similar to equating, but for item parameter estimates rather than scores.
“Load” is a verb used to describe what an item does on a factor. For example: “Item 4 loads onto the both the local dependence factor as well as the general factor in this model.”
“Local dependence” (LD) is a violation of the local independence assumption in which items are related for some reason other than the latent trait. If local dependence appears to exist in the data, it can be accounted for by either modeling a correlation between the items or by creating a local dependence factor. This can be due to a large number of reasons, such as similar wording, nearly identical content, and the location of the items on a measure (this last example occurs frequently on the last items of a long measure).
“Local independence” refers to an assumption in psychometrics that states that the behavior of items is due to the latent traits in the model and item-specific error and nothing else. When items violate this assumption, they are said to be locally dependent.
“Manifest variable” is a generic term for a variable that is measured directly, and includes items, covariates, and other such variables.
“Measure” or “measurement” refers to a collection of items that attempt to measure the level of some latent trait. It may be used interchangeably with assessment, test, questionnaire, index, or scale.
“Model” in psychometrics is a combination of the response model and the structural model. In general terms, it describes both the format of the data and how the data recorded in the model's variables should be related.
“Model fit” is a term used to describe how well a model describes the data. This may be done in a variety of ways, such as comparing the observed data against the predictions made by the model or comparing the chosen model against a null model (a model in which none of the variables are related). The metrics used to assess model fit are called fit statistics.
“Multidimensional” is a term used to describe a measure that records more than one latent trait.
“Multigroup analysis” in IRT refers to the process by which the sample can be split into different groups, and parameter estimates specific to each group may be estimated.
“Nominal model” is similar to the graded response model, but for items with response options that are categorical rather than ordinal.
“Oblique” is an adjective used to describe factors that are correlated.
“Ordinal” describes the way an item records data. For example, possible responses to an item are a series of categories ordered from low to high or high to low.
“Orthogonal” describes factors that are restricted to a zero correlation.
“Parameter estimate” is a statistically-derived value estimated by psychometric software. It is a generic term that may include things like item discriminations, factor loadings, or factor correlations.
“Path diagram” is a diagram meant to illustrate the relationships between items, latent traits, and covariates. In a path diagram, rectangles/squares represent observed variables (i.e., items, covariates, or any modeled variable for which there is explicitly recorded information), ovals/circles represent latent traits or variables for which there is no explicitly recorded information, one-headed arrows reflect one-directional relationships (as in a regression), and two-headed arrows reflect correlation/covariance between modeled variables.
“Polytomous” is a term for items with more than one response option, and may be either ordinal or categorical.
“Pseudobifactor model” is a bifactor model where not all items cluster onto specific factors. Instead, some items may only load onto the general factor.
“Psychometrician” is a kind of statistician that specializes in measurement.
“Psychometrics” describes the statistics used in creating or describing measures.
“Rasch model” is a response model that hypothesizes that all item discriminations are equal to 1.00. It usually is not used unless this assumption is true or nearly true. This assumption eases interpretation of scores and difficulties and allows use of item response theory on (relatively) small sample sizes, but it is very uncommon that all item discriminations behave identically. It is a simplified case of the two-parameter logistic model, which allows the item discriminations to vary. Because of this, the Rasch model is sometimes referred to as the one-parameter logistic model (1PL). It may be used when the responses are dichotomous.
A “respondent” is someone who answers items on a measurement.
A “response” is a respondent's answer to an item.
“Response categories” are the different options a respondent may select as a response to an item. If items yield dichotomous responses, the data are recorded as either correct (1) or incorrect (0).
“Response model,” in item response theory, refers to the way a measurement model handles the format of the responses. Popular response models include the Rasch model, the two-parameter logistic model, the three-parameter logistic model, the graded response model, and the nominal model.
A “response pattern” is a series of numbers representing a respondent's answers to each question on a measurement.
“Root mean square error of approximation” (RMSEA) is a fit statistic in applied psychometrics. It measures the closeness of the expected data (the data that the model would produce) against the observed data. It is usually desirable that the RMSEA is below 0.08, though some of ordinary skill in the art desire that the RMSEA be below 0.05.
A “score” is a numeric value meant to represent the level or amount of the latent trait a respondent possesses. Classical test theory computes scores as the sum of item responses, while item response theory estimates these using both response patterns and item qualities.
“Sigmoid” (literally, “S-shaped”) is an adjective is occasionally used to describe the shape of the TCC or the ICC of a 2PL item.
“Simple structure” is a structural model where all items load onto one factor at a time.
“Specific factor,” in a bifactor model, is a factor onto which a set of items load.
“Structural equation modeling” (SEM) is an extension of confirmatory factor analysis (CFA) that allows relationships between latent variables like latent traits. If all latent variables in the model are latent traits, structural equation modeling (SEM) and CFA are often used interchangeably.
“Structural model” is a mathematical description that represents a system of hypotheses regarding the relationships between latent traits and items. It is depicted as a path diagram.
“Sum score” is a score computed by summing the numeric value of all responses on a measure.
“Sum score conversion” (SSC) is a table that shows the relationship between the sum scores and an IRT scores.
“Test characteristic curve” (TCC) is a figure that plots the relationship between sum scores and IRT scores.
“Testlet” is a small collection of items that measure some component of the overall latent trait. Creating a measure comprised of testlets can lead to more easily interpreted scores when the definition of the latent trait is clearly defined beforehand.
“Threshold”: see “difficulty”.
“Tucker-Lewis Index” (TLI) is a fit index that compares the performance of the constructed model against the performance of a model that postulates no relationships between the variables. A good fitting model usually has a TLI of greater than 0.95.
“Three-parameter logistic model” (3PL) is an extension of the two-parameter logistic model that also includes a “guessing” parameter. For example, in a multiple-choice item with 4 choices, even guessing randomly results in a 25% chance of answering correctly. The 3PL allows for this non-zero chance of answering correctly. It is used when responses are dichotomous.
“Trace line”: see “item characteristic curve.”
“Two-parameter logistic model” (2PL) is like the Rasch Model, but allows item discriminations to vary. It may be used when item responses are dichotomous.
“Unidimensional” is a term used to describe a measure that records only one latent trait.
“Variable” is a generic word used to describe a set of directly (manifest) or indirectly (latent) recorded data that measures a single thing.
“Weighted root mean-square error/standardized root mean-square error” (WRMR/SRMR) is a fit statistic that measures the magnitude of a model's residuals. Residuals are the differences between the observed data and the data that the model predicts. The typical recommended WRMR value is below 1.00, though this recommendation may change based on size of the sample or complexity of the model. The WRMR is used when there is at least one categorical variable in the model, while the SRMR is used when all variables are continuous.
In 101, an item set 200 is identified. In an embodiment, clinicians may be queried to provide their input on appropriate items to include in the item set 200, based on their training, education, and experience. Examples of clinicians may include physicians, physical therapists, occupational therapists, speech language pathologists, nurses, and PCTs. Items from the item set 200 may come from a variety of outcomes measures known in the art.
In 102, the items from the item set 200 may be grouped into one or more of a plurality of areas, called “domains”, that are relevant to therapy or clinical outcomes. The clinicians may identify these domains. In an embodiment, items from the item set 200 may be grouped into three domains, titled “Self-Care”, “Mobility”, and “Cognition”. It should be understood that other groupings of additional and/or alternative domains are possible.
In 103, a related analysis step may occur. For instance, the frequency with which an item in the item set 200 is used in traditional practice to assess a patient in a medical setting may be analyzed. Alternately, the cost of equipment to conduct the item may be assessed. The clinical literature may be reviewed to identify the outcomes measures with items in the item set 200 that are psychometrically acceptable and clinically useful. For instance, the reliability and validity of an outcomes measure with one or more items in the item set 200 may be reviewed to ensure it is psychometrically acceptable. As another example, each outcomes measure and/or item may be reviewed to ensure it is clinically useful. For example, while there are many items used to test a person's balance that are available in the literature, not all of them are appropriate for patients in a rehabilitation context. Based on these and similar factors, the initial set of items may be narrowed to reduce the burden to patients, clinicians, and other health care providers.
In 104, a revised plurality of items is collected. A pilot study may be conducted on the plurality of items. The pilot study may be conducted by having clinicians assess patients on the revised items in a standardized fashion, such that each clinician assesses each patient using all of the revised items. In another embodiment, the clinicians may select which items should be used to assess a patient, based on the patient's particular clinical characteristics. The determination as to selection of specific items may be made based on information received during the patient's rehabilitation stay, for instance, at the inpatient evaluation at admission. The item may be administered at least twice during the patient's inpatient stay in order to determine patient progress. The pilot study may be facilitated using an electronic medical record system, such that clinicians enter item scores into the electronic medical record.
In 105, a pilot study analysis may be performed. For instance, items that take too much time for a clinician to conduct with a patient may be removed.
In 106, the original paper-based items for the preliminary outcomes measure 100 are implemented in an electronic medical record. Individual item-level rating can be recorded electronically. For instance, the items to be implemented may be the items that are the result of the pilot study analysis in 105. However, pilot study analysis is not required. Alternately, the items in the preliminary outcomes measure 100 could be implemented in an electronic system, such as a database, that is external to an electronic medical record. In one embodiment, the external electronic system may be in communication with the electronic medical record, using methods that are known in the art, such as database connection technologies. In 107, the items for the preliminary outcomes measure 100 are programmed into the EMR using known methods, allowing clinicians to input their ratings into the electronic medical record. In an embodiment, the EMR may provide a prompt to alert, remind, and/or require the clinician to enter certain ratings for certain items of the preliminary outcomes measure 100. Such prompts may improve the reliability and completeness of clinician data entry into the EMR.
Although the discussion above with reference to
The assessment may be an initial assessment conducted at or shortly after the time of admission of the patient to a hospital. In an embodiment, each patient receiving care during a period of time, such as a month or a year, is assessed. In another embodiment, a majority of patients receiving care during a period of time are assessed. In yet another embodiment, a plurality of patients are assessed. In other embodiments, the patient population may be refined to include only inpatients, only outpatients, or a combination thereof.
In various embodiments, certain tests in the preliminary outcomes measure 100 may be conducted once at or shortly after admission, and again at or shortly prior to discharge. In various embodiments, certain tests in the preliminary outcomes measure 100 may be conducted weekly. In various embodiments, certain tests in the preliminary outcomes measure 100 may be conducted more than once per week, such as twice per week.
In an embodiment, the assessments may be conducted in a centralized location specific to conducting assessments. The assessments may be conducted by a set of clinicians whose specific function is to conduct assessments. A centralized location with qualified staff and adequate equipment to objectively assess a patient's functional performance may be conducted through a standardized process in a controlled and safe environment. In an embodiment, a clinician provides an order for a lab technician assessment. For example, a clinician (such as a physiatrist, therapist, nurse, or psychologist) orders a specific test (such as a test of gait and balance) or a group of tests. The test order may be sent electronically to the assessment department (“AAL”) and a hard copy may be printed for the patient. When the AAL is ready, the patient may travel to the AAL, with assistance if necessary. Staff, such as a technician, performs the ordered test(s). Test results may be recorded and entered/transmitted into the electronic medical record. The clinician may review test results to modify care plan if necessary. This process can reduce the amount of time clinicians require to learn how to conduct a test. One benefit of an AAL is that other clinicians do not need to learn how to conduct various tests every time a new test is introduced. Clinicians will only need to learn how to read the test results, not how to conduct the test. Qualified personnel with proper training can perform the tests. Clinical staff can focus on treatment rather than on assessment. More treatment sessions or additional time can be provided to improve outcomes. Test equipment is centrally kept to reduce the need for multiple units and maintenance costs. Tests can be conducted in a well-controlled, standardized and safe environment. The technician may utilize standardized procedures to avoid potential rater induced bias (tendency for higher ratings to show improvement over time), thus improving data quality.
The ratings from each assessment may be saved in the EMR. For instance, they may be saved in a preliminary ratings dataset 150. In 202, data analysis and cleanup may be performed on the preliminary ratings dataset 150 to improve data quality. For example, out-of-range ratings may be removed from the preliminary ratings dataset 150. Patterns of data in the preliminary ratings dataset 150 from the same clinician may be reviewed and cleaned using methods known in the art. Ratings in the preliminary ratings dataset 150 from patients that show a large increase in rating from “dependent” to “independent” may also be discarded. Suspect data from a particular evaluation may be discarded.
In 203, the ratings data may be further extracted, cleaned, and prepared using methods known in the art to get the data in a form in which the data may be queried and analyzed. Data may be reviewed for quality, and various data options, such as data pivoting, data merging, and creation of a data dictionary may be performed for the preliminary ratings dataset 150. Data from the preliminary ratings dataset 150 may be stored in the EMR or in a different form, such as in a data warehouse, for further analysis. It will be understood by one of ordinary skill in the art that many ways exist to structure the data in the preliminary ratings dataset 150 for analysis. In one embodiment, the preliminary ratings dataset 150 is structured so that item ratings are available for analysis across a plurality of dimensions, such as time period and patient identification.
Once the preliminary ratings dataset 150 has been prepared for analysis, a psychometric evaluation may be performed on the preliminary ratings dataset 150. A psychometric evaluation assesses how well an outcomes measure actually measures what it is intended to measure. A psychometric evaluation may include a combination of classical test theory analysis, factor analysis, and item response theory, and assesses the preliminary ratings dataset 150 for various aspects, which may include reliability, validity, responsiveness, dimensionality, item/test information, differential item functioning, and equating (score crosswalk). In one embodiment, classical test theory analysis may be employed to review the reliability of the items in the preliminary outcomes measure 100, and how the preliminary outcomes measure 100 and the domain work together.
Item Reduction.
The item reduction step 152 assists in reducing the items from the preliminary outcomes measure 100 that do not work as anticipated. Factors can include reliability, validity, and responsiveness (also known as sensitivity to change). The purpose of the item reduction step 152 is to eliminate potential item content redundancy from items in the preliminary outcomes measure 100 to a minimal subset of items in an IRT outcomes measure 180 without sacrificing the psychometric properties of the data set. The item reduction step 152 may be performed using a computer or other computing device, for instance, using a computer program 125. The computer program 125 may be written in the R programming language or another appropriate programming language. The computer program 125 provides an option to allow the number of desired items (as well as options to include specific items) to be specified and computes the Cronbach's coefficient α reliability estimate for every possible combination of items within those user-defined constraints. Acceptable ranges of Cronbach's coefficient α may also be defined in the computer program 125. Additionally, the computer program 125 may construct and run syntax for a statistical modeling program, such as Mplus (Muthen & Muthen, Los Angeles, Calif., http://www.statmodel.com) to determine the fit of a 1-factor confirmatory factor analysis (CFA) model to each reduced subset 155 of items.
The computer program 125 may be used to analyze several of the outcomes measures included in the preliminary outcomes measure 100 (such as the FIST, BBS, FGA, ARAT, and MASA), and searched for unidimensional subsets between four and eight items with Cronbach's α reliabilities between 0.70 and 0.95. Using these constraints, the number of items in many measures may be reduced substantially. For instance, measures may be reduced by at least half of their original length while maintaining good psychometric properties. The resulting item subsets served as building blocks for the confirmatory factor analysis (CFA). In an embodiment, certain items may not be included in the item reduction process, such as the items from the FIM®. In an embodiment, the item reduction step 152 may performed multiple times. For instance, it may be performed on each outcomes measure included in the preliminary outcomes measure 100.
In the item reduction step 152, the computer program 125 determines the extent to which items are related to each other. The computer program 125 may determine the extent to which items within an outcomes measure in the preliminary outcomes measure 100 are related to each other. In one embodiment, items are related to each other within an outcomes measure if they have responses which correlate highly. The analysis may start by providing an initial core set of items, the number of which may be determined with clinician input, based on correlations between item pairs. For example, the computer program 125 may determine how item A relates to item B, where item A and item B are both in the same outcomes measure. If there is a high correlation, both item A and item B are included in the core set. Then, the computer program 125 may determine how new item C correlates to the set of items {A, B}. If there is a high correlation, item C is included in the core set. The method may be repeated with additional items, D, E, F, etc. As described above, the program assess the reliability (Cronbach's α) of every possible subset of items. The program correlates the responses from one set of items with the responses from a second set of items. Chronbach's α is known in the art but a brief example is hereby provided. The information used in computing Cronbach's α are the correlations between every possible split-half in the subset of items. For example, using 3 items {A, B, C}, Cronbach's α averages the correlations A vs. BC, B vs. AC, and C vs. AB. In other words, the correlations are computed between every pair of unique subsets of a set. The purpose of the correlational analysis is to help ensure that items are measuring the same underlying construct and improving reliability.
Table 1 lists an exemplary output of the item reduction step 152 for the Berg Balance Scale (“BBS”) outcomes measure, setting the sample size equal to five items. The numbers in each cell in the “item” columns reflect the number of the question on the BBS (1: sitting unsupported; 2: change of position—sitting to standing; 3: change of position—standing to sitting; 4: transfers; 5: standing unsupported; 6: standing with eyes closed; 7: standing with feet together; 8: tandem standing; 9: standing on one leg; 10: turning trunk (feet fixed)). Each reduced subset 155 is shown along with its associated Cronbach's α value. The first reduced subset has the highest Cronbach's α of the reduced subsets in Table 1. In one embodiment, the reduced subset with the highest Cronbach's α is used as the initial reduced subset for the CFA step 160, which is described below in further detail.
Confirmatory Factor Analysis.
Factor analysis is a statistical method that is used to determine the number of underlying dimensions contained in a set of observed variables and to identify the subset of variables that corresponds to each of the underlying dimensions. The underlying dimensions can be referred to as continuous latent variables or factors. The observed variables (also known as items) are referred to as indicators. Confirmatory factor analysis (CFA) can be used in situations where the dimensionality of a set of variables for a given population is already known because of previous research. CFA may be used to investigate whether the established dimensionality and factor-loading pattern fits a new sample from the same population. This is the “confirmatory” aspect of the analysis. CFA may also be used to investigate whether the established dimensionality and factor-loading pattern fits a sample from a new population. In addition, the factor model can be used to study the characteristics of individuals by examining factor variances and covariances/correlations. Factor variances show the degree of heterogeneity of a factor. Factor correlations show the strength of association between factors.
Confirmatory factor analysis (CFA) may be performed using Mplus or other statistical software to validate how well the item composition within the pre-specified factor structure holds statistically. CFA is characterized by restrictions on factor loadings, factor variances, and factor covariances/correlations. CFA requires at least m{circumflex over ( )}2 restrictions where m is the number of factors. CFA can include correlated residuals that can be useful for representing the influence of minor factors on the variables. A set of background variables can be included as part of a CFA.
Mplus can estimate CFA models and CFA models with background variables for a single or multiple groups. Factor indicators for CFA models can be continuous, censored, binary, ordered categorical (ordinal), counts, or combinations of these variable types. When factor indicators are all continuous, Mplus has seven estimator choices: maximum likelihood (ML), maximum likelihood with robust standard errors and chi-square (MLR, MLF, MLM, MLMV), generalized least squares (GLS), and weighted least squares (WLS) also referred to as ADF. When at least one factor indicator is binary or ordered categorical, Mplus has seven estimator choices: weighted least squares (WLS), robust weighted least squares (WLSM, WLSMV), maximum likelihood (ML), maximum likelihood with robust standard errors and chi-square (MLR, MLF), and unweighted least squares (ULS). When at least one factor indicator is censored, unordered categorical, or a count, Mplus has six estimator choices: weighted least squares (WLS), robust weighted least squares (WLSM, WLSMV), maximum likelihood (ML), and maximum likelihood with robust standard errors and chi-square (MLR, MLF).
Using the highly-reliable subsets of items from the measure reduction step, a model may be defined in statistical software such as Mplus that hypothesizes that all items within a domain are interrelated. The model also may measure specific constructs under the preview of that domain. For example, all item subsets taken from Self Care measures may be hypothesized to measure Self Care, but also simultaneously measure one of Balance, Upper Extremity Function, and Swallowing. Constructing the model in this way allows for the measurement of both an overall domain (e.g., Self Care) as well as a set of interrelated constructs that compose that domain (e.g., Balance, UE Function, and Swallowing—the constructs composing Self Care). Given the data, the structure of the model implies a set of expected correlations between each pair of items. However, these (polychoric) correlations can be computed directly from the data. These are the observed correlations. The appropriateness of the constructed model, called “model fit” in statistics, may be determined using the root mean square error of approximation (RMSEA), which is a measure of the difference between the observed and expected correlations. In a preferred embodiment, if the value of that difference is low (for instance, less than 0.08) the model has acceptable fit.
After applying the CFA step 160 on a reduced subset 155, the output of the CFA step 160 may contain factor loadings, including a General Factor loading. The General Factor loading may be between −1 and 1, with values of the General Factor loading of between 0.2-0.7 indicating whether a factor assesses the relevant item well. The output of the CFA step 160 may provide additional factor loadings for each item. In an embodiment, each item may have a factor loading for each sub-domain. For instance, each item may have a factor loading value for Balance, a factor loading value for Upper Extremity, a factor loading value for Swallowing, and a factor loading value for each other sub-domain. Where the item is relevant to a sub-domain, the factor loading value will be non-zero, in an embodiment.
In certain instances, applying the CFA step 160 on a reduced subset 155 can create problems that require selection of a new reduced subset 155. For instance, a general factor loading value higher than 0.7, or particularly a value closer to 1.0, indicates redundancy. For instance, the way items are scored on the Action Research Arms Test (ARAT) outcomes measure necessarily forces too high of a reliability. Patients who achieve a maximum score on the first (most difficult) item are credited with having scored 3 on all subsequent items on that scale. If the patient scores less than 3 on the first item, then the second item is assessed. This is the easiest item, and if patients score 0 then they are unlikely to achieve a score above 0 for the remainder of the items and are credited with a zero for the other items. This method of scoring forces the too-high reliability. In other instances, if a factor loading value is greater than 1, it reflects that a pair of items has a negative variance (which is not possible) and so the CFA step 160 must be run on a new reduced subset 155. A new reduced subset 155 may be selected from the group of reduced subsets generated by item reduction step 152. For instance, new reduced subset may be selected that has the next-highest Chronbach's α, then applying CFA step 160 to the new reduced subset.
Additionally, during the process of running the CFA step 160, it may be apparent that items designated by clinicians as falling within one sub-domain should be moved to a different sub-domain in order to improve the fit of the model used to generate the IRT outcomes measure 180 (discussed further below). For example, during the development of the embodiments described herein, items identified by clinicians as relating to “Strength” were initially placed in the Self-Care domain. In running the CFA step 160, however, it was determined that these items did not fit the model. Moving these items to the “Upper Extremity Function” sub-domain improved the fit of the model.
Table 2 below shows the fit statistics of a 1-factor CFA containing groupings 1-10 set out in Table 1. In the CFA step 160, an assessment may be conducted as to whether the fit statistics listed in Table B meet usual “good fit” criteria. In one embodiment, these criteria are RMSEA<0.08, CFI>0.95, TLI>0.95, and WRMR<1.00. Those of ordinary skill in the art will appreciate that other good fit criteria could be used.
Although the example above is given only with respect to one outcomes measure, the Berg Balance Scale, it should be understood that the CFA step 160 is applied to each outcomes measure in the preliminary outcomes measure 100.
Item Response Theory. In an embodiment, the IRT outcomes measure 180 may be structured to contain a plurality of high-level domains. For example, the IRT outcomes measure 180 may be structured to include a “Self Care” domain (which includes items determined to reflect a patient's capability to perform self care), a “Mobility” domain (which includes items determined to reflect a patient's capability to be mobile), and a “Cognition” domain (which includes items determined to reflect a patient's cognitive capabilities). Within each higher-level domain, specific assessment areas, also referred to as “factors” or “clusters,” may be identified. Table 3 reflects exemplary assessment areas associated with each higher-level domain.
Because the measurement goals of the IRT outcomes measure 180 involved measuring general domains (i.e., Self Care, Mobility, and Cognition) as well as specific assessment areas within those domains, a bifactor structure for each of the domains may be targeted (the general factor and domain-specific factor). The composition of the specific factors may be determined by the content of each item set. For example, items from the FIST, BBS, and FGA may be combined to form the “Balance” assessment area within the Self Care domain. Acceptable fit of the bifactor model to the data was assessed using the criterion of RMSEA<0.08 (Browne & Cudeck, 1992), and modification indices were also computed to check for local item dependence and potential improvements to the model, such as additional cross-loadings (in other words, an item contributes to several factors).
Item Response Theory reflects a mathematical model that describes the relationship between a person's ability and item characteristics (such as the difficulty). For example, a more able person is more likely to be able to perform a harder task, and can allow a more tailored intervention based on a series of questions. Other item characteristics may be relevant as well, such as an item's “discrimination,” which is its ability to distinguish between people with high or low levels of a trait.
After constructing the CFA models for each of the domains, the final structures may be coded to run in an item response theory software package, such as flexMIRT (Vector Psychometric Group, Chapel Hill, N.C., US). flexMIRT is a multilevel, multidimensional, and multiple group item response theory (IRT) software package for item analysis and test scoring. The multidimensional graded response model (M-GRM) may be chosen to account for the ordered, categorical nature of the item responses from the clinician-rated performance ratings. For example, the dimensions may be “Self-Care,” “Mobility,” and “Cognition”. Sub-domains for “Self-Care” may be “Balance, “Upper Extremity Function,” “Strength,” “Changing Body Position”, and “Swallowing.” Sub-domains for “Mobility” may be Balance, Wheelchair (“W/C”) Skills, Changing Body Positions, Bed Mobility, and Mobility. Sub-domains for “Cognition” may be “Awareness,” “Agitation,” “Memory,” “Speech,” and “Communication.”
In a preferred embodiment, however, sub-domains may be reduced in order to focus on key subdomains of ability. For “Self-Care”, for example, these may be Balance, UE Function, and Swallowing. For “Cognition” these may be Cognition, Memory, and Communication. For “Mobility,” there may be no sub-domains—in other words, the sub-domains may all be clustered together.
The analysis also may be multigroup in nature. For example, the Self Care and Mobility samples may be split into groups determined by the level of balance (sitting, standing, or walking). As another example, the Cognition sample may be split into broad diagnostic categories (stroke, brain injury, neurological, or not relevant). In an embodiment, in order to accommodate the complexity of the models, the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm (Cai, 2010) may be used for more efficient parameter estimation. MH-RM cycles through the following three steps repeatedly until the differences between two consecutive cycles are smaller than a chosen criterion. In Step 1 (Imputation), random samples of the latent traits are imputed from a distribution implied by the item parameter estimates taken from the preceding cycle. If it is the first cycle, then the distribution implied by the algorithm's starting values are used. This imputation can be performed using the MH sampler. In Step 2 (Approximation), the log-likelihood of the imputed data is evaluated. In Step 3 (Robbins-Monro Update), new parameter estimates for the next cycle are computed by using the Robbins-Monro filter on the log-likelihood in step 2. Step 1 is then repeated using the information from Step 3. Slopes can reflect item discriminations and intercepts can reflect item difficulties.
In addition to the item slopes and intercepts, maximum a posteriori (MAP) latent trait scores, which reflect the patient's level of ability, may be computed for each patient.
The principal coding for IRT focuses on translating the mathematical structure chosen after CFA into one that can be assessed using IRT. The data used for the analysis may be, for instance, simply the patients' ratings on all items on which they were assessed. For consistency, the most recent available data for each patient on each item they were administered may be used. This has the convenience of putting patient scores in a particular frame of reference: typical discharge level. MAP (maximum a posteriori) scoring may be used, but other scoring methods are known which could be employed instead, such as ML (maximum likelihood), EAP (expected a posteriori), or MI (multiple imputation). Additionally there are different estimation methods that could be employed. For instance, marginal maximum likelihood using the expectation-maximization algorithm (MML-EM) may be used. However, this method can suffer when working with more than a few dimensions. In a preferred embodiment, the Metropolis-Hastings Robbins-Munro (MH-RM) estimation is used.
Maximum a posteriori (MAP) scoring requires two inputs: the scoring density of the population (usually assumed to be standard normal for each dimension) and the IRT parameters for each item that a patient was rated on. Multiplying the population density by the IRT functions for each item results in what is known as a likelihood—in other words, mathematical representation of the probability of various scores, given what is known about the items and how the patient was rated on each of the items. The location of the maximum value of that function is the patient's MAP score.
Sometimes, response options on an item are only selected very rarely, which may cause problems with estimating IRT parameters for that item (and also implies that that response option may have been unnecessary). In such cases, those responses can be collapsed into an adjacent category. For example, if an item has responses {1, 2, 3, 4} and response 2 is very rarely seen in the data, we may recode the data {1, 2, 2, 3}. It should be understood that the actual value of the number is unimportant in IRT analysis, and that instead, the ordinality matters.
Group composition: The IRT analyses used here can be multigroup in nature to allow for more targeted assessment. For Self Care and Mobility, patients may be grouped according to their level of balance (none, sitting, standing, and walking). Similarly, groups may be formed in the cognitive domain according to their cognitive diagnosis (stroke, brain injury, neurological, or none). This method can result in multiple test forms that only contain items appropriate for each patient. For instance, they may contain test forms as follows: for “Self-Care” and “Mobility”, no balance, sitting balance, (up to) standing balance, and no balance restrictions; for “Cognition,” stroke, brain injury, neurological, or not disordered. The forms may tailored according to group membership, rather than to assessment areas. For example, the patient's balance level may affect which balance measure items appear on the Self Care and Mobility domains, while the patient's cognitive diagnosis (if any) may affect which measures may appear on the form. For instance, the ABS is only used on the Brain Injury form of the Cognition measure and the KFNAP is only used on the Stroke measure.)
Item response theory results in a distinct score for each domain. For example, a patient may score a 1.2 in the “Self-Care” domain, a 1.4 in the “Mobility” domain, and a 3 in the “Cognition” domain. In an embodiment, these scores may be reported to clinicians, patients, and others separately. In other embodiments, these scores may be combined into a single score. In an embodiment, a score of +1 means the patient is 1 logit above average. A score of −1 means the patient is 1 logit below average. Values below −3 and above 3 are highly improbable, because the mathematical assumptions underlying IRT are such that scores follow an average distribution. It should be recognized by one or ordinary skill in the art that other numbers reflecting the standard deviation and logit could be employed instead. For instance, a score of 3 could mean the patient is average, so scores would range between 0 and 6. As another example, a score of 50 could mean the patient is average, and a score of +10 could mean that the patient is 1 logit above average, so scores would range from 20 to 80.
An example of running the IRT step 170 is now provided, with respect to the Self-Care domain. Seven factors are provided to the IRT step 170: the Self Care factor, the Balance factor, the UE Functioning factor, the Swallowing factor, a hidden factor for ARAT, a hidden factor for overcoming a negative correlation between the FIST and FGA outcomes measures, and a hidden factor specific to the FIST so it is not overweighted in the result. The IRT step 170 (for instance, using the MH-RM estimation) returns a discrimination matrix 172 and a difficulty matrix 174. For instance, these matrices may be presented in slope/intercept formulation, where slope reflects item discrimination and intercept reflects item difficulty.
Table 4 displays an exemplary discrimination matrix 172 for the Self-Care domain of an exemplary IRT outcomes measure 180. The column headings a1-a7 in Table 4 represent the following, with “hidden” factors listed in parentheses: (a1: Self Care; a2: (ARAT local dependence); a3: Upper Extremity Function; a4: Swallowing; a5: Balance; a6: (Reduction of FIST influence); a7: (Negative relationship of BBS and FGA)). Table 4 lists the slope values for each item for each factor a1-a7. The item naming in Table 4 is also reflected in Table 6 in Appendix 1, listing the items in an exemplary IRT outcomes measure 180.
Table 5 displays an exemplary difficulty matrix 174 for an IRT outcomes measure 180. Table 5 displays the intercept values for each item, for each factor d1-d6. The column headings d1-d6 in Table 5 represent the following, with “hidden” factors listed in parentheses: (d1: Self Care; d2: (ARAT local dependence); d3: Upper Extremity Function; d4: Swallowing; d5: Balance; d6: (Reduction of FIST influence)).
It should be understood that discrimination matrix 172 and a difficulty matrix 174 may be prepare for each domain in the IRT outcomes measure 180.
An exemplary score/probability response may be plotted, where the X-axis reflects the score and the Y-axis reflects the probability of response. The product of the curves results in a likelihood curve that somewhat appears like a bell curve. The peak of the curve can be used as the score for the patient.
Input from Therapists to Ensure Clinical Relevance.
Each item may be labeled with a cluster that most appropriately describes its role in the IRT outcomes measure 180. This labeling may be done by clinicians on the basis of their education, training, and experience. For example, a clinician may label an item that measures balance, such as items that test function in sitting, as falling within the “Mobility” domain and the “Balance” factor in Table 1.
Because item selection (retention or removal) in the item reduction step of the analysis is predicated on psychometric and statistical evaluations, in an embodiment, clinical experts may review the item content covered in the reduced item sets for further feedback. For example, a pool of clinicians may be surveyed for input on whether items should be added or removed from the subsets taken from each of the full outcomes measures. Their input may be used to construct the final models for each domain, to help ensure the retained items are psychometrically sound and clinically relevant.
Remodeling to Derive Final Sets of Items.
After the negotiated item sets, considering both psychometric evaluation and clinical judgement, were in place, the CFA and IRT steps may be carried out. Left-out items with a large clinical endorsement may be added back into the models, while included items with low endorsement may be removed. The fit of the models to the data may then be assessed using the root mean square error of approximation (RMSEA) computed during the CFA, and new item parameter estimates and latent trait scores were computed during the IRT analysis. Table 6 in Appendix 1 to this Specification lists the items in a preferred exemplary IRT outcomes measure 180.
Display
Various aspects of data relating to an individual patient's score may be displayed for a clinician and/or a patient.
The value of the IRT score becomes apparent from an analysis of
However, the FIM score is deficient in showing gains when progress is made within a FIM level. Suppose another patient is admitted with a score of −2.00, and progresses all the way to −1.00. Even though the patient made just as much progress as the previous patient (+1.00), it still looks like the patient has not improved her functional level on upper body dressing, as the change in the FIM for this item is 0. As a result, one benefit of the IRT score is that it can detect improvement where the FIM cannot. In our experience, the expected change in Self Care for individuals with Nontraumatic spinal cord injury and Neurological injuries is fairly dramatic when using the IRT.
Prediction estimates may be derived in a variety of ways. In one embodiment, Hierarchical Linear Modeling (HLM) may be used, incorporating information regarding past patients' diagnoses, the severity of those diagnoses (the “case mix group”, a measure of the patient's condition's severity within a diagnosis), the days on which measures were administered to the patients, and the scores on those days. The modeling may output a predictive curve for every severity within every diagnosis for up to 50 days of inpatient stay. When plotting the information, the x-axis may be the number of days since admission and the y-axis may be the IRT (MAP) score.
Other methods of prediction could be used, including data science methods like neural networks and random forest models. Furthermore, additional patient information may be incorporated in the prediction process.
In an embodiment, a patient may be assessed using the IRT outcomes measure 180 over multiple days. For instance, the patient may be assessed on a first subset of questions from the IRT outcomes measure 180 on a first day, and then assessed on a second subset of questions on a second day. The data feed may be set up such that it collects the most recent item value.
Adaptive testing may be employed, such that the items in the IRT outcomes measure 180 are selected for assessment in response to the score from an already assessed item. For example, the clinician may assess the patient with the items in the IRT outcomes measure 180 from the FIST test; compute an initial IRT score based off the results; and then select a next item (or a plurality of next items) most appropriate, based on the initial IRT score. This process may be applied iteratively until the patient's score can be determined to be accurate within a pre-determined uncertainty level. For instance, once uncertainty is at or below 0.3, the adaptive testing method may stop providing additional items for assessment and provide a final IRT score for the patient, clinician, or others to review.
A clinician may review the IRT score with the patient's score on a particular FIM item to determine whether additional interventions are appropriate. For example, if the patient has AQ score of 1, a score of 4 on the FIM toileting measure is expected. But, if the FIM toileting measure is lower, the clinician can use that as an indication to adjust therapy to specifically target improved toileting.
Prediction
Prediction of AQ score may be based on various factors, such as medical service group; case mix group (CMG); and/or lengths of stay. Within CMG, age may be a factor used to assist in the prediction.
Data generated by predictive models may be used in various ways. For example, a patient's length of stay can be predicted via his/her medical condition, level of impairment, and other demographic and clinical characteristics. As another example, if a patient is below their prediction on a given domain, clinicians can target those areas for more focused therapies. As another example, if a patient's progress in one domain has begun to taper off, clinicians could note this and prioritize balanced treatment in that domain. As another example, given some financial information, it would be possible to assess the dollar value of expected improvement over a period of time and compare it to the cost of inpatient care over that same time frame. Discharge decisions could be made using the ratio of value of care to cost of care. Additionally, predicting success in other treatment settings is possible. Given similar assessments in other levels and locations of care (e.g., outpatient, SNF, etc.), a prospective look at the course of improvement in those settings could be determined. Better decisions regarding care in those settings could potentially be made.
The present application claims the priority benefit of U.S. Provisional Patent Application 62/563,960, filed Sep. 27, 2017, which is incorporated herein by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8463623 | Ware et al. | Jun 2013 | B2 |
| 20060282306 | Thissen-Roe | Dec 2006 | A1 |
| 20090075246 | Stevens | Mar 2009 | A1 |
| 20090202969 | Beauchamp et al. | Aug 2009 | A1 |
| 20100191071 | Anderson et al. | Jul 2010 | A1 |
| 20120075586 | Kirschen | Mar 2012 | A1 |
| 20120108909 | Slobounov et al. | May 2012 | A1 |
| 20130096942 | Bowles | Apr 2013 | A1 |
| 20140081658 | Irving et al. | Mar 2014 | A1 |
| 20140279727 | Baraniuk et al. | Sep 2014 | A1 |
| 20160293036 | Niemi | Oct 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 2005-228305 | Aug 2005 | JP |
| Entry |
|---|
| Pathways to Success “Clinical Assessment Toolbox” Aug. 2016 (Year: 2016). |
| Fabio La Porta et al. Unified Balance Scale: An activity-based, bed to community, and aetiology-independent measure of balance calibrated with Rasch analysisJ Rehabil Med 2011; 00: 00-00 (Year: 2011). |
| PCT Search Report and Written Opinion issued in related application PCT/US2018/052875, dated Dec. 6, 2018, 7 pages. |
| Bjorner et al., “Developing tailored instruments: item banking and computerized adaptive assessment,” Quality of Life Research, Feb. 15, 2007, vol. 16, pp. 95-108. |
| Cella et al., “A Discussion of Item Response Theory and Its Applications in Health Status Assessment,” Medical Care, Sep. 2000, vol. 38(9), Suppl. II, pp. 66-72. |
| Cella et al., “Defining higher order dimensions of self-reported health: further evidence for a two-dimensional structure,” Evaluation & The Health Professions, Jun. 2005, vol. 28(2), pp. 122-141. |
| Chang, “Finding Two Dimensions in MMPI-2 Depression,” Structural Equation Modeling, 1996, vol. 3(1), pp. 41-49. |
| Chang et al., “Equating health-related quality of life instruments in applied oncology settings,” Physical Medicine and Rehabilitations: State of the Art Reviews, 1997, vol. 11(2), pp. 397-406. |
| Chang et al., “Detecting Unexpected Variables in the MMPI-2 Social Introversion Scale,” Journal of Applied Measurements, 2001, vol. 2(3), pp. 227-240. |
| Chang et al., “Real-Time Clinical Application of Quality-of-Life Assessment in Advanced Lung Cancer,” Clinical Lung Cancer, Sep. 2002, vol. 4(2), pp. 104-109. |
| Chang et al., “Should symptoms be scaled for intensity, frequency, or both?” Palliative and Supportive Care, Mar. 2003, vol. 1(1), pp. 51-60. |
| Chang et al., “Psychometric evaluation of the Chicago Multiscale Depression Inventory in multiple sclerosis patients,” Multiple Sclerosis, Mar. 2003, vol. 9(2), pp. 160-170. |
| Chang et al., “The Washington Psychosocial Seizure Inventory (WPSI): psychometric evaluation and future applications,” Seizure, Jul. 2003, vol. 12(5), pp. 261-267. |
| Chang et al., “Item Response Theory and Its Applications to Patient-Reported Outcomes Measurement,” Evaluation & The Health Professions, Sep. 2005, vol. 28(3), pp. 264-282. |
| Chang et al., “The SF-36 physical and mental health factors were confirmed in cancer and HIV/AIDS patients,” Journal of Clinical Epidemiology, Jan. 2007, vol. 60(1), pp. 68-72. |
| Chang, “Patient-reported outcomes measurement and management with innovative methodologies and technologies,” Quality of Life Research, May 26, 2007, vol. 16, pp. 157-166. |
| Chang et al., “A System for Interactive Assessment and Management in Palliative Care,” Journal of Pain and Symptom Management, Jun. 2007, vol. 33(6), pp. 745-755. |
| Chang et al., “Psychometric evaluation of the Taiwan Chinese version of the EORTC QLQ-PR25 for HRQOL assessment in prostate cancer patients,” Health and Quality of Life Outcomes, Aug. 2012, vol. 10:96, 10 pages. |
| Edwards, “An Overview of Item Response Theory,” Vector Psychometric Group, Sep. 2014, <https://www.vpgcentral.com/wp-content/uploads/2014/10/VPG-DIA-9.21.14.pdf>, 50 pages. |
| Gehlert et al., “Establishing the Diagnostic Validity of Premenstrual Dysphoric Disorder Using Rasch Analysis,” Journal of Outcome Measurement, 1997, vol. 1(1), pp. 2-18. |
| Gehlert et al., “Factor Structure and Dimensionality of the Multidimensional Health Locus of Control Scales in Measuring Adults with Epilepsy,” Journal of Outcome Measurement, 1998, vol. 2(3), pp. 173-190. |
| Gehlert et al., “The WOMQOL instrument measured quality of life in women of reproductive age with no known pathology,” Journal of Clinical Epidemiology, May 2006, vol. 59(5), pp. 525-533. |
| Leonelli et al., “Interpretation of a Full-Information Item-Level Factor Analysis of the MMPI-2: Normative Sampling and Nonpathognomonic Descriptors,” Journal of Personality Assessment, Jun. 2000, vol. 74(3), pp. 400-422. |
| Liang et al., “An empirical comparison of the WHOQOL-BREF and the SGRQ among patients with COPD,” Quality of Life Research, 2008, vol. 17, pp. 793-800. |
| Lin et al., “Development of a real-time clinical decision support system upon the web mvc-based architecture for prostate cancer treatment,” BMC Medical Informatics and Decision Making, 2011, vol. 11(1): 16, 11 pages. |
| Lin et al., “A real time online assessment system with modelized architecture on clinical infometrics for patient reported outcomes of prostate cancer,” Computer Methods and Programs in Biomedicine, Jun. 2012, vol. 106(3), pp. 249-259. |
| Muthen et al., “Examples: Confirmatory Factor Analysis and Structure Equation Modeling,” Mplus User's Guide, Seventh Edition, Chapter 5, Sep. 2012, Los Angeles, CA, US, pp. 55-110. |
| Richards et al., “Palliative Care Symptom Assessment for Patients with Cancer in the Emergency Department: Validation of the Screen for Palliative and End-of-Life Care Needs in the Emergency Department Instrument,” Journal of Palliative Medicine, Jun. 2011, vol. 14(6), pp. 757-764. |
| Snyder et al., “Can Patient-Reported Outcome Measures Identify Cancer Patients' Most Bothersome Issues?” Journal of Clinical Oncology, Mar. 20, 2011, vol. 29(9), pp. 1216-1220. |
| Snyder et al., “Feasibility and Value of Patient Viewpoint: A Web System for Patient-Reported Outcomes Assessment in Clinical Practice,” Psychooncology, Apr. 2013, vol. 22(4), pp. 895-901. |
| Thissen et al., “Methodological issues for building item banks and computerized adaptive scales,” Quality of Life Research, 2007, vol. 16, pp. 109-119. |
| Wolf et al., Development and validation of the Communication and Attitudinal Self-Efficacy scale for cancer (CASE-cancer), Patient Education and Counseling, Jun. 2005, vol. 57(3), pp. 333-341. |
| Zheng et al., “Content-balancing strategy in bifactor computerized adaptive patient-reported outcome measurement,” Quality of Life Research, Apr. 2013, vol. 22(3), pp. 491-499. |
| Examination Report issued in related application AU 2018342443, dated Nov. 3, 2021, 5 pages. |
| Reigart, “Why Standardized Tests are so Important,” May 12, 2017, retrieved from internet at <https://fprehab.com/2017/05/12/why-standardized-tests-are-so-important/> on Feb. 23, 2021, 3 pages. |
| Examination Report issued in related application AU 2018342443, dated Feb. 9, 2021, 6 pages. |
| Office Action issued in related application JP 2020-517303, dated Mar. 30, 2021, with English language translation, 7 pages. |
| Anonymous, “Electronic health record,” Wikipedia, Aug. 31, 2017, retrieved from Internet: https://en.wikipedia.org/w/index.php?title=Electronic_health_record&oldid=798111921 on May 14, 2021, 44 pages. |
| Anonymous, “Rasch Model,” Wikipedia, Jun. 7, 2017, retrieved from Internet: https://en.wikipedia.org/w/index.php?title+Rasch_Model&oldid=784244967 on May 14, 2021, 11 pages. |
| Haley et al., “Activity Outcome Measurement for Postacute Care,” Medical Care, Jan. 1, 2004, vol. 42(1), pp. I-49-I-61. |
| Haley et al., “Computerized Adaptive Testing for Follow-Up After Discharge From Inpatient Rehabilitation: I. Activity Outcomes,” Archives of Physical Medicine and Rehabilitation, Elsevier, Amsterdam, NL, Jul. 28, 2006, vol. 87(8) pp. 1033-1042. |
| World Health Organization, “International Classification of Functioning Disability and Health (ICF),” Dec. 31, 2001, retrieved from Internet: https://apps.who.int/iris/bitstream/handle/10665/42407/9241545429.pdf on May 14, 2021, 315 pages. |
| Office Action issued in related application CA 3,076,219, dated Apr. 30, 2021, 7 pages. |
| Extended European Search Report and Search Opinion issued in related application EP 18861460.6, dated May 26, 2021, 12 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20190096513 A1 | Mar 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62563960 | Sep 2017 | US |
