Patent Application
20170220773
Clinical studies are generally very expensive to undertake and may often take a long time to complete. During the course of a clinical study, many steps may be taken to ensure optimal execution of the study and maintain high quality of the data collected. These steps may be applied to various aspects of the study data, study participants, and study execution at the level of study sites, which constitute the main operating units in a clinical study. One component of site (and study) monitoring is to monitor the progress of a clinical study.
Up to now, progress of a clinical study has generally been measured very coarsely, either based on the progress of the whole study or possibly based on the progress of each site. For example,
Where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements. Moreover, some of the blocks depicted in the drawings may be combined into a single function.
In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the invention. However, it will be understood by those of ordinary skill in the art that the embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present invention.
Poor performance in the progress of a clinical study may be indicated by large delays or completion differences at one site compared to another, or by not meeting expected study targets. By being able to identify progress trends on a site basis, a clinical study basis, or other parametrical basis, site monitors may be able to identify poor-performing sites or slow sites, and concentrate on monitoring on those sites, with the possible goal of finishing the clinical study more quickly and therefore saving money for the sponsor or contract research organization (“CRO”). Such monitoring may include calling the site and asking why there are deviations or sending people to the site to verify the data or the collection process.
Embodiments of the present invention may be used to track the progress of clinical studies, but the invention is not limited to such embodiments. As alluded to above, a coarse study or site view may not provide enough information regarding the study's progress or the performance of each site. For example, site and study monitoring may not provide a comprehensive view of how data are collected and managed. In tracking the progress of a clinical study, one may examine a variety of data elements—a data point, a data record, or a data page. In a clinical study that records information using, for example, case report forms (“CRFs”), whether manual or electronic (“eCRF”), a data point (or “datapoint”) may be a field on such a form, a data record may be a group of fields on the form, and a data page (or “datapage”) may be the form itself. The fields may include a subject's demographic data (e.g., name, address, age, gender, birthdate) and may also include clinical data for a visit, e.g., blood pressure, temperature, heart rate, and pain assessment. A record may be a composite data element—a collection of several fields in a form. A datapage may also imply a composite data element in which history and statuses of individual (“lower”) data elements may be combined into a bigger picture. Any of these types of data elements may be monitored to understand the progress of the clinical study. One may track a clinical study by examining the chronology of various aspects related to these forms.
Traditional data tracking tools may use only the current status of these data elements (sometimes called “object status”) and may not include input from historical changes in the data collection process. Such progress tracking is additionally difficult to assess as it does not provide fine-grain monitoring that contextualizes the monitoring data. Rather than indicating that a site is not performing well, it would be beneficial to contextualize such monitoring data to indicate more specifically in which ways the site is not performing well.
To that end, the inventors have developed a context-sensitive system for tracking progress of a study, in this case, a clinical study, whereby the study is divided into stages, and the progress of each stage may be monitored to determine where progress may be stalled.
The study design elements in the Screening stage may include surveying demographics (S1) (such as by an intake or enrollment questionnaire), measuring vital signs (S2), and generating clinical laboratory data (S3) in order to screen against clinical study requirements and establish a baseline. Study design elements in the Drug Administration stage may include performing a physical exam (DA11, DA21, since there may be an exam in each drug administration sub-stage), measuring vital signs (DA12, DA22), dispensing the drug being studied (DA13, DA23), and possibly generating clinical laboratory data again (DA24). Study design elements in the Follow-up stage may include performing a physical exam (F1) and generating clinical laboratory data again (F2).
Three stages are shown in
The underlying relations and properties of clinical study design elements as shown in
For a given clinical (or custom) study stage, the system may provide progress status information and derive unique characterizations including metrics and predictions. Using this view of a clinical study, the fact that data entry has stalled in the drug administration stage may indicate delays in entering the data, whereas a lack of data in the screening (or enrollment) stage may indicate low enrollment rates in the study or the site. Such contextualized or finer-grain monitoring may provide both operational and clinical insights into the categorical domain of the data being collected, which may relate to adverse events (AEs), lab, drug administration, randomization, follow-up, and other stages, which may be reflected in the types of forms generated. For example, tracking AE-type forms may provide insights into the occurrence of AEs in the study. Similarly, tracking lab, drug administration, randomization, follow-up, and other types of forms may also be insightful.
A context-sensitive system for progress tracking may establish and represent study status in unique detail by using additional information about the data elements in the study. Specifically, the system may combine monitoring data based on progress curves for individuals or groups of data elements within parts of a study to implement metrics and tools that provide insight into the progress of the study. These progress curves may be generated as described in U.S. patent application Ser. No. 14/492,597, assigned to the same assignee and incorporated by reference herein in its entirety.
The system monitors how different stages of a study contribute to the progress of the study at a particular point in time. But sometimes data element classification and stages are not readily available, in which case the system may use a classification module to automatically generate classifiers that classify data elements by evaluating clinical data elements and thus enable annotation of clinical study stages with attributes in any clinical study. With annotated/classified data elements and characterized clinical study stages, the system may use an aggregation module to aggregate datapoints and other data elements in a clinical study, enabling comparison of contextualized study properties between subjects and/or sites within a study and/or between different studies. This specification will describe the stages in more detail, and then discuss the classification and aggregation.
Progress tracking that does not evaluate clinical study stage attributes and clinical data elements generally does not provide fine-grain monitoring that contextualizes the monitoring data. Such progress tracking is shown in
The example of a clinical study having three clinical study stages, screening, drug administration, and follow-up, some of which may have sub-stages such as DA1 and DA2 in drug administration, is shown in a slightly different configuration in
The system may aggregate the annotation of the study stages/sub-stages and data statuses to generate data representations in
The data may be summarized in other ways. The bars in
Using these system-generated annotations, the system may calculate and present the progress of a study for each subject, as shown in
Blocks 620, 630 may reside on one computer, having a processor and memory, or on a series of computers, processors, and memory that may themselves be connected via a local or wide-area network. Block 620 may comprise classifier application subsystem 621 and study stage annotation subsystem 629 in
When data elements classification and stages are not readily available, the system provides a set of modules in classifier generator 630 that, based on study metadata in clinical trial databases dB1 . . . dBn, induce data container classifiers that are applied in the analysis pipeline in block 620.
The clinical study design elements used by this system may be defined through clinical data containers, which were described above as generic data structures that allow various combinations of clinical data containers to form other clinical data containers and ultimately complete a clinical study. (Note that in this specification, we use study design elements and data containers interchangeably.) As an example, a clinical study may be defined by a collection of clinical data containers of type Folder, each of which contains multiple clinical data containers of type Form, each of which contains multiple clinical data containers of type Field. Some clinical study design elements may include demographics, vital signs, clinical laboratory, physical exam, drug exposure, etc.
In one embodiment, a minimal set of properties for these containers may include: <identifiers, content_properties, rendering_properties, parent, children>. “Identifiers” may represent one or more ways to refer to an instance of the container, such as name, sex, birthdate, place of birth. “Content_properties” may represent constraints (e.g., format, allowed range of values, default value) or enrichment (e.g., data dictionary) of the content in the clinical data container instance. “Rendering_properties” may represent behavior when a clinical data container instance (i.e., a data element) is displayed. “Children” may represent other clinical data containers that are part of this clinical data container.
For example, a form called “DEMOGRAPHICS” may have the following properties:
The form has the following fields (i.e., children):
where “type” is the type of clinical data container (e.g., folder, form, field), “edit” is the type of entry (e.g., “string” for an alphanumeric string, “choice” for a defined selection, “date”), “dictionary” is a reference to a pre-defined list of terms acceptable as data entries for the field, and “order” shows the order of fields in the form.
The system may normalize clinical data elements across different clinical studies for meaningful comparison and interpretation of the data from the different studies. Clinical data elements may be manually pre-annotated at creation time with standardized terms and categories or curated later. Examples of pre-annotations are therapeutic area for study, visit type for folder (e.g., screening, baseline, administration, follow-up), form type for form (e.g., SD™ domain such as vital signs, demographics), and standardized terms for field (e.g., subject sex, date of birth, medication name). (“SD™” stands for Study Data Tabulation Model, see www.cdisc.org/sdtm.) A set of tags may be assigned to all clinical data elements, and a tag may be a classifier that allows consolidating multiple clinical data elements.
For example, the system may reconcile the following two fields, “sex” and “gender,” by assigning the tag <tag:FIELD_TAG_DM_SEX>:
Additionally, the system may reconcile the following two forms by assigning the tag <tag:FORM_TAG_DM>:
As a general case though, complete and all-purpose annotation is unlikely to be encountered in data due to heterogeneous sources of data, legacy data, and variations in annotation definition and requirement. To enable annotation for broader sources of data, the system includes tools and methods to automatically characterize clinical data elements. Characterization may be based on supervised learning or unsupervised learning (without training data).
A system (shown in
In
Data container classifiers induction module 634 takes as input a set of clinical data container attribute values and generates data container classifiers (or “calls”), which are the class of the inputs represented by the attributes. The system may train the generated classifier in operation 760, using an annotated training set of data and a selected methodology such as, for example, Linear Regression, Support Vector Machines, and Maximum Entropy. Training typically means that many inputs are used to induce or generate the classifier. Each of the inputs has several attributes and a class assignment. For example, a classifier to determine whether an applicant should be given a line of credit may have several attributes (age, marital status, education, income, zip code) and the label (class) may be whether that candidate is approved. The system may test the generated classifier in operation 765, using an annotated test set of data. Once the classifier is trained/induced, then a similar set of previously unused inputs is used to assess the performance of the classifier in guessing the class on previously unseen inputs from a test dataset. Each of the test set inputs is classified by the classifier, and it is recorded whether the classification module “guessed” the label/class correctly. The system may validate the generated classifier in operation 770, such that given a clinical data element input, one or more attributes with a level of confidence may be returned by the classifier. Since the testing and training can be repeated many times on various combinations of inputs, validation is often used to assess the performance of the classifier on a completely different input set. In some ways, validation is a form of testing, but the use of different data is what makes it different.
An example of this process may be that given a set of clinical study clinical data containers (“training set”) in which each clinical data element is assigned, for example, a Therapeutic Area (e.g., Cardiovascular, Oncology, Endocrine, etc.), the system may build a set of features, which may consist of the term frequencies of all distinct words used across all names of forms in the study. The system may find all clinical data containers where type=“form” and may return an associated name. The system may then extract all unique words in the associated form names and may compute a word (term) frequency in a clinical study clinical data container. For example, all of the text in forms may be split by punctuation and white space leaving behind each word, and then the rate at which each specific word is found is counted. Thus, if “ECG” is prominent in the form names, then this may be used to classify a study as “cardiovascular”; similarly if “chemotherapy” is used frequently in a study, then it is likely that this is an oncology study.
The system may generate an N×M matrix, where N is the number of studies with a given annotation and M is the number of all unique terms found across all forms in all clinical data containers in the training set. The system may use this matrix and the N attribute assignments (or annotations) to induce a classifier using the matrix as a feature vector, which is a vector whose elements are attributes combined or transformed as suitable for the classifier, in this case term frequencies. For any given set of features, various machine learning methods, such as low-memory multinomial logistic regression, also known as maximum entropy, and Stabilized Linear Discriminant Analysis, may be used to induce a classifier.
When the attribute assigned to the clinical data container is not binary (i.e., it can take more than two values), the system may use two approaches to induce classifiers. A single multi-class classifier may be used, where one classifier may be induced and may be given a clinical data container input to the classifier to return one of the full set of attributes. Multiple two-class classifiers may also be used, where for each class value a separate classifier may be induced. Here the two-class classifier determines only if the clinical data container is in one of the multiple classes or not. For example, there may be N classes in which the clinical data containers are labeled C1, C2, . . . CN. For each of the N two-class classifiers (the i-th classifier), intermediate labels TRUE and FALSE may be created: TRUE, if the element has the label Ci, or FALSE, if the element does not have the label C. In this case, a clinical data container may be presented to all classifiers, and the set of classifications may determine whether an attribute is assigned or not to the clinical data container. Based on the confidence of the output of each classifier (which is either “N/A” (not able to make a call) or a class “Ci”), the system may decide which labels are assigned to the clinical data container (e.g., using a threshold for the confidence measure—how strong is the call of Ci).
In another embodiment, in the absence of a training set one may be built by using domain knowledge to define the annotations for data containers.
The process shown in
Besides the operations shown in
In general,
Based on the similarities between tags (e.g., label, dictionary, synonymous identifier), the system may use an algorithm as discussed below to group fields such as these, sex and gender, together and leave other fields to be grouped elsewhere.
One algorithm may be based on a distance function that may be a composite measure of similarity between various clinical data container attributes. For example, if each clinical data container has N properties, the following may define the distance function DIST between any two clinical data containers:
DIST(<attr1, attr2, . . . attrN>cde1, <attr1, attr2, . . . attrN>cde2, <sm1, sm2, . . . smN>, <w1, w2, . . . , wN>)
where attr=clinical data container attributes; sm=similarity measures (e.g., string distance, equivalence of values each returning a value between 0.0 and 1.0); and w=weights to ensure that w1*sm1+w2*sm2+ . . . +wN*smN=1.0. For this algorithm example, since distance is a decimal number between 0 and 1, two containers having a distance of 0.04 would be considered nearly identical, whereas two containers having a distance of 0.954 would not be considered at all similar. In the case in which two or more fields are grouped, a manual step may assign the FIELD_TAG_DM_SEX tag to these fields, or an automated tag may be generated based on most prominent attributes, for example SEX_GENDER or “Enter subject sex” for the fields above.
Referring now to
Another embodiment shows the progression of a study per custom study stage, which may allow the generation of a progress status model relative to a baseline or an industry standard computed from historical data of a group of clinical studies, as shown in
These progress statistics may be used by the system to generate an industry standard in operation 930 by comparing data across all subjects and sites in a study, as well as equivalent data from previous studies (and sites in previous studies). By classifying and annotating study data elements, tracking progress of datapoints, and matching data elements and stages across related studies, the system may generate this industry standard by calculating progress statistics on previous studies with similar stages and data elements, which may be used to score sites or an entire study relative to a standard performance.
Besides the operations shown in
In another embodiment, the system may use these progress statistics to generate metrics that track enrollment and screening failures. For example, using system-generated characterized study design elements, subjects may be characterized using input such as data entered in forms of given type (e.g., demographics, vitals, inclusion/exclusion in screening); data entered in all forms of a stage (e.g., screening) for patients to be considered promoted to subsequent stages; and lengths of time between time points (e.g., data element creation, first data input, last data input) for duration of a stage. The system may evaluate such measurements to generate metrics to provide certain insights and then modify the study.
For example, the data may show how consistent a site is with respect to data entry across subjects. Some sites may be entering all screening data for all patients on time (e.g., within 1 day of visit), while other sites may be adding a lag of one week or more. In another example, the data may show that the rate of subjects entering screening is substantially lower in one site compared to another, or the rate of subjects entering screening compared to subjects entering randomization is lower in one site than another. A corrective action could include revisiting targets for these two sites and increasing the target for the site that has been steadily recruiting patients.
Additionally, based on statistical models of subject enrollment, the system may be used to estimate time-to-target enrollment of sites and model changes in target enrollment across sites in the study. Typically, enrollment and recruitment model parameters rely on or can be derived from data similar to the metrics previously described (e.g., rate of new subjects, fraction of subjects completing stage, length of stage per subject). Using such data from prior studies can be used to derive parameters for existing models to provide estimates for new future recruitments at each site. Given these estimates, one may compute at which point in the future the target enrollment will be reached.
Referring back to
Metrics module 670 calculates metrics based on the progress curves. As described above, metrics module 670 may describe the progression of a study by custom study stage and may track enrollment and screening failures. This module may provide recommendations and/or status of the clinical study, and may provide alerts if a clinical study is not proceeding according to plan.
With defined custom study stages, progress information can be determined and unique characterizations can be derived in the form of study states based on one or more progress status models. Such states may be in the form of metrics or predictors. One set of metrics may be similar to progress curves generated in the previous application, U.S. patent application Ser. No. 14/492,597, and may be used to view and quantify cumulative properties in the creation and processing of data. Specifically, the system may extract relevant data from a custom study stage, normalize associated clinical study stage data across time and states, compute subject/site/study metrics, and present data to the user. This process may be used to track and measure the progress of data entered/modified/verified in real time or historically using classified study data elements and to perform a comparative analysis of data entered or processed in the custom study stage.
With respect to prediction/optimization model 680, in one embodiment the system may build predictors that estimate the completion of stages within a study and the study overall based on the duration of study stages and data element processes. These predictors may estimate the time to complete a stage in a study, the time to complete a status change (e.g., verification, locking), or the expected time to complete the overall study, even before enrollment is completed. More specifically, the system may generate a model in block 645 to which is inputted the target number of subjects for the stage (or the study), the sites used, and the type of study (e.g., which therapeutic area and phase) and which can output the estimated time (e.g., number of weeks) to completion. Generating such a model involves providing data from past studies for the specific stage, for example, and the measured completion to derive a model (e.g., linear regression). Thus the system is able to provide precise and normalized data to better derive such models.
Visualization module 690 may present model states and progress curves on a user's computer, allowing the user 695 to manipulate how the data may appear. User 695 may be able to view the model states on a laptop or desktop computer or on a smartphone or tablet. Some examples of visualization are shown in
In sum, a system has been developed that is able to track the progress of a clinical study by defining custom study stages and tracking those stages. In defining the stages, the system may use pre-annotated clinical study data elements or classify clinical study data elements based on annotations from a number of clinical studies. The system may directly use such annotation to delineate a custom study stage, or it may be used in combination with the described study element classification methods to provide finer granularity of data elements within such a custom study stage, enhanced by classification tools discussed above.
Aspects of the present invention may be embodied in the form of a system, a computer program product, or a method. Similarly, aspects of the present invention may be embodied as hardware, software, or a combination of both. Aspects of the present invention may be embodied as a computer program product saved on one or more computer-readable media in the form of computer-readable program code embodied thereon.
For example, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, an electronic, optical, magnetic, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. A computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Computer programs that may be associated with applications of the system for monitoring the progress of a clinical study (called “computer control logic”) may be stored in the main memory or in secondary memory. Such computer programs may also be received via a communications interface. Such computer programs, when executed, may enable the computer system to perform the features as discussed herein. In particular, the computer programs, when executed, may enable the processor to perform the described techniques. Accordingly, such computer programs may represent controllers of the computer system.
Computer program code in embodiments of the present invention may be written in any suitable programming language. The program code may execute on a single computer, or on a plurality of computers. The computer may include a processing unit in communication with a computer-usable medium, wherein the computer-usable medium contains a set of instructions, and wherein the processing unit is designed to carry out the set of instructions.
In one embodiment, the computer-based methods may be accessed or implemented over the World Wide Web by providing access via a Web Page to the methods described herein. Accordingly, the Web Page may be identified by a URL. The URL may denote both a server and a particular file or page on the server. In this embodiment, it is envisioned that a client computer system may interact with a browser to select a particular URL, which in turn may cause the browser to send a request for that URL or page to the server identified in the URL. Typically, the server may respond to the request by retrieving the requested page and transmitting the data for that page back to the requesting client computer system (the client/server interaction may be typically performed in accordance with HTTP). The selected page may then be displayed to the user on the client's display screen. The client may then cause the server containing a computer program to launch an application, for example, to perform an analysis according to the described techniques. In another implementation, the server may download an application to be run on the client to perform an analysis according to the described techniques.
The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
