Patent Application
20160171181
Clinical trials (sometimes called “clinical studies”) are often used to assess the safety and efficacy of a drug or a medical device. In some trials, hundreds or thousands of test sites enroll thousands or tens of thousands of patients or subjects.
It is very expensive to monitor all of the test sites to ensure compliance with a clinical trial protocol. In the past, site monitors would visit each site on a frequent basis to manually review all of the subject records to ensure compliance. More recently, centralized site monitoring has emerged in which site monitors remotely examine different metrics related to various aspects of site quality and performance to look for sites that are statistical outliers and thus in need of closer inspection. Examples of a method and apparatus for remote site monitoring are disclosed in U.S. Pat. No. 8,706,537, assigned to the same Applicant and assignee, Medidata Solutions, Inc., and is hereby incorporated by reference in its entirety.
One metric that may be monitored during a clinical trial is the occurrence of subject events, which comprise anything that may happen to a clinical trial subject that is not specifically prescribed by the clinical trial protocol and that would be of clinical interest to report. One type of subject event is an adverse event, sometimes abbreviated “AE.” An AE typically includes any event that is observed to occur to a subject during his/her participation in the trial that may have a negative impact on health or well-being, and may include headache, stomachache, dry mouth, high blood pressure, fast heart rate, migraines, seizures, stroke, heart attack, etc. Other types of subject events may include concomitant medications—the use of one or more medications other than the medication under investigation—while a subject participates in a clinical trial; and disease-specific events, such as the number of times a clinical trial subject wakes up in the middle of the night due to a specific disease or condition.
One way of measuring site compliance with subject event (“SE”) reporting is by examining the subject event rate, which has been calculated as (Total Count of Subject Events)/(Total Count of Subjects), and comparing that to a clinical trial benchmark. But that subject event rate calculation is too simplistic and in many situations does not do a good job of identifying clinical sites that may have subject event reporting problems.
Where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements. Moreover, some of the blocks depicted in the drawings may be combined into a single function.
In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the invention. However, it will be understood by those of ordinary skill in the art that the embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present invention.
Embodiments of the present invention may be used with respect to clinical trials, but the invention is not limited to such embodiments. In monitoring the subject event rates or adverse event rates for a clinical trial, one goal is to detect sites that are under-reporting or over-reporting subject events relative to some nominal expectation or benchmark (e.g., overall trial rate). Three criteria that may be used to evaluate how well any given method is at achieving this goal include:
Although this specification covers subject event reporting, in several places, for simplicity and because those of skill in the art are more familiar with the term “AE rate,” the term “adverse event” or “AE” may be used in certain contexts rather than “subject event,” knowing that the term “subject event” can easily be substituted.
As mentioned above, subject event rate may be a simple ratio:
This simple ratio, however, does not take into account relative subject participation time at each site. In other words, at any given time during a trial, the amount of time each subject has participated in the trial will vary significantly. This variation is driven primarily by two factors. First, subjects begin participation in the trial at different times. The span of time over which all subjects begin trial participation is generally referred to as the “trial enrollment window,” which may last for many months. Second, some subjects will exit the trial prior to full/successful completion, limiting their total participation time by varying amounts depending on when they exited.
Subjects who have been participating in a given trial longer than other subjects—and who have been exposed to the investigational product (e.g., drug or medical device being investigated) and trial procedures longer—presumably also have had greater opportunity for SEs to occur. Thus if one site in the trial enrolled its subjects earlier than most other sites in the trial, one would expect the rate of SEs reported per subject to be significantly higher for this earlier-enrolling site than for other sites across the trial, and an assessment based on this rate would unfairly flag this earlier-enrolling site as exhibiting an elevated risk for over-reporting of SEs. Equation (1) for SE Rate therefore may not be a reliable or consistent measure.
Equation (1) may be improved by re-defining the denominator in the SE rate to account for total subject participation time. For example:
The units of this rate may be expressed as SEs per unit time, e.g., per subject day or subject year. This definition is a form of normalization that provides more reliability and consistency of SE rate assessment across sites in a trial. However, the expected rate of SEs per subject may not be uniform over the period of each subject's participation. For example, the number of SEs reported for a given subject during months 0 through 6 of the subject's participation may differ from those reported during months 7 through 12. This variation in SE rates may depend on factors such as investigational product exposure, frequency of subject visits, and frequency/intensity of trial procedures, to name a few.
The inventors have analyzed the progression of AE rates per subject over participation time on a number of clinical trials stored in a database. The graph in
As a result of this variability in expected SE rates over time, the SE rate for a site with more recently enrolled subjects will likely be very different from the rate for a site with longer-participating subjects, and may be unfairly flagged for under- or over-reporting if the average participation time of that site's subjects differs from most of the other sites in the trial.
The inventors then realized that they could model different discrete phases of subject participation time, one example of which is the subject visits defined in a protocol's expected visit calendar (i.e., protocol schedule of events) for a particular trial. They analyzed the progression of SE rates over subject participation time with the addition of subject visit dates. It was observed that in this context, SE rates consistently trended lower between subject visits, and trended higher when closer in time to each subject visit. This was a consistent trend across many trials, and one possible interpretation of this result is that subjects are relatively less likely to report their SEs to sites when not in regular contact with the sites, but will report the SEs they remember when visiting the site and being explicitly solicited by site staff to report their SEs. The subject's recent memory is often better, which results in a higher level of reporting of SEs that have occurred closer in time to the visit (e.g., most recent week or two). Another factor may be the impact of intrusive procedures and/or investigational product administration typically performed at each subject visit, leading to an increase in reported SEs during or immediately following each subject visit.
Based on this analysis, the rate of SE reporting is more closely related to the volume of subject visits, and not simply the total subject participation time. Thus a further improvement can be made to the definition of SE Rate in Equation (2) as follows:
This form of normalization uses a more relevant measure of subject participation cadence with respect to SE reporting (vs. simple calendar time), which yields a more reliable assessment of true SE (or AE) rate. The graph in
Reference is now made to
The data may be associated with a clinical trial for a drug or medical device. Subject event and visit count processor 10 may calculate total site subject event count 14 and total site visit count 18 for each site based on data received and reported from that site, including reported subject events 4 and reported subject visits 7. Then, using site subject event rate assessor (or processor) 30, trial-level subject event rates are computed and then used to derive an expected total site subject event count for each site, which is then used to determine whether the particular site's total site subject event count 14 is in line with the expected total site subject event count or whether there is a reporting problem 95.
Calculating a trial-level subject event rate may be performed by dividing the sum of the total site subject event counts for all sites by the sum of the total site visit counts for all sites. Calculating an expected total site subject event count may be performed by multiplying total site visit count 18 by the trial-level subject event rate.
In certain scenarios, such as when the system expects a certain amount of data by a specific visit but has not received such data, there may be a weighted visit count for each reported visit, which represents an estimated proportion of all required visit data that have already been reported for the subject for the given reported visit. For example, each patient may have six data forms for six different assessments for a visit in week 4. This sets up an expectation that there will be six forms for each patient. If the system receives only three forms, then the data actually received would be weighted as 0.5 or 50%.
Going beyond Equation (3), the inventors made a further observation that there is a distinct SE rate “footprint” that can be associated with each subject visit in any given trial. In particular, the overall rate of SEs observed for Visit 1 across all sites and subjects is unique and different from the overall rate of SEs observed for Visit 2, Visit 3, etc. The primary reason for this may have to do with the timing/cadence of investigational product administration—variable by visit—and the impact that has on subject well-being. Another factor may be the relative intrusiveness of other procedures administered to the subject that also vary across visits.
Thus SE Rate may be most effectively measured and assessed on a distinct subject visit basis. The specific approach taken by the inventors involves the following steps:
Reference is now made to
Reference is now made to
Site Visit N Count: The Site Visit N Count is a weighted sum of subjects at a given site for whom a given distinct visit (e.g., Visit N) has been reported.
For each subject at the site:
Equation (5) calculates Site Visit N Count (operation 340) using Effective Visit N Date and Final Effective Visit Date:
For each distinct visit (e.g., Visit N) in the expected visit calendar:
The purpose of assessing in Equation (5) whether greater than 70% of expected eCRF visit forms for Visit N have been submitted is to determine through estimation whether the site has already reported any observed SEs associated with Visit N. If the site has not yet reported a clear majority of the expected Visit N data, we assume that entry is still in progress and therefore expect less than a full representation of SEs for the visit. In this case Visit N for the given subject will not be counted as an entire completed visit, and will instead be evaluated to determine whether it warrants partial or “weighted” counting. Thus, a weighted visit N count is determined for each subject (operation 335). Note that for the purpose of this invention, thresholds other than 70% of the expected visit data may be considered, as well as other methods to estimate the completion of SE reporting for a given subject visit.
Subject visits that cannot be counted fully (e.g., because less than 70% of expected visit data has been reported) may be evaluated to determine whether they warrant partial counting. In particular, if the Effective Visit N Date is in the past, it is counted as one-half (0.5) of a visit (the constant 0.5 is near the end of Equation (5)). If the Effective Visit N Date is in the future and is the next visit scheduled to occur for the subject, it is counted less than one-half (0.5), based on the elapsed duration from the subject's previous visit until TODAY (the day the calculation is being performed) as a proportion of the total duration from the subject's previous visit until the next scheduled visit. The rationale for this weighting (as represented in Equation (5)) is based on an estimate that the volume of SEs reported between visits is typically half of the total observed volume once the next visit occurs and all SEs have been reported. Note that a value different from one-half (0.5) may be used based on further industry analysis of the timing of SE reporting between visits. This constant may also be actively computed or derived during a trial by measuring the percentage of SEs that are reported on or following subject visit dates as compared to being reported prior to the subject visit date (e.g., using an audit timestamp for entry of SE records by the site).
It is also possible to set the above constant higher than the average estimated rate of SE reporting between visits, based on the knowledge that subject under-reporting of SEs between visits reflects an undesired site behavior, i.e., sites should be doing everything possible to get their subjects to actively report SEs that are occurring in between their scheduled on-site visits. Setting this constant higher sets a higher “expectation” of inter-visit SE reporting and may tend to more quickly expose those sites that are not complying with that expectation.
Other embodiments are possible that do not include an expected visit calendar. In some instances, we may know visit dates for visit 1 and visit 3 may be known, but not the date of visit 2. With an expected visit calendar, visit 2 could be added in and SEs could be slotted to it; without an expected visit calendar, visit 2 may not be modeled at all—all the SEs after visit 1 may instead be associated with visit 3.
Site Visit N SE Count and SE Visit Slotting: The Site Visit N SE Count is the count of SEs across all subjects at the site that have been associated with—or “slotted” to—Visit N. The method of slotting SEs to subject visits is detailed as follows.
To calculate Site Visit N SE Count in operation 302, the system assigns (or slots) SEs across all subjects at a given site to a subject visit (Visit N) in the protocol visit schedule. This SE Visit Slotting operation is depicted in operation 355 in
If the SE Onset Date is not valid for a given SE, then the SE is assigned to the subject visit containing the Final Effective Visit Date. If the SE Onset Date is less than or equal to the earliest Effective Visit N Date for the subject, the SE will be assigned to the subject visit with the earliest Effective Visit N Date (e.g., Visit 1). If the SE Onset Date is greater than the Final Effective Visit Date for the subject, the SE will be assigned to the subject visit containing the Final Effective Visit Date. If the SE Onset Date falls between two contiguous Effective Visit N Dates for the subject, i.e., Effective Visit K Date<SE Onset Date≦Effective Visit L Date, where K<L and L≦the subject visit containing the Final Effective Visit Date for the subject, then the SE will be assigned to Visit L. This method can be termed “forward slotting,” since SEs are assigned forward to the next visit chronologically following the SE Onset Date.
In operation 360, the system calculates Site Visit N SE Count:
Site Visit N SE Count=Σ all SEs that have been slotted to Visit N for subjects at the site (6)
In operation 365, the system calculates Total Site SE Count, which is subsequently assessed against an Expected Total Site SE Count:
Total Site SE Count=Σ Site Visit N SE Count across all visits in the trial's expected visit calendar (7)
Effective Visit N Dates, Final Effective Visit Dates, and reported SEs 5 are used in box 430 to calculate Site Visit N SE Counts (e.g., Site Visit 1 SE Count, Site Visit 2 SE Count, etc., . . . Site Visit N SE Count). All of the Site Visit N SE Counts are then summed in box 440 to generate Total Site SE Count 15.
Once Site Visit N Count 17 (operation 340) and Site Visit N SE Count 13 have been calculated as shown in
Trial Visit N SE Count=Σ Site Visit N SE Count across all sites in the trial (8)
And in operation 552:
Trial Visit N Count=Σ Site Visit N Count across all sites in the trial (9)
Then, in operation 560:
For the more general situation shown in
Trial SE Count=Σ Site SE Count across all sites in the trial (11)
And in operation 502:
Trial Visit Count=Σ Site Visit Count across all sites in the trial (12)
Then, in operation 510:
Expected Site Visit N SE Count=(Trial Visit N SE Rate)*(Site Visit N Count). (14)
Then, in operation 610, the total expected count of SEs for the site is calculated:
Expected Total Site SE Count=Σ Expected Site Visit N SE Count across all visits in the trial's expected visit calendar. (15)
Assessing the likelihood that a site is under- or over-reporting subject events may be accomplished by calculating the probability that the subject event count for the specific clinical trial site may naturally deviate from the expected subject event count by the amount observed. A Poisson distribution model may help assess each site's actual SE count against the expected SE count for the site. According to the Poisson model, the standard deviation of a given expected count lambda (λ) is the square root of lambda (√{square root over (λ)}). Thus in operation 615, when lambda (λ) is represented by Expected Total Site SE Count,
SE Count Std Dev=√{square root over (Expected Total Site SE Count)} (16)
In operation 620, a z-score (or z-value) representing the number of standard deviations by which the site's SE count differs from the expected count may be calculated:
Assuming the Poisson model, the z-score may be approximated as coming from the normal distribution and the following formula may be used in calculating a directional p-score (operation 625):
SE Count p-Score=100*P(Z<z)=100*Φ(z) (18)
where φ(z) is the cumulative distribution function of a Normal distribution. For easier implementation (e.g., less computational power), the approximation of this function using elementary functions translates into the following formula:
where the 1.6 and 0.07 have been derived by the inventors. This score has a range of values from 0 to 100, where a value of 50 indicates that the site's total SE count matches the expected total SE count exactly (i.e., z=0); values higher than 50 indicate the site's SE count is higher than expected; and values lower than 50 indicate the site's SE count is lower than expected. This is why the p-score is called “directional.” Thus, in operation 630, a comparison is made between the SE Count p-score, and the closer that value gets to 0 or 100, the more unexpected the site's SE count becomes, and the more risky the site's SE reporting is and should be investigated.
Besides the operations shown in
Similarly, the parts and blocks shown in
One benefit of the present invention is that adverse event reporting is based on more relevant and accurate measures of what is happening at sites, based on the age of each site (that is, how long it has been operating in the current trial), and length of time the subjects at each site have been participating in the trial. A key further refinement of the present invention is the recognition that subject participation time itself is best represented by the individual and distinct trial visits that have been reported for each subject, since expected AE reporting rates vary by subject visit. These factors provide a much more reliable way of comparing each site against other sites in the same trial which have been operating for different lengths of time.
The present invention differs from other systems that track AE rate or AE reporting compliance. For example, those systems do not take into account the variability in site age and cadence of subject visits, looking instead at overall rate of AEs per site or per subject.
Aspects of the present invention may be embodied in the form of a system, a computer program product, or a method. Similarly, aspects of the present invention may be embodied as hardware, software or a combination of both. Aspects of the present invention may be embodied as a computer program product saved on one or more computer-readable media in the form of computer-readable program code embodied thereon.
For example, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, an electronic, optical, magnetic, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.
A computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Computer program code in embodiments of the present invention may be written in any suitable programming language. The program code may execute on a single computer, or on a plurality of computers. The computer may include a processing unit in communication with a computer-usable medium, wherein the computer-usable medium contains a set of instructions, and wherein the processing unit is designed to carry out the set of instructions.
The above discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.
