Patent Grant
10585778
The embodiments described herein pertain in general to conducting controlled, randomized experiments and more particularly to implementing a reset policy during a variation test of content.
Content variation testing is an experiment that tests changes to content against another version of the content. Content variation testing is applied to many different internet services such as electronic-commerce sites to social networks to market places. One example of content variation testing is web page variation testing where an experiment is conducted that tests changes to a web page against another design of the web page. During variation testing of a web page, changes to a web page that produce positive results are determined. The positive results associated with a particular change to a web page validates that the change to the web page should be used.
To obtain valid results in a conventional variation test such as a fixed-horizon variation test, a strict set of guidelines are followed when performing the variation test. The guidelines include setting a minimum detectable effect and sample size in advance to conducting the test, refraining from viewing the results of the variation test prior to the completion of the variation test, and refraining from testing too many goals and variations in the variation test.
However, an experimenter can now continuously monitor the results of a variation test due to the advancement of experimental technology. This poses a problem as conventional variation testing requires that the experimenter refrain from viewing the results until the completion of the variation test. Thus, the experimenter may view invalid results if the experimenter views the results of the variation test before the completion of the variation test and may make decisions based on those invalid results.
The embodiments herein describe a variation testing system for performing variation testing of content. The content undergoing the variation test is associated with multiple variations of the content. Client devices of users that request the content are provided one of the variations of the content.
In one embodiment, the variation testing system allows an experimenter to continuously monitor the results of the variation test. Thus, the experimenter can request the current results of the variation test and the variation testing system provides valid results to the experimenter prior to completion of the variation testing. To ensure that the experimenter is provided with valid results at all times, the variation testing system implements a reset policy that resets the variation test if the difference in observed conversion rates between the variations of the content are not within a confidence interval. The confidence interval is a range of differences in conversion rates that has a high probability of containing the actual difference of conversion rates between the different variations of the content. If the observed difference in conversion rate is outside of the confidence interval, the results of the variation test may be invalid thereby warranting a reset of the variation test.
Features and advantages described in this summary and the following detailed description are not all-inclusive. Many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims hereof.
The figures depict an embodiment for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.
Network 109 enables communication among the entities connected to it. In one embodiment, network 109 is the Internet and uses standard communications technologies and/or protocols. Thus, network 109 can include links using technologies such as Ethernet, 802.11 (WiFi), worldwide interoperability for microwave access (WiMAX), 3G, Long Term Evolution (LTE), digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, etc. Similarly, the networking protocols used on network 109 can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), etc. The data exchanged over the network 109 can be represented using technologies and/or formats including the hypertext markup language (HTML), the extensible markup language (XML), etc. In addition, all or some of links can be encrypted using conventional encryption technologies such as the secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc. In one embodiment, network 109 may include but is not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN). In another embodiment, the entities use custom and/or dedicated data communications technologies instead of, or in addition to, the ones described above.
Client devices 102 communicate with the third-party content provider 107 via the network 109 to receive content items such as web pages from the third-party content provider 107. In one embodiment, each client device 102 is a computer system capable of communicating with the variation determination system 105 and the third-party content provider 107. Examples of client devices 102 include a personal computer, a mobile phone, a tablet, a personal digital assistant (PDA), or a laptop. As shown in
As mentioned above, the environment 100 includes a third-party content provider 107. Hereinafter, the third-party content provider 107 is referred to as a “content provider 107” for ease of description. The content provider 107 may be an individual, a group of individuals, or an entity such as a company. In one embodiment, the content provider 107 provides content items to client devices 102 such as web pages. While only one content provider 107 is shown, it is understood that any number of content providers are supported and can be in the environment 100 at any time.
In one embodiment, the content items provided by the content provider 107 include web pages. However, the content provider 107 may provide other types of content items such as video, audio, a combination of video and audio, still images (e.g., JPEG), text documents, advertisements, and/or any other types of content. For purposes of convenience and the description of one embodiment, the content items provided by the content provider 107 will be referred to as a web pages, but no limitation on the type of content items are intended by this terminology.
In one embodiment, the content provider 107 operates in conjunction with the variation testing system 105 to perform variation tests on web pages. The content provider 107 may display different variations of a web page to client devices 102 based on instructions from the variation testing system 105 that is conducting a variation test on the web page for the content provider 107. A variation test for a web page tests changes to the web page against the current variation of the web page to determine whether the changes produce a desired result. Examples of a desired result do to a change to a web page is an increased selection of an advertisement(s) included in the web page or increased purchases of a product advertised on a web page. Thus, variation testing can validates a new design of a web page or modifications to elements on the web page before the new design or modifications are put into production by the content provider 107.
For a given web page, the content provider 107 may have one or more variations of the web page that are used in a variation test for the web page. In one embodiment, a variation test of a web page involves an experiment that tests an “A” variation known as the control and a “B” variation known as the variant on users requesting the web page from the content provider 107. For ease of discussion, the embodiments discussed herein describe a web page having only two variations: the control and the variant. However, in other embodiments, a web page can have any number of variants.
Referring back to
In one embodiment, the web server 111 links the content provider 107 to one or more client devices 102 and the variation testing system 105 via the network 109. The web server 111 serves content items such as web pages, as well as other web related content, such as Java, Flash, XML, and so forth. The web server 111 may include a mail server or other messaging functionality for receiving and routing messages between the variation determination system 105 and one or more client devices 102.
The variation testing system 105 conducts variation tests on web pages provided by the content provider 107. As shown in
The policy database 125 stores policies for variation tests on web pages being tested by the variation testing system 105. Each policy describes rules associated with the variation test that specify the distribution of the variations of the web page to users such as a percentage of users that should receive a control version of the web page or the variant of the web page. For example, a policy for a web page may specify that half of the users that request the web page receive the control version of the web page and half of the users that request the web page receive the variant of the web page. However, other distributions may be used in other embodiments.
The variation determination module 119 selects a variation of a web page from a plurality of variations of the web page to provide to a client device 102. In one embodiment, the variation determination module 119 receives a request from the content provider 107 to provide instructions indicating which variation of a web page should be provided to a client device 102 that submitted a request for the web page. For example, the variation determination system 105 determines whether to provide the control variation of a web page or the variant variation of the web page to the client 102 responsive to the request. The request from the content provider 107 is an indication to the variation determination system 105 that a web page was requested by a client device 102.
In one embodiment, the variation determination module 119 selects a variation of a web page based on the policy for the variation test of the web page. Responsive to a request from the content provider 107 to provide instructions indicating which variation of a web page should be provided to a client device 102, the variation determination module 119 accesses the policy associated with the web page from the policy database 124. The variation determination module 119 determines from the policy the distribution in which variations of the web page should be provided to users. The variation determination module 119 then assigns the user to a particular variation of the web page (e.g., the control or the variation) according to the determined distribution. For example, if the policy describes a 50/50 distribution of the control version and the variant of the web page, the variation determination module 119 alternates between instructing the content provider 107 to provide the control version or the variation of the web page. Alternatively, the variation determination module 119 randomly assigns users to one of the plurality of variations of the web page.
Once the variation determination module 119 determines which variation of the web page should be provided to the client device 102, the variation determination module 119 transmits a notification to the content provider 107. The notification includes an instruction for the content provider 107 to provide a particular variation of the web page to the client device 102 that requested the web page. The content provider 107 then provides to the client device 102 a baseline web page and instructions (e.g., JavaScript) that describes the different variations to apply to the baseline web page based on the notification sent by the variation testing system 105. The client device 102 executes the instructions associated with the selected variation causing the client device 102 to modify the baseline web page in order to display the variation of the of the web page.
In one embodiment, the interaction module 123 receives indications from the content provider 107 of user interactions with the variations of the web page provided to client devices 102 of users. Each indication specifies whether a conversion occurred on the web page such as the user interacting with second advertisement of web page shown in
In one embodiment, the statistics module 121 conducts variation tests on web pages. Particularly, the statistics module 121 conducts a sequential test on a web page. However, the sequential test conducted by the statistics module 121 allows an experimenter to continuously monitor the results of the sequential test. That is, an experimenter is allowed to view the results of a sequential test being performed on the web page prior to completion of the sequential test and the statistics module 121 provides results of the variation test to the experimenter that are valid when the experimenter requests to view the results.
Note that usage of variation test and sequential test may be used interchangeably herein. Furthermore, in one embodiment an experimenter is a person(s) that represent the content provider 107 that provides the web page undergoing the sequential test such as an employee of the content provider 107. An experimenter may also be a person(s) that represent the variation testing system 105 conducting the sequential test on the web page on behalf of the convent provider 107 such as an employee of the variation testing system 105.
Generally, an experimenter runs the variation test on the web page to determine which variation provides an improvement over the control version. That is, the experimenter wants to determine the “winners” or “losers” of the variation test. A “winner” is considered a variation of the web page that shows improvement in conversion rate over the control version of the web page. Conversely, a “loser” is considered a variation of the web page that did not show improvement in conversion rate over the control version of the web page. In one embodiment, the statistics module 121 declares the results of the variation test conclusive if the difference in the conversion rates {circumflex over (θ)}n between the variations of the web page is significantly different from zero (either positive or negative) as will be further described below.
After every user receives a variation of the web page, the conclusion module 401 determines the magnitude by first determining the conversion rate for each variation up to the most recent user that received the web page. In this example, assume that the distribution is evenly split so half the users n/2 receive the control version of the web page and half of the users n/2 receive the variation of the web page where n is the index of the number of users that visited the web page undergoing the variation test. The conclusion module 401 calculates the conversion rate {circumflex over (X)}n for the control version of the web page and the conversion rate Ŷn for the variation of the web page.
The conclusion module 401 determines the conversion rate for each variation of the web page by calculating the quotient of the number of conversions that occurred on the variation of the web page and the total number of users that visited all variations of the web page. The conclusion module 401 then calculates the difference in the conversion rates {circumflex over (θ)}n where {circumflex over (θ)}n=Ŷn−{circumflex over (X)}n. The difference in the conversion rates {circumflex over (θ)}n signifies how the variation of the web page is performing with respect to the control version of the web page in terms of conversions. Thus, a higher difference in conversion rate is indicative of increased interaction of the elements of the modified elements of the control version of the web page that are included in the variation of the web page. As mentioned above, the conclusion module 401 may calculate the difference in the conversion rates {circumflex over (θ)}n value responsive to each user visiting the web page undergoing the variation test.
In one embodiment, the conclusion module 401 determines that the results of the variation test are conclusive based on condition (1) shown below:
In condition (1), α is a predetermined statistical significance, τ is a constant with a positive value, and Vn is an estimate of the variance of the observed difference of conversion rates. In one embodiment, the value of α is received from the experimenter. One example value for α is 0.1 which controls the rate of false positives to be 10%. In one embodiment, the value for τ is set by an employee of the variation testing system 105 or the experimenter. The value of τ is selected to reflect improvements that are likely across all content providers 107 based on historical data of variation experiments conducted by the variation testing system 105. Vn is the sum of the estimates of the variances of {circumflex over (X)}n and Ŷn and is calculated according to the following equation.
After every user n that visits the web page, the conclusion module 401 determines the difference in the conversion rates {circumflex over (θ)}n and determines whether condition (1) is satisfied. If the conclusion module 401 determines that the absolute value of the difference in the conversion rates {circumflex over (θ)}n is greater than
the conclusion module 401 declares the end of the variation test. When the variation test is finished, the conclusion module 401 can declare either the control version or the variation of the web page as the “winner” of the variation test based on the observed improvement of the winner. Otherwise, the conclusion module 401 allows the variation test to continue if condition (1) is not satisfied.
Referring back to
In one embodiment, the confidence interval represents a range of difference in the conversion rates {circumflex over (θ)}n that is likely to contain the actual improvement in the conversion rate of the variation of the web page compared to the conversion rate of the control version of the web page. For example, a 90% confidence interval on the difference in conversion rates has at least a 90% chance of containing the actual difference of average conversion rates between the variation and the control version of the web page. The difference in conversion rates {circumflex over (θ)}n approaches the actual difference of average conversion rates as more users visit the web page. Thus, a difference in conversion rate {circumflex over (θ)}n calculated from 10,000 visitors will likely be closer to the true difference than one calculated from 100 visitors and will eventually be arbitrarily close to the true average as long as the underlying process causing conversion does not change.
In one embodiment, the reset module 403 calculates the confidence interval by determining a desired level of confidence (1−α) for the variation test. In one example, the desired level of confidence (1−α) for the variation test may be received from the experimenter. The reset module 403 sets a range of conversion rate differences (e.g., the confidence interval) equal to
to calculate the confidence interval. The results of the calculation include the bounds of the confidence interval that are associated with the desired level of confidence (1−α). For example, the resulting calculation may result in a 3% to 7% conversation rate difference. In one embodiment, the reset module 403 identifies the intersection of the confidence interval and the confidence interval at (n−1) visitors to determine the confidence interval for n visitors.
In one embodiment, after receiving an indication of each user that visits the web page, the reset module 403 compares the difference in conversion rate {circumflex over (θ)}n calculated for all users up to the most recent user to the confidence interval. The reset module 403 allows the variation test to continue if the comparison indicates that the difference in conversion rate {circumflex over (θ)}n is within the confidence interval for the variation test. Thus, the reset module 403 takes no further action. However, if the reset module 403 determines that the difference in conversion rate {circumflex over (θ)}n is not within the confidence interval, the reset module 301 resets the variation test for the web page.
In one embodiment, resetting the variation test for the web page restarts the beginning of the variation test to the time when the reset occurred. By resetting the variation test, the reset module 403 disregards all of the accumulated conversion rate difference values {circumflex over (θ)}n for the users that visited the web page up to when the reset is implemented. In one embodiment, after a threshold number of users visit the web page (e.g., 100), the reset module 403 calculates a new confidence interval. Thus, after the variation test is reset, the reset module 403 compares the difference in conversion rate value {circumflex over (θ)}n for any subsequent users that visit the web page to the new confidence interval to determine whether another reset is warranted.
Furthermore, the reset module 403 also re-calculates the magnitude of the difference in the conversion rates {circumflex over (θ)}n between the variations of the web page required to deem the variation test conclusive according to condition (1). In one embodiment, after receiving an indication of each user that visits the web page, the reset module 403 also determines whether condition (1) is satisfied. If condition (1) is satisfied, a winner of the variation test is declared as described above. Further details of the reset module 403 is described below.
Providing Variations of Web Pages
In one embodiment, the variation testing system 105 receives 501 indications from a content provider 107 that a web page is provided to a plurality of client devices 102. The web page provided to each of the client devices 102 is one of a plurality of variations of the web page. Responsive to each indication, the variation testing system 105 determines 503 an observed difference in conversion rates {circumflex over (θ)}n between the variations of the web page. The variation testing system 105 compares 505 the observed difference in conversion rates {circumflex over (θ)}n to a confidence interval. The confidence interval represents a range of {circumflex over (θ)}n values indicative that there is an actual improvement in the conversion rate of a variation of the web page compared to the conversation rate of another variation of the web page. The variation testing system 105 implements 507 a reset policy based on the comparison. In one embodiment, if the observed difference in conversion rates {circumflex over (θ)}n is outside of the confidence interval, the variation testing system 105 implements the rest policy.
Further Details of the Statistics Module 121
A Type I error, or false positive, occurs when a statistical decision procedure rejects the null hypothesis, even though the null hypothesis was true, i.e., where there is no difference between the treatment and control groups. The Type I error rate is the frequency with which such errors are made on true null hypotheses; in other words, how often do we declare a result significant on an A/A test. For example, if a procedure has a 10% Type 1 error rate, then about 1 detection in 10 A/A tests is expected, 2 in 20, 5 in 50, and so on. Type I error rate is a key statistical measurement, and indeed it is the underpinning of any valid statistical decision procedure.
Power is the probability that an A/B test with a true difference between treatment and control will be declared significant, and called in the right direction. Since decision procedures based on data all have underlying uncertainty, requiring zero chance of a Type I error, also means zero power because no tests can ever be called significant. Conversely, having 100% power also means 100% Type I error, since all tests must lead to significant detections. In general, the experimenter trades off one against the other: increasing power brings with it increasing Type I error rate, and “good” statistical procedures are those that yield relatively high power while maintaining a reasonably low Type I error rate. For example, most A/B testing platforms suggest controlling Type I error rate at around 5-10%, and choosing a sample size to ensure statistical power is at least 80%.
The statistics module 121 is designed to better match the way experimenters use the platform, while still providing both control of statistical error and good power. The statistics module 121 includes two innovations in particular: sequential hypothesis testing, and control of false discovery rate for multiple hypothesis testing.
Continuous Monitoring and Sequential Testing
In traditional statistics used in A/B testing such as fixed-horizon statistics; it presumes that a sample size is fixed in advance. In such contexts, fixed-horizon statistics is optimal in the following sense: given a bound on the proportion of false positives, the probability of detecting any true effect with the predetermined data set is maximized, irrespective of the size of that effect.
For modern A/B testing the logistical constraints fall away, and with the detailed real-time data we have now it should be possible to do much better. One of the greatest benefits of advances in information technology, computational power, and visualization is precisely the fact that experimenters can watch experiments in progress, with greater granularity and insight over time than ever before. A key goal of the statistics module 121 is to take advantage of this opportunity with a novel statistical approach.
How can we take advantage of real-time data? Note that there is a subtlety in the optimality property of fixed-horizon statistics described above: while the power (probability of detection) is maximized for any true effect, the power can still be very low if the effect is small compared with the sample size. To have adequate power for detecting small effects, the experimenter commits to a large sample size up front, and this will be wasted effort if it turns out the true effect could have reliably been detected much sooner.
When a sample size need not be set in advance, the data should let you learn the size of the effects to consider adaptively, and so optimize the sample size at which the test terminates. But fixed-horizon statistics offers no mechanism for mid-experiment optimization, so unfortunately the additional flexibility of A/B testing has so far only been considered detrimental, driving up the number of false positives if you are not savvy.
Generally, continuous monitoring of an experiment increases the Type I error rate when fixed-horizon statistics is used, whereas the statistics engine 121 leverages real-time data correctly through a framework known as sequential testing.
Statistical Decision Procedures, Continuous Monitoring, and Decision Boundaries
For convenience we will just consider a binary goal, although the discussion is essentially the same for real-valued goals. Assume a web page includes only a baseline (the “control”) and a single variation (the “treatment”). In one embodiment, n is used to index the number of visitors that have arrived. Further assume the allocation is evenly split so that n/2 visitors are in the baseline and variation as mentioned above; all the discussion below is equally valid (just with more notation) even if the allocation is unequal. For the rest of the discussion herein,
The statistics module 121 runs A/B tests for content providers 107 to learn which variations are winners (and which are losers). Intuitively, this means results should be declared conclusive if {circumflex over (θ)}n is “significantly” different from zero (either positive or negative). In other words, we can think of a statistical decision procedure as telling us how large the magnitude of {circumflex over (θ)}n needs to be for results to be conclusive. A classical result in statistics (the Sufficiency Principle) tells us that all good decision procedures have this form.
To see this approach in action, consider fixed-horizon statistics. If a sample size of n is established, then fixed-horizon statistics would tell you that the test is conclusive at a given level α (e.g. 5%) if
and k is a fixed constant.
Note that for binary goals, Vn is an estimate of the variance of the observed difference of conversion rates; in particular, it is the sum of the estimate of the variances of
Fixed-horizon statistics ensures that Type I error is controlled under the assumption that the experimenter only looks at the data once. However, if rather than setting the sample size in advance, the experiment is continuously monitored as it is running, there are many potential decision points. When viewing a test through the lens of continuous monitoring, it's useful to visualize the decision boundary of a procedure. For example, if a test is stopped the first time that (1) is satisfied, the decision boundary as a function of n is the right hand side of (1).
Why does Continuous Monitoring Inflate Type I Error?
A key point is that continuous monitoring of fixed-horizon statistics severely inflates Type I errors, losing signal about truly significant results in a sea of false positives. Suppose that the experimenter were to stop the first time that constraint (1) is satisfied. Such a rule is guaranteed to report a false positive; in other words, the test will eventually cross the decision boundary under the null hypothesis of no difference between baseline and variation. To see why, note that as n→∞, the right-hand side of constraint (1) is approximately proportional to 1/√{square root over (n)}. However, it can be shown that under the null hypothesis, the largest fluctuations of {circumflex over (θ)}n are approximately proportional to √{square root over ((log log n)/n)}; this is known as “the law of the iterated logarithm”. Precisely, under the null hypothesis, this law implies that with probability one, there exist arbitrarily large values of n for which
The right hand side is guaranteed to eventually exceed any threshold that is proportional to 1/√{square root over (n)}; in particular, it is guaranteed to exceed the decision boundary on the right hand side of (1), so the Type I error rate of such a rule is 100%!
In practice, the Type I error of such a rule is somewhat lower because the experimenter doesn't wait indefinitely. Even so, with reasonable sample sizes, it is typically the case that Type I error rate will exceed 70-80%. It's clear from the preceding decision that becoming more conservative (i.e., using a higher constant k in constraint (1)) doesn't work, because regardless of the value of k the law of the iterated logarithm says the decision boundary will eventually be crossed.
Towards a Solution
Our goal is to let the experimenter make decisions as soon as possible. However, naive continuous monitoring can lead to spurious decisions—ones based on chance trends in experimental data—overwhelming good ones.
The previous section illustrates the problem: while a decision boundary that scales like 1/√{square root over (n)} works when we look only at a single point in time, it does not prevent the sample path of {circumflex over (θ)}n under the null hypothesis from crossing at any point in time. This is what causes the spike in Type I error with continuous monitoring: eventually, {circumflex over (θ)}n will cross the decision boundary. What is needed is a decision boundary that ensures the probability that {circumflex over (θ)}n ever crosses is controlled at a desired level α.
One way of choosing such a boundary is to choose a very large constant C, and only stop if |{circumflex over (θ)}n|>C. This will indeed lead to a very low Type I error, since the chance of |{circumflex over (θ)}n| crossing a fixed large value under the null hypothesis can be made arbitrarily low. The problem, of course, is that this probability will also be low even if there really is a difference—in other words, the statistical power of such a boundary will be terrible. What the statistics module 121 strives to ensure is detecting an effect if one really exists. Note that if the true conversion rate difference is non-zero, |{circumflex over (θ)}n| will converge to this true value, so a decision boundary that converges to zero as n→∞ is needed. Such a boundary is guaranteed to be crossed when the effect is nonzero; in fact, the faster such a boundary converges to zero, the faster we will detect true effects—at the expense of a potentially inflated Type I error rate.
A problem is identifying a decision boundary that converges to zero quickly (so that true effects are detected fast), but is not crossed too often under the null hypothesis (so that Type I error is controlled)? A clue to the solution is from the law of the iterated logarithm. |{circumflex over (θ)}n| will cross √{square root over ((log log n)/n)} infinitely often, a boundary is needed that goes to zero slower than this. However, a boundary that goes to zero as fast as possible is also needed.
The statistics module 121 uses a boundary proportional to √{square root over (log n/n)}. This boundary goes to zero as n→∞, so that all true effects are detected. But it doesn't do so overly quickly, so that Type I error can be controlled.
Formally, suppose that the experimenter has fixed a desired bound α on Type I error. With the statistics module 121, the test is deemed conclusive once the following condition holds:
First, note that Vn scales approximately proportionally to 1/n for large n. Note that the right hand side is approximately proportional to √{square root over (log n/n)} for large n.
Second, note that there is a constant τ that determines the exact decision boundary. This constant is a free parameter, in the sense that regardless of the value of τ (as long as it is positive), the resulting test gives valid Type I error control while ensuring any true nonzero effect will eventually be detected. However, careful tuning of τ can have a significant effect on the speed of detection of true effects.
The Technical Appendix below details the derivation of constraint (2) further, but the idea for the boundary can be summarized as follows. The key is to consider a likelihood ratio test. In particular, fix a positive θ; the likelihood ratio (LR) given the data so far is the ratio of the probability of seeing the data if the true difference were θ, to the probability of seeing the data if the true difference were zero. When the LR is large, it means that the true effect of θ is more likely than the null hypothesis of zero difference. As a result, we can consider a simple rule: stop if the LR ever crosses a fixed threshold. Prior results in sequential testing have shown that this test (with an appropriate choice of fixed threshold) has the desired properties, if we knew in advance the only possible value of a true effect is θ. This test is known as the Sequential Probability Ratio Test (SPRT).
Unfortunately, in practice, the true effect θ is unknown. Of course, w prior knowledge about what kinds of effects might be “reasonable” to see are available, and so a natural approach in this case is to put a prior distribution on θ. Formally, the statistics module 121 computes the average LR against a prior distribution on the true effect θ, and wait to see if this average LR ever crosses a fixed threshold. This test (rewritten) is exactly the test described in (2).
The prior lets us focus on the effect sizes we most anticipate, and the test is optimized for fast detection on θ where the prior is large, i.e., for those effects we are likely to observe. For the normal prior, this is controlled by the parameter τ. We chose our prior, and hence the parameter τ, as the result of extensive analysis of historical experiments run on the statistics module 121.
Statistical Significance
In classical statistics, the significance level is closely related to the p-value. Informally, the p-value reports the chance of seeing the observed data, if the null hypothesis were true; the significance level is one minus the p-value. In other words, a high significance level suggests it is safe to declare a test as conclusive.
There is a straightforward way to obtain p-values from a decision rule like the one in constraint (2). In particular, we define the p-value after n visitors to be the least α such that the threshold has already been crossed. Under the null hypothesis, the chance of ever crossing the threshold in (2) is α. With this definition, this property guarantees that the chance the p-value ever drops below α is no larger than α.
In practice, regardless of when an experimenter chooses to look at the experiment, the experimenter is guaranteed that the test is stopped when the p-value is below α (i.e., your statistical significance is above 1−α), the Type I error rate will be no more than α. The control on false positives is controlled, while still making it possible to stop tests as quickly as possible when the data justifies it (giving good statistical power). This balancing of dual goals is what makes sequential testing so powerful.
Multiple Hypothesis Testing and Controlling False Discovery Rate
The second key challenge of the statistics module 121 is making it easier for content providers 107 to include multiple goals and variations, and confidently interpret the results. While controlling Type I error is a fundamental aspect of nearly all statistical decision procedures, there is a significant pitfall to be aware of as the number of simultaneous tests increases: controlling Type I error rate does not lead to actionable insight as the number of tests increases.
What are False Discovery Rates?
False discovery rate (FDR) is a different measurement designed to address the issues with reporting Type 1 error while running many concurrent experiments. It is defined as the expected proportion of false detections among all detections made:
where V is the number of tests declared significant which were truly null (i.e., effectively like A/A tests), and R counts the overall number of tests declared significant. The convention is to take V/R=0 if no rejections were made (R=0), as denoted by the V sign for maximum.
Does Controlling the False Discovery Rate Lead to Actionable Insight?
To understand why FDR is useful in a business decision making context, a comparison is made on how the decision maker might react to seeing, say, 5 conclusive tests (winners and losers combined). If all that is known is that the decision procedure controlled Type I error, then the number of other tests run is very important: if 50 other tests were run, these 5 may be highly likely to be conclusive by random chance alone; but if only 5 other tests were run, then the decision maker may become more confident the 5 conclusive results are meaningful. In other words, the number of other inconclusive tests affects whether the decision maker believes her results are actionable.
This consequence is eliminated with procedures that control FDR. Reporting a set of A/B Tests with, for example, “false discovery rate less than 10%” is completely actionable without knowing anything about non-significant variations. The statement means that at least 90% of the reported results have a true difference between variation and baseline.
Why are FDR estimates more actionable with multiple A/B tests? Mathematically, the main observation is that FDR reverses the conditional probability that defines Type I error. In classical statistics, a p-value reports estimates of the probability of the data given there is no difference between baseline and variation. False discovery rates reverse this conditional probability: in particular, FDR is an estimate of the probability of no difference (or a false detection) between the baseline and variation, given the data you observed.
The questions from before are exactly answered by only reporting conclusive results where FDR is less than a threshold, say 10%. Note that 1−FDR is an estimate of the chance there is a true difference between baseline and variation, given the data observed.
In fact, it is possible to further put an ordering on the significance level of significant A/B tests. If we have a group of conclusive results when we control FDR at 10%, and a (smaller) conclusive subset of results when we control FDR at 1%, we can say that the whole group of significant results is 90% likely to give you actual lift when implemented, while the smaller group is 99% likely. Risk trade-offs guide a consumer of these results to make judgments on which variations to implement or further investigate.
It is for these reasons that the statistics module 121 reports significance as 1−FDR for each A/B test.
Why does FDR Control Work?
The relationship between FDR control and the conditional probability is as follows:
P(θ=0|{circumflex over (θ)}n)
using the notation from section 2.1. {circumflex over (θ)}n is used as short-hand for “all observed data required for the statistical decision procedure at hand.” In particular, it may now also depend on quantities like Vn. We can use the definition of conditional probability to write
A (empirical) Bayesian perspective on false discovery rate control is that we try to estimate the three quantities on the right of the equals sign to get an estimate of the conditional probability on the left. Then all hypotheses with an estimate lower than a pre-specified threshold are reported.
There are three steps. First, notice that {circumflex over (p)}≡P({circumflex over (θ)}n|θ=0) is simply the Type 1 error, or p-value, and we have those for each A/B test. Next assume that there is an unobserved prior proportion, π0, of null tests (no true difference).
Finally, the words data, observed difference, and p-value interchangeably. This is because a p-value is the amount of evidence collected against a null hypothesis of no true difference in conversion rates (θ=0). Smaller p-values denote more evidence against the null hypothesis so
P({circumflex over (θ)}n)=P(p≤{circumflex over (p)})
where {circumflex over (p)}={circumflex over (p)}({circumflex over (θ)}n) is the p-value as a function of {circumflex over (θ)}n, and this probability is computed over all tests, whether θ=0 or not. The unbiased, maximum likelihood estimate of this probability is just the number of boxes with p-values as small as the p-value of the test, divided by the number of boxes in the matrix (N). Or, if we order the A/B tests by their p-values, from smallest to largest, then
is the estimated FDR of the ith A/B test. The Benjamini-Hochberg method upper bounds π0≤1, and then rejects hypotheses based on this estimate. Since the Benjamini-Hochberg method controls the FDR described in (3), we get a duality between FDR and P(θ=0|{circumflex over (θ)}n).
The result is a group of hypotheses, all of which are guaranteed to have both definitions of false discovery rate, (3) and (4), below a threshold. The winning and losing variations (green and red boxes) reported are those with FDR less than the value set in your project level settings. The significance value reported for each A/B tests are estimates of P(θ=0|{circumflex over (p)}i).
Using FDR for Confidence Intervals
Along with false discovery rate is a closely related measurement which is called false coverage rate (FCR). False coverage happens when a confidence interval (on the lift in conversion rates) does not contain the true difference between variation and baseline, θ.
Intuitively, one wants a method controlling FCR to give the same guarantee as one for FDR, for example, at most 10% of significant confidence intervals do not contain the true difference. This statement seems innocuous. By definition, a marginal confidence interval on one A/B test already attains at most 10% chance of not covering the true difference, so looking at a collection of confidence intervals should similarly have at most 10% error rate of coverage. But as with false discovery rate, the danger lies in the single word significant because it encapsulates a selection procedure. The confidence intervals that end up being reported are not on a random selection of A/B tests, but on a specific subset of A/B tests—the ones that had high significance values. This selective action by the experimenter changes the coverage rate.
As a simple example, suppose we use the selection procedure—“only report a confidence interval if it doesn't contain 0.” Then no matter what coverage level the marginal confidence interval had, it now has exactly 0 chance of covering the true difference between variation and baseline when it is 0.
One procedure that guarantees FCR control regardless of the selection procedure is to require that at most 10% of all confidence intervals do not contain the true difference, and is achieved by reporting 1−q/N confidence intervals where q is the desired error rate and N the number of A/B tests. This is too stringent for the same reason that thresholding the chance of at most 1 false detection is. The FCR is then controlled too well in the sense that FCR is closer to 0 than q and so confidence intervals are too wide.
On the other hand, FCR should be controlled at the very least on all winners and losers. This led to the definition of a FCR controlling procedure which reports 1−qm/N confidence intervals (m is the number of significant variations). This lower bounds the proportion of false coverage by q/2 and has confidence intervals which do not contain 0 exactly when they are winners and losers by the FDR thresholding procedure.
Extending FCR to an A/B testing platform means that FCR should also be controlled on the test currently visible on a user's dashboard, regardless of its significance. The act of viewing an A/B test must mean that it is a candidate in an experimenter's selection procedure. Similarly, unobserved and non-significant A/B tests cannot be because we stipulate that a current significance level must be observed for selection.
To this end, we implement the following procedure for generating confidence intervals with FCR controlled at level q: for the ith box, report a standard confidence interval using the following altered nominal coverage levels
where: m is the number of conclusive results after applying a method to control FDR at threshold q.
This procedure has FCR control on precisely the confidence intervals of the set of results with estimated FDR below a threshold, and, in addition, the confidence interval currently viewed by the user, regardless of its significance.
Our modification shows another fundamental difference between online experimentation and classical hypothesis testing. An experimenters behavior may be directly observed during the test, and this observed behavior can be directly incorporated into inferential guarantees.
Does FDR Control Work with Sequential Testing?
The sequential testing described herein is designed to commute with applying the Benjamini-Hochberg procedure at every point in time, and still give the expected guarantees. In other words, FDRT is defined as the expected false discovery rate after t minutes have gone by, with t=0 being the time that the first test began, then we guarantee
E[FDRT]≤q
with q a pre-determined threshold, and T a stopping time, or rule describing a decision to stop and look at the matrix of tests. One example of such a decision T is “report results when 5 variations x goal pairs all have FDR below 10%.” As before, regardless of when you look at the matrix, if you call the tests with FDR≤q, then you are guaranteed to control false discovery rate at level q.
Technical Appendix: Sequential Testing with Likelihood Ratios
Below is the full derivation of the threshold on θn given in constraint (2). To do this, we change perspective from θn to the likelihood function. Let X1, X2, . . . denote the sequence of observations for visitors in the baseline group (where a one means the visitor converted, and zero means they did not); and let Y1, Y2, . . . denote the sequence of observations for visitors in the variation group. Given any candidate θ for the true conversion rate difference, the likelihood of θ based on the first n observations is defined as:
Ln(θ;{circumflex over (θ)}n)=Pθ(X1, . . . ,Xn/2,Y1, . . . ,Yn/2)
where Pθ denotes probabilities when the true difference is θ, and {tilde over (θ)}n is a sufficient statistic encoding all the information needed from the first n observations to calculate Ln. Until the end of this section, we drop the parameter {tilde over (θ)}n for ease of explanation, because it is not needed for the arguments here. It becomes important again in section 5, where we combine our results.
The likelihood function is a fundamental quantity that is of central importance across statistics. In any statistical model, it formalizes the plausibility of a candidate parameter value θ, given the data observed.
Recall that our overall objective comprises two parts: to adaptively learn the effect size we should anticipate, and then to stop as soon as possible to detect that effect when it holds. We begin by isolating the latter goal. Suppose that we knew that the true conversion rate difference was either zero or some given non-zero alternative {tilde over (θ)}. Then the likelihood ratio of {tilde over (θ)}n against zero is defined as
This represents the relative plausibility of the alternative.
At the start of the experiment L0({tilde over (θ)})=L0(0)=1 so this ratio is 1. Over the course of the experiment, the value of Λn({tilde over (θ)}) will fluctuate as θn does. In fact Λn({tilde over (θ)}) depends on the data only through the value of θn (that is the sense in which θn is “sufficient”) so placing a threshold on θn is equivalent to placing a threshold on Λn. But unlike θn, Λn does not converge to zero when the variation and the baseline are the same. Rather, an important property of the likelihood ratio is that, if the true difference is zero, then Λn({tilde over (θ)}) is a martingale. This means that, conditional on the value Λn we observe after n visitors, at a later stage m>n the expected value of Λm is still Λn; there is no average drift upwards or downwards.
Without drift, we can get to our goal just by placing a constant threshold on Λn. For any positive martingale started at 1, the probability of hitting a constant threshold b>1 is less than 1/b. So the rule: stop when Λn({tilde over (θ)})≥1/α gives always valid Type I error control at level α.
The proof of this fact (which relies on the Optional Stopping Theorem) is roughly as follows. Let Tb be the least n at which Λn≥k is achieved, with Tb=∞ if the threshold is never crossed. Fix some maximum number of visitors t and let TbΛt be the lesser of Tb and t. After TbΛt visitors arrive the expected value of Λn is still its initial value 1. But, by construction,
Taking the maximum visitors t→∞ bounds the probability of ever crossing the threshold at
P0(Tb<∞)≤1/b
We want to extend this approach so that the true difference can take any value. This formula is at the heart of the formulation of a sequential test as we now can control the probability that our likelihood ratio ever crosses the threshold b. It is important to note however, that a constant threshold on a single Λn({tilde over (θ)})≥ is not sufficient, because when we map back to θn the resulting threshold does not converge to zero. For example, while we are guaranteed to detect an effect of size {tilde over (θ)} if it exists, we are not guaranteed to detect smaller effects like {tilde over (θ)}/2 or {tilde over (θ)}/5. The target is to consider the likelihood ratio of every potential effect θ against zero simultaneously, and hone in on the appropriate candidate. This motivates a Bayesian approach, where we place a prior, π(θ), on the true effect.
Then we can monitor the average likelihood ratio
Λn({circumflex over (θ)}n)=∫Λn(θ)π(θ)dθ
which represents the relative plausibility of the aggregate alternative that considers every θ, weighted according to π(θ). Again we highlight the fact that our final statistic is a function of the data {circumflex over (θ)}n. In Bayesian terminology, Λn is called the Bayes factor in favor of the alternative hypothesis. Any mixture of martingales is itself a martingale (if each statistic tends to stay where it is, then so does the average). So, as before, we can get an always valid threshold on {circumflex over (θ)}n, by placing a constant threshold of 1/α on Λn.
We choose a normal prior
π(θ)˜N(0,τ)
both for computational convenience and because central limiting arguments make the distribution of effect sizes effectively normal for experiments run on any even moderately popular A/B testing platform.
Using this prior and integrating, we obtain the boundary in (2).
Given thresholds, it is easy to convert these tests into p-values. A p-value is the most conservative significance level at which the test is conclusive, so we take the p-value after n visitors to be the least α such that the threshold has already been crossed. As the experiment progresses, more thresholds are crossed and the p-value decreases, approaching some asymptotic value at n=∞. In terms of p-values, the statement that the probability that the α-level test is ever conclusive is α is equivalent to saying that the asymptotic p-values are uniformly distributed. So, at n=∞ the p-values are as aggressive as they can be, while still controlling Type I error. However during the experiment, the p-value is an overestimate of this asymptotic p-value so we are slightly more conservative: this can be considered the mathematical penalty we must pay to be free to stop tests as early as possible.
Traditional, fixed-horizon statistics
{circumflex over (p)}=p({circumflex over (θ)}n,θ′), traditional p-value for evidence against the null hypothesis, HO: θ=θ′
C({circumflex over (θ)}n,α)=[θ|p({circumflex over (θ)}n,θ)≥α], traditional confidence interval with 1−α coverage level
Statistics module 121 statistics
Λ({circumflex over (θ)}n), average likelihood ratio; inverse of new p*-value
FDR-adjusted p*-value
C*({circumflex over (θ)}n,α)=[θ|Λ({circumflex over (θ)}n)<1/α], new confidence interval with 1−α, coverage
Q*({circumflex over (θ)}n,α)=C*({circumflex over (θ)}n,FCR(α)), FCR-adjusted coverage interval
Shown on results
A reset of a variation test simultaneously performs two functions:
A confidence interval is a range of values that is likely to contain the true, or underlying value as mentioned above. For example a 90% confidence interval on the difference in conversion rates is a range of values, say CI(:9)=[c1; c2], that has at least a 90% chance of containing the actual difference of average conversion rates between variation and baseline, call them rv, and rb. In mathematical notation, Prob(c1≤rv−rb≤c2)≥0.9.
The confidence interval is intimately linked to statistical significance by the following statement: the statistical significance of a test crosses an α threshold at exactly the same time that an α confidence interval no longer contains 0.
The other fact used is that the observed conversion rate difference will get closer and closer to the actual difference of average conversation rates as more visitors enter a test. A conversion rate difference calculated from 10,000 visitors will likely be closer to the true difference than one calculated from 100 visitors and will eventually be arbitrarily close to the true average, as long as the underlying process causing conversion does not change.
For the first function of a rest, by setting a significance threshold of α=0.9 for example, no more than 10% of tests with no underlying difference are mistaken as erroneous winners or losers, due to chance variation—a false positive. Being in one of these 10% of tests means that our statistical significance rose above 90%, which in turn means that the corresponding 90% confidence interval no longer contained 0 conversion rate difference (fact 1). Furthermore, it is understood that the true difference is indeed 0 (no underlying difference) and so the observed conversion rate will eventually come arbitrarily close to 0 (fact 2). Hence, the observed conversion rate will eventually leave the confidence interval and not return, indicating that a test is currently being run that is falsely positive.
The second function is conceptually very similar. The confidence interval will eventually contain the observed difference in conversion rates α percent of the time, regardless of the size of the actual difference, as long as the underlying process does not change. So if the observed difference leaves the confidence interval, and stays outside the confidence interval, a conclusion is made that either we are in 1−α percent of tests where chance variation caused us to calculate an erroneous confidence interval, or the underlying process causing visitors to convert changed.
The reset policy is implemented by taking the canonical definition of a confidence interval described above, applying it to a sequential variation test to calculate statistical significance, and monitoring for events where the observed conversion rate difference leaves the confidence interval.
An implementation of a confidence interval which agrees with the canonical definition is the intersection of all intervals found by inverting the inverse likelihood ratio calculation at successive visitors counts. Let iL(θ,n) denote the inverse likelihood ratio for a test of the null hypothesis rv−rb=θ with data seen up to n visitors. The inverse, iL−1 (1−α,n) involves either the solution to a quadratic or cubic, hence an interval of possible values of θ. The confidence interval at N visitors is then implemented by:
CI(α,N)=∩n=1NiL−1(1−α,n),
This intersection corresponds mathematically to the minimization taken in the calculation of significance:
A reset is triggered once the observed conversion rate difference is no longer contained in the confidence interval. For example if xv and xb denote the observed conversion rates, a reset is triggered when:
xv−xb∉CI(α,N).
A reset is acted upon by removing the intersection in the confidence interval, and similarly the minimization in significance calculations:
CI(α,N)=iL−1(1−α,N); pN=L(0,N).
Finally, a version of the above procedure is implemented to properly account for multiple testing corrections. Since multiple testing corrections cause on test's significance calculation to be impacted by other tests, and a triggered reset causes an update in significance calculations, it may be that one test's reset triggers an update in another test's significance calculation. It is likely true that the rest-significance update loop cannot persist for more than a single cycle.
In one embodiment, a full implementation of the reset policy, confidence interval calculations and multiple testing updates will have complexity upper bounded by the following:
2(2N+N(N+1)/2)K
Where N is the number of visitors, or time points to calculate results, and K is the number of tests.
The implementation is a less computationally intensive version which opts for a more conservative definition of a confidence interval as the intersection of the current inverse likelihood ration and the one at the time the minimum significance was attained,
CI(α,N)=iL−1(1−α,N)∩iL−1(1−α,n*)
Where n* is the visitor count at which pN attained its minimum. This definition still preserves all properties of the confidence interval. An implementation of this simplified confidence interval, along with the reset policy and multiple testing updates is 6NK.
The foregoing described embodiments have been presented for the purpose of illustration; they are not intended to be exhaustive or to limiting to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.
Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, described modules may be embodied in software, firmware, hardware, or any combinations thereof.
Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.
Embodiments of the invention may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may include one or more general-purpose computing devices selectively activated or reconfigured by one or more stored computer programs. A computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
Described embodiments may also relate to a product that is produced by a computing process described herein. Such a product may include information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.
Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.
This application is a continuation of U.S. patent application Ser. No. 14/863,310, filed on Sep. 23, 2015, which is incorporated by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20060143545 | Reaume | Jun 2006 | A1 |
| 20070271352 | Khopkar | Nov 2007 | A1 |
| 20070271501 | Vasilik | Nov 2007 | A1 |
| 20090125270 | O'Shea | May 2009 | A1 |
| 20140020089 | Perini, II | Jan 2014 | A1 |
| 20140108640 | Mathis | Apr 2014 | A1 |
| 20140372901 | Catlin | Dec 2014 | A1 |
| Entry |
|---|
| AnyLogic, “AnyLogic Help”, 2014,Published by AnyLogic at https://help.anylogic.com/index.jsp?topic=%2Fcom.anylogic.help%2Fhtml%2Fanalysis%2FStatistics.html (Year: 2014). |
| Patel, “What Spending $252,000 On Conversion Rate Optimization Taught Me”, 2012, Published at https://www.quicksprout.com/2012/10/08/what-spending-252000-on-conversion-rate-optimization-taught-me/ (Year: 2012). |
| Quality America, “Data Collection, Analysis and Reporting”, 2016, Published by Quality America, at http://qualityamerica.com/LSS-Knowledge-Center/qualityimprovementtools/data_collection_analysis_and_reporting.php (Year: 2016). |
| Wingify, “What is A/B Testing?” The Complete Guide to A/B Testing, Aug. 2014, 8 Pages, Can be retrieved at <URL:https://vwo.com/ab-testing/>. |
| Optimizely, Inc., “What is A/B Testing?” Feb. 2014, 3 pages, Can be retrieved at <URL:https://www.optimizely.com/ab-testing/>. |
| Sauro, J., “10 Things to Know About A/B Testing,” Measuring U, 2012, 6 Pages, Can be retrieved at <URL:http://www.measuringu.com/blog/ab-testing.php>. |
| Castrounis, A., “How A/B Testing Works (for Non-Mathematicians),” kissmetrics, 2014, 7 Pages, can be retrieved at <URL:https://blog.kissmetrics.com/how-ab-testing-works/>. |
| Bartlett, J., “A/B testing—confidence interval for the difference in porportions using R,” The Stats Geek, Feb. 15, 2014, 4 pages, Can be retrieved at <URL: http://thestatsgeek.com/2014/02/15/ab-testing-confidence-interval-for-the-difference-in-proportions-using-r>. |
| Patel, N., “What Spending $252,000 on Conversion Rate Optimization Taught Me”, Quick Sprout, LLC, Oct. 8, 2012, 46 pages, can be retrieved at <URL:https://www.quicksprout.com/2012/10/08/what-spending-252000-on-conversion-rate-optimization-taught-me/>. |
| Qygard, A., “Rapid A/B-testing with Sequential Analysis”, Dec. 25, 2014, 6 Pages, can be retrieved at <URL:https://www.auduno.com/2014/12/25/rapid-a-b-testing-with-sequential-analysis/>. |
| Pekelis, L., “Statistics for the Internet Age: The Story Behind Optimizely's New Stats Engine,” Optimizely Blog, Jan. 20, 2015, 11 Pages, can be retrieved at <URL:https://blog.optimizely.com/2015/01/20/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/>. |
| Office Action for U.S. Appl. No. 14/863,310, dated Jan. 5, 2017, 22 Pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20170344456 A1 | Nov 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14863310 | Sep 2015 | US |
| Child | 15682163 | US |
