1. Technical Field
The present disclosure relates to market testing and more specifically to scheduled split testing by assigning user IDs to mutually exclusive groups.
2. Introduction
Market testing is an important part of almost every consumer-oriented business. Websites and Internet components of businesses have an advantage in that market testing can be conducted easily and at relatively low cost compared to “real-world” focus groups, and the testing can be conducted without the knowledge (and, therefore, bias) of the consumer. In most cases these tests evaluate a change in webpage design, content or features.
As with any test, variables must be tightly controlled in order to understand the cause of any variance between test results and historical performance. A typical way of achieving this is to use a control group in addition to the experimental test groups. Such testing—wherein a population is divided into a control group and one or more test groups—is well known and often referred to as split testing.
When split testing, a tester will often choose to enroll only a limited percentage of an entire audience into a test. In a test on a website, a determined percentage of users visiting the website can be enrolled into a test at random. In a naïve implementation, every visitor who is not already enrolled into a test is subject to the random possibility of being enrolled into the test. However, this technique results in a slow increase in the percentage of users enrolled because unenrolled users are continually subject to new chances for enrollment.
For example, consider that a test is meant to enroll ten percent of the total 1,000 visitors to a website and the test is to run for one week. A user who visits the website multiple times is subjected to the possibility of being selected for the test each time he visits the website. Such “double jeopardy” for unenrolled users results in a gradual increase in the proportion of users enrolled into the test beyond the intended ten percent.
An additional consideration is that prior art split testing techniques often are not suitable for businesses with many different business units. Identification of users who appear in the user populations of multiple business units needs to be accounted for in some testing scenarios. The ability to restrict a user to one test within a business unit would offer protection against “shadow” or “halo” interaction effects while still allowing multiple business units in the network to enroll a particular user at the same time. A further consideration is that users making return visits are not treated uniformly on subsequent visits to the website. Such inconsistent treatment can conflate control and test group experiences. Users making return visits should be handled consistently when they return to a website.
Additional features and advantages of the disclosure will be set forth in the description that follows. They will be obvious from the description or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations of embodiments particularly pointed out in the appended claims. These and other features of the disclosure will become more fully apparent from the following description and appended claims, or can be learned by the practice of the principles set forth herein.
Disclosed are systems, methods, and non-transitory computer-readable storage media for scheduling users into tests. In a preferred embodiment the users are visitors to a website, and the website administrator desires to know how the population of users is likely to react to one or more new features, layouts, content items, etc. on the website. The website administrator can configure the present technology to schedule a percentage of the visitors to the website into a test in such a fashion that the user is treated consistently during each visit to the website and that avoids errors that are common in other testing systems, particularly those in which visitors are repeatedly assessed for inclusion into a given test.
In one embodiment a first set of user groups called “enrollment buckets” is allocated to a business unit. The enrollment buckets can be subdivided such that a portion of the enrollment buckets is allocated to at least one test. When a user navigates to the webpage, a user ID associated with the user can be hashed using a first hash function to map the user to one of the enrollment buckets.
A second set of buckets called “test buckets” is also subdivided such that a portion of the buckets is allocated to a test variation (other portions can also be allocated to other test variations, if applicable) and a portion of the test buckets is allocated to a control group. If the user was mapped to a hash bucket that is allocated to a test in the enrollment buckets, the user ID will again be hashed using a second hash function to map the user to one of the test buckets.
Users whose user IDs have been mapped to a test variation according to the test buckets will be exposed to the test page or test page element(s) corresponding to that variation of the test. Users whose user IDs have been mapped to the control group will be exposed to the control page or control page element(s).
In some embodiments the user's ID can be combined with additional information before it is hashed by the one or more hash functions. For example, in an enterprise of many different business units it might be desirable, in some situations, to combine the user's ID with a business unit ID which corresponds to the web page the user is visiting. In these embodiments the user can be recognized as a unique user to the system when accessing the system through different portals, when such is desired.
In some embodiments the first and second hash function can be the same hash function. Persons of skill in the art will recognize the appropriateness of selecting different hash functions for specific systems.
In order to describe the manner in which the above-recited and other advantages and features of the disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only exemplary embodiments of the disclosure and are not therefore to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying drawings in which:
Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the disclosure.
With reference to
The system bus 110 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. A basic input/output (BIOS) stored in ROM 140 or the like, may provide the basic routine that helps to transfer information between elements within the computing device 100, such as during start-up. The computing device 100 further includes a storage device 160 such as a hard disk drive, a magnetic disk drive, an optical disk drive, tape drive or the like. The storage device 160 can include software modules 162, 164, 166 for controlling the processor 120. Other hardware or software modules are contemplated. The storage device 160 is connected to the system bus 110 by a drive interface. The drives and the associated computer readable storage media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the computing device 100. In one aspect, a hardware module that performs a particular function includes the software component stored in a non-transitory computer-readable medium in connection with the necessary hardware components, such as the processor 120, bus 110, output device 170, and so forth, to carry out the function. The basic components are known to those of skill in the art and appropriate variations are contemplated depending on the type of device, such as whether the device 100 is a small, handheld computing device, a desktop computer, or a computer server.
Although the exemplary embodiment described herein employs the storage device 160, it should be appreciated by those skilled in the art that other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, digital versatile disks, cartridges, random access memories (RAMs) 150, read only memory (ROM) 140, a cable or wireless signal containing a bit stream and the like, may also be used in the exemplary operating environment. Non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.
To enable user interaction with the computing device 100, an input device 190 represents any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device 170 can also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems enable a user to provide multiple types of input to communicate with the computing device 100. The communication interface 180 generally governs and manages the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.
For clarity of explanation, the illustrative system embodiment is presented as including individual functional blocks including functional blocks labeled as a “processor” or processor 120. The functions these blocks represent may be provided through the use of either shared or dedicated hardware, including, but not limited to, hardware capable of executing software and hardware, such as a processor 120, that is purpose-built to operate as an equivalent to software executing on a general purpose processor. For example the functions of one or more processors 120 presented in
The logical operations of the various embodiments are implemented as: (1) a sequence of computer implemented steps, operations, or procedures running on a programmable circuit within a general use computer, (2) a sequence of computer implemented steps, operations, or procedures running on a specific-use programmable circuit; and/or (3) interconnected machine modules or program engines within the programmable circuits. The system 100 shown in
Having disclosed some components of a computing system 100, the disclosure now turns to
The present technology sets up a test by designating a first set of buckets 205 called “enrollment buckets.” The buckets can be any type of data structure known to those skilled in the art, such as an array. In some embodiments, each bucket can be configured to contain a test ID signifying what test, if any, a user should be enrolled. A test ID can be any variable such as an integer or string which can be used to assign a value to the bucket. By defining the test ID value for each bucket, a subset of the enrollment buckets can be assigned into one or more test groups 210.
Returning to
In some embodiments, each of the enrollment buckets can include a map which allows each bucket to be assigned to a given test according to a map key. For example, if the map key is date, the test ID can be assigned individually for each day. A tester can, therefore, schedule and inventory a test in advance. For example, if the map key is date and the map is configured to accept 30 inputs, a tester can assign the test ID for the bucket for the next 30 days so that the bucket is assigned according to the value of the test ID assigned to the specific day. Although date is used as an example, the map key can be any interval such as week, month, hour, etc. In some embodiments, the map key is not based on time, for example a map key can be whether a user is a first time visitor, or some other characteristic of a user, connection, or other known or detectable attribute.
The number of enrollment buckets as well as the number of buckets assigned in their corresponding subsets can be configured to test any predetermined percentage of users. Returning to
The same concept can be used to employ multiple test variations as shown in
To assign a user to a bucket, a user ID assigned to a user is entered into a hash function which associates the user ID with a bucket.
The hash function can be any method of hashing known to one of skill in the art. However, it is that the specific function that is selected consistently assign users with the same user ID to the same bucket such that every time an identified user returns to the system the user will be treated in the same fashion as in previous visits. In addition, it is preferable that the function exhibit random uniformity such that roughly the same number of users is assigned to each bucket.
A hash function may also incorporate a modulo operation to ensure that each hash key is assigned to a bucket within the assigned range. For example, if 10,000 user IDs are hashed, and only 1000 buckets are configured, the hash function may apply a modulo operator that divides by the total number of buckets, in this example 1000, and then returns the remainder to ensure that the user IDs are only hashed to buckets 0-999. For example, if the user ID inputted into the hash function returned 2,325, the modulus function would divide by 1000 and return the remainder of 325. The user ID would then be assigned to bucket 325. The modulo operator may be incorporated into the hash function itself or applied to the output of the hash function.
The user ID assigned to a user can be created using any well-behaving method of ID assignment. For example, in some embodiments Apache's mod_unique_id is used. The user ID can be stored as a cookie in the user's browser, however other client-side storage mechanisms or session-based methods can be used, as can session-based methods. In some embodiments HTML5's local storage feature can be used.
In some embodiments the method determines whether the user's computer accepts cookies by attempting to set a cookie while responding to the user's initial request with an HTTP redirect. If the user's browser accepts cookies, the newly created user ID will be stored in the user's browser. The new location of the HTTP redirect can be identical to the original URL except for a new query argument added to the URL query parameters. This query parameter is a “cookied already” flag which indicates that there has been an attempt to set the cookie, so that infinite redirects can be avoided when a client is not accepting cookies. If a request is received with the flag set, but no user ID is present in the call, then it can be concluded that the user has cookies disabled and the user will not participate in the test.
In some embodiments reliable and consistent assignment of a user to a bucket can be accomplished without using scripts on a webpage and without storing cookies with assigned user IDs. In such embodiments the website can require that a user first sign in using a user account such that the user can be uniquely identified.
When the script determines that a user does have a user ID 710, the user ID is sent to the test server and entered into a hash function 730 where it is associated with an enrollment bucket. In some embodiments, the hash function may output a number that is entered into a modulo function to ensure the output is within the range of buckets. In some embodiment the modulus function may be incorporated into the hash function so that the hash function assigns the user to a bucket.
After a user ID has been hashed to an enrollment bucket 730, it is determined whether the bucket is assigned to a test or remains unassigned 735. This can be done by checking the test ID associated with the bucket. If the bucket is not assigned to a test 735 then the user is not enrolled into a test 720. If the bucket is assigned to a test 735 then the user's user ID is hashed into the test buckets 740 associated with the test to which the enrollment bucket was assigned. If the user ID was hashed to a test bucket assigned to a test variation then the test variation is displayed to the user 745. If the user was hashed to a test bucket assigned to the control then control 745 is displayed to the user.
One advantage of the present technology is that a user is treated consistently every time he returns to the website. Because the enrollment buckets and the test buckets are predetermined, and because the user is consistently assigned to the same bucket, his treatment will be the same on return visits. The present technology also takes into account that a user might access the testing system through multiple business units of the same company. While the present technology has been discussed primarily with respect to hashing users into enrollment and test buckets, it should be appreciated that any method that consistently assigns users into buckets can be used consistent with the other principles of the present technology herein.
Thus far, the present technology has been primarily discussed with respect to a single website or single publisher environment. However, some embodiments of the present technology are suitable for larger publishers with more than one property. Within a network of sites, a user should be allowed to participate in more than one test, particularly if those tests are on distinct sites or on distinct sections within a single site, where interaction effects can much more safely be ignored. In some embodiments this can be accomplished by partitioning the website inventory, and the pages on which the tests are to be run can be assigned a partition key. The present invention calls this key “publisher ID.” The publisher ID can represent a site, a group of sites, a publisher, or even an arbitrary collection of pages within a site. In such embodiments, the user ID and publisher ID are hashed together to assign the user into the appropriate bucket.
For example, to create a test, a tester may log in to the TMI 905 and set test parameters such as test name, begin and end dates, metrics to be recorded, percentage of audience to be tested, number of variations, and percentage of tested users to see each variation or control. A tester may also upload any content to be displayed when a user is served a variation of a webpage.
After receiving test parameters from a tester, the TMI 905 initializes the test and then transmits the test data to a test server 910. Initialization may include: designating the enrollment and test buckets 915, 920, assigning subsets of each to meet test parameters, and generating any scripts which may be needed to implement the test. The generated scripts are outputted to the tester to be placed into the web pages to be tested. The scripts can be of any type known by those skilled in the art, e.g. JavaScript.
In one embodiment, a JavaScript script 925 embedded in a webpage 930 to be tested is used to communicate between the test server 910 and the user's computer. The script 925 can be configured to check if a user's device 935 is set to accept cookies, check if a cookie is on the user's device 935, set a cookie on a user's device 935 if allowed, and identify the publisher ID. Publisher ID can be used to partition the website inventory, i.e., the pages on which the tests are to be run. The publisher ID can represent a site, a group of sites, a publisher, an arbitrary collection of pages within a site, etc. The script 925 calls the test server 910, transmitting parameters which may include publisher ID, test ID and user ID (cookie value).
The data sent to the test server 910 from the script 925 is then used to determine whether the user is enrolled in the test, and if so, whether the user should be served a test variation or the control. As explained above, if the script 925 determines that the user ID cookie already exists on the user's device 935, the user ID is sent to the test server 910 where it is used as a key in the first hash function, to be hashed into the enrollment buckets 915. In some embodiments if enrollment buckets 915 have been allocated to multiple business groups as illustrated by
If the user ID is hashed into a bucket assigned to a test in the enrollment buckets 915, the user ID is then entered into the second hash function to be assigned to a test bucket 920. Depending on whether the user ID is hashed into a test bucket 920 assigned to a test variation or control, the corresponding content is served to the user. A test group ID identifying which variation was served is also returned to the script 925.
A record of the user's interaction with the website 930 as well as the test ID are collected at a data warehouse 940 and then processed by a test data processing application 945. Metrics derived from this record compare user behavior between the control group and variations. Statistical tests determine whether differences between the control group and the variations are significant. Metrics collected can include, but are not limited to: amount of time the user spent on the site, which links were clicked, which site the user entered from, the time of day, the day of the week etc. A metric can also be a ratio where the numerator is a sum or count of some value and the denominator is a count of users or sessions that produced the value. The metric can also be calculated for a portion of the total audience, referred to as a dimension, rather than the full audience. Some of examples of dimensions include: new visitors, weekend visitors, visitors arriving from a search engine, registered users, etc. Different tests can use different dimensions and/or different metrics. In some embodiments, the collected data can be stored in a predetermined structure.
Using the technology described herein users can be effectively scheduled to participate in a test and be treated in a consistent manner. The present technology gives the tester greater control over the population selected for the test, thus reducing unanticipated errors and tainting the test candidate pool with other tests.
In some embodiments, each bucket can contain a map with keys corresponding to different values. For example, in some embodiments a map's keys can be dates such that a bucket can be mapped into each day from the present day through the following 30 days and the map's values are lists of tests, each of the lists of tests representing the tests in which the users of the bucket will participate on the specified date. In some embodiments the list length can be limited to 1 to completely avoid potential problems caused by test interaction. Although date has been chosen as an example of the key, a key with a finer granularity can be used. For example, tests can be scheduled by the hour or minute. Users can only participate in the tests to which they are assigned.
Returning to
In some embodiments the test server 910 can be constrained so that once buckets are chosen for a proposed test, the buckets cannot be changed over the course of the test. This ensures that users do not shift among multiple variations during the test. In this type of embodiment, the test server 910 can find buckets which are free for the entire duration of a proposed test, and those buckets remain that test's buckets for the entire duration. New buckets can be added to the test; however, none can be removed. In some embodiments, the buckets can be reallocated on future days as long as all users who hash into a given bucket will continue to hash to the same set of tests during the duration of the test.
In some embodiments a test can be run across multiple publishers. For example a publisher or collection of publishers may wish to make a change across multiple divisions of their company and view the results as a whole with a consistent set of users enrolled in the test across each site. For example, an enterprise that has separate news and sports web sites may treat those as distinct publishers; however, a tester may instead wish to test a common change across both sites. To accomplish this goal a tester can group multiple publishers together to create a super-publisher and run a test so that users are hashed into the same set of enrollment buckets when they enter a site from any of the publishers within the test. In some embodiments a user's ID will be consistent across all publishers so that users are treated consistently when entering the test from multiple sites.
In some embodiments a publisher can run a test across multiple publishers while concurrently running a single publisher test. For example, Publisher A can be a part of a multi-publisher test with Publisher B and run a single publisher test at the same time. In some embodiments, a publisher can designate a portion of their total user traffic to be for the multi-publisher test while a portion of the remaining user traffic can be designated to participate in another test running on the publisher's site.
When a user visits a publisher site, the user will be consistently hashed into the same enrollment bucket for that publisher. Although a user may be treated consistently every time they visit a publisher, in some embodiments a user may not be treated consistently across multiple publishers. For example, as illustrated, when User 1 enters through Publisher A 1305, User 1 is hashed into Publisher A's Bucket 1 which has been assigned to the multi-publisher test, however when User 1 enters through Publisher B 1310, User 1 is hashed into Publisher B's Bucket 3 which is unassigned.
One potential outcome of the embodiment illustrated in
When a user is hashed into an unassigned bucket for a publisher that is part of a multi-publisher test, that user's ID can be hashed into the enrollment buckets of the other publishers in the multi-publisher test to check whether the user has been enrolled in the multi-publisher test. If the user has been enrolled into the multi-publisher test by a different publisher then the user will be treated as part of the multi-publisher test. The user's ID and publisher ID will be hashed into the multi-publisher test buckets 1325 to determine what treatment the user should be given, i.e., test, control, etc.
The embodiment illustrated in
In some embodiments, the test system can be configured to give priority to a multi-publisher test. In such embodiments, whenever a publisher is enrolled in a multi-publisher test, the system can hash the user ID as if the user had entered through each publisher to determine if the user is part of a multi-publisher test. If the user is part of any multi-publisher test, the user can be preferentially associated with the multi-publisher test.
Embodiments within the scope of the present disclosure may also include tangible and/or non-transitory computer-readable storage media for carrying or having computer-executable instructions or data structures stored thereon. Such non-transitory computer-readable storage media can be any available media that can be accessed by a general purpose or special purpose computer, including the functional design of any special purpose processor as discussed above. By way of example, and not limitation, such non-transitory computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions, data structures, or processor chip design. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable media.
Computer-executable instructions include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Computer-executable instructions also include program modules that are executed by computers in stand-alone or network environments. Generally, program modules include routines, programs, components, data structures, objects, and the functions inherent in the design of special-purpose processors, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
Those of skill in the art will appreciate that other embodiments of the disclosure may be practiced in network computing environments with many types of computer system configurations, including personal computers, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
The various embodiments described above are provided by way of illustration only and should not be construed to limit the scope of the disclosure. Those skilled in the art will readily recognize various modifications and changes that may be made to the principles described herein without following the example embodiments and applications illustrated and described herein, and without departing from the spirit and scope of the disclosure.
This application is a continuation-in-part of U.S. Utility patent application Ser. No. 13/190,320, filed Jul. 25, 2011, and also claims the benefit of U.S. Provisional Patent Application No. 61/536,816, filed Sep. 20, 2011, the disclosures of which are incorporated herein by reference in their entirety.
Number | Date | Country | |
---|---|---|---|
61536816 | Sep 2011 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 13190320 | Jul 2011 | US |
Child | 13287827 | US |