Patent Application
20210295358
Advertising (ad) campaign strategies are increasingly reliant on the collection of vast amounts of data regarding potential customers to determine when and where to target advertisements in order to best ensure a successful ad campaign. Moreover, in online advertising, it is important to be able to correctly and accurately evaluate the performance of an ad campaign in real-time. The performance of an ad campaign demonstrates whether or not an advertising strategy results in a positive return on investment (ROI) and guides further actions, such as budget allocation and campaign operations, for example.
One technique for assessing ad campaign performance during the campaign is to evaluate surveys completed by human responders who are exposed to the advertising, as well as from human responders who are not. Surveys are typically collected from exposed-group responders (responders that have been exposed to an ad campaign that is being assessed) and control-group responders (responders that have not been exposed to the ad campaign), in real-time. Because the effectiveness of the ad campaign may be determined by the difference between the survey responses collected form the exposed-group and control-group responders, the accuracy of that effectiveness determination depends on how well the sampled population of responders represents the overall target consumer population for the ad campaign.
There are provided systems and methods for synchronizing survey collection with ad distribution during an ad campaign, substantially as shown in and/or described in connection with at least one of the figures, and as set forth more completely in the claims.
The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals. Moreover, the drawings and illustrations in the present application are generally not to scale, and are not intended to correspond to actual relative dimensions.
The present application discloses systems and methods for synchronizing survey collection with ad distribution during an ad campaign which overcome the drawbacks and deficiencies in the conventional art. It is noted that, as used in the present application, the term “ad” refers to an advertisement or advertisements, or to advertising, as is readily interpretable from the context in which the term “ad” is used. By way of example, the expressions “ad data” and “ad volume” refer respectively to advertisement data and advertisement volume, while the expression “ad campaign” refers to an advertising campaign.
It is further noted that, as used in the present application, the terms “automation,” “automated”, and “automating” refer to systems and processes that do not require human intervention. Although, in some implementations, a human system administrator may review or even modify survey distribution decisions made by the systems and according to the methods described herein, that human involvement is optional. Thus, the survey distribution determinations described in the present application may be performed under the control of hardware processing components executing them.
As further shown in
By way of overview, when a survey is run, the same survey questionnaire is typically served to the exposed-group and control-group within a target audience. These two sets of surveys are collected simultaneously but from different people (i.e., audience members in the exposed-group and control-group are mutually exclusive). The total number of surveys to be collected from each group during an ad campaign, (hereinafter “the survey quota”) are usually equal, but don't have to be. Business rules typically determine how many control-group and exposed-group surveys are desirable. The numbers of collected control-group surveys and exposed-group surveys are counted separately, each under the guide of its own survey control rates.
With respect to the representation of system 100 shown in
It is further noted that although
Computing platform 102 may correspond to one or more web servers, accessible over a packet-switched network such as the Internet, for example. Alternatively, computing platform 102 may correspond to one or more computer servers supporting a wide area network (WAN), a LAN, or included in another type of private or limited distribution network.
By way of background, a live ad campaign typically runs on multiple content distribution platforms 136a and 136b, e.g., digital streaming services such as those provided by Hulu® or Twitch®, for example that display ads during the ad campaign. Each content distribution platform delivers some volume of ad units on any day during the ad campaign. Marketing researchers have found that audience members sharing the same demographic parameters, such as gender and/or age, for example, and who receive ads from the same content distribution platform can be shown statistically to respond very similarly to an ad campaign. According to the present inventive principles, a target audience for an ad campaign includes a cohort of individuals who receive content from the same content distribution platform and share at least one other audience parameter in common, such as a demographic parameter, for example.
Thus all members of target audience 150 are users of the same content distribution platform, e.g., content distribution platform 136a or 136b, and share at least one other audience parameter. Similarly, all members of target audience 154 are users of the same content distribution platform, e.g., content distribution platform 136a or 136b, and share at least one other audience parameter. Examples of audience parameters may include demographic parameters such as age, and/or gender, and/or household income, as well as data management platform (DMP) segments providing Internet tracking data indicative of audience member interests, and/or content data such as a show name or episode number of content being consumed by an audience member, to name a few. It is noted that target audiences 150 and 154 must differ from one another with respect to at least one characteristic. For example, target audiences 150 and 154 may be identified with the same content distribution platform but would then differ in at least one other audience parameter. Conversely, target audiences 150 and 154 may share all other relevant audience parameters in common but receives its content from a different one of content distribution platforms 136a or 136b.
It is further noted that the audience parameter and content distribution platform distinctions between target audiences 150 and 154 described above are by reference to a single ad campaign. More generally, multiple ad campaigns may run concurrently and may be directed substantially to the same target audiences having the same demographic characteristics and utilizing the same content distribution platform. That is to say, multiple ad campaigns can use target audience 150 (defined by a demographic parameter and a content distribution platform) and target audience 154 (defined by a demographic parameter and a content distribution that may overlap with but have differences from what defines target audience 150) without changing the members of each target audience. For example, concurrent ad campaigns for competing cola flavored soft drinks may both be directed to target audience 150, as well as to target audience 154. It is also noted that although
Although, as noted above, each of target audiences 150 and 154 includes a cohort of individuals who receive content from the same content distribution platform and share at least one other audience parameter in common, those features shared in common by the members of each of target audiences 150 and 154 may be more extensive than merely a common content distribution platform and one or more demographic parameters. That is to say, one of the advantages of the present solution for synchronizing survey collection with ad distribution is that it can be easily extended to target audiences based on their sharing any of several or hundreds of parameters in common. Thus, in addition to, or as alternatives to content distribution platform utilization and demographics, each of target audiences 150 and 154 may be identified based on audience parameters such as their common regional location, the type of device used to view or otherwise consume content (e.g., smartphone, tablet computer, television), or the type of content (e.g., movies, television shows or other episodic content, video game gameplay, e-sports tournaments, etc.), to name a few examples.
Not all members of a particular target audience receive the ads being evaluated during the ad campaign. For example, only exposed-group 151 of target audience 150 receives ads 132, and only exposed-group 155 of target audience 154 receives ads 132. Control-group 152 of target audience 150 does not receive ads 132, and control-group 156 of target audience 154 does not receive ads 132. Moreover, not all members of a particular target audience are surveyed during a particular time interval of the ad campaign. According to the present concepts, for each of target audiences 150 and 154, two ratios are particularly relevant to the accuracy with which the success of an ad campaign can be evaluated: 1) the ratio of the number of members of the target audience who are surveyed to the number of members of all target audiences included in the ad campaign who are surveyed, i.e., the “survey group”; and 2) the ratio of the number of all members of the target audience to the number of all members of all target audiences included in the ad campaign. If those two ratio numbers match, or are very close to each other for every target audience included in the ad campaign, then members of every target audience are equally represented in the survey group as well as the overall audience population as a whole. When that is the case, the collected survey data can be interpreted as accurately reflecting the collective target audience response to the ad campaign.
As a specific example, suppose an ad campaign delivers one hundred thousand (100k) ad units on two content distribution platforms 136a and 136b. Among them, 30k, 30k, 15k, 25k ad units are delivered, respectively, to female viewers of content distribution platform 136a (F-136a), male viewers of content distribution platform 136a (M-136a), female viewers of content distribution platform 136b (F-136b), and male viewers of content distribution platform 136b (M-136b). In such a case, the different target audiences identified by content distribution platforms F-136a, M-136a, F-136b, and M-136b account for 30%, 30%, 15%, and 25% of the overall audience, respectively. If 1,000 surveys are collected, it would be desirable for 300, 300, 150, 250 survey responses to be collected, respectively, from target audiences identified by content distribution platforms F-136a, M-136a, F-136b, and M-136b. When that happens, surveyed users are considered to be aligned with the overall audience, and the collected surveys may form the basis for accurate ad campaign evaluations.
An additional complication is caused by fluctuations in traffic on content distribution platforms 136a and 136b, such as daily fluctuations, or fluctuations over any other predetermined time interval of the ad campaign. In order for the survey process to form the basis of accurate ad campaign evaluations, the collection of surveys has to follow the pace of ad distribution, for each and every day or other predetermined time interval during the ad campaign. Again, as a specific non-limiting example, and based on the exemplary ad and survey numbers described above, if 10k of the 30k content distribution platform F-136a ad units were delivered on one particular day, it would be desirable to collect 10k/30k=⅓ of total surveys on the same day, i.e., collect 100 surveys that day from the target audience identified by content distribution platform F-136a.
Thus, given the survey quota and the total number of ad units to be distributed during the ad campaign (hereinafter “the ad budget”), the challenge is to develop a survey distribution control solution that adjusts survey distribution volume every day, or during each of any predetermined time interval of the ad campaign, for each combination of content distribution platform and audience member demographics, so that surveys are collected at substantially the same pace as ad units are distributed during each predetermined time interval of the ad campaign.
From a computational standpoint, let S(p, d, t) denote the number of surveys collected on content distribution platform p for user demographic d during predetermined time interval t, and let SQ denote the survey quota. In addition, let A(p, d, t) denote the number of ad units (i.e., the ad volume) distributed on content distribution platform p for user demographic d during predetermined time interval t, and let AT denote the total ad budget. The survey distribution control solution disclosed by the present application adjusts the distributed survey sampling rate SSR(p, d, t) on content distribution platform p for user demographic d during predetermined time interval t, such that:
S(p,d,t)/SQ=A(p,d,t)/AT
for every combination of p, d, and t during the ad campaign.
It is noted that the above calculation is performed separately for the exposed-group members and the control-group members of each target audience. Thus, where SControl(p, d, t) denotes the number of surveys collected on content distribution platform p for control-group members sharing user demographic d during predetermined time interval t, SQ-Control denotes the control-groups survey quota, SExposed (p, d, t) denotes the number of surveys collected on content distribution platform p for exposed-group members sharing user demographic d during predetermined time interval t, and SQ-Exposed denotes the exposed-groups survey quota it is desirable that:
S
Control(p,d,t)/SQ-Control=SExposed(p,d,t)/SQ-Exposed=A(p,d,t)/AT
where once again A(p, d, t) denote the number of ad units distributed on content distribution platform p for user demographic d during predetermined time interval t, and let AT denote the total ad budget.
In addition to survey distribution control module 218, in various implementations software code 210 may include one or more of present error module 212 generating present error value 222, forward cumulative error module 214 generating forward cumulative error value 224, and backward cumulative error module 216 generating backward cumulative error value 226. Also shown in
Ad data 234, survey data 246, and next survey sampling rate 228 correspond respectively in general to ad data 134, survey data 146, and next survey sampling rate 128, in
Software code 210 corresponds in general to software code 110, in
The functionality of software code 110/210 will be further described by reference to
Referring now to
In some implementations, an ad campaign may span several days, weeks, or months. However, in other implementations an ad campaign may be relatively brief, lasting no longer than the telecasting of a live event, such as a sporting event for example. As a result, and depending on the desired frequency with which survey collection and ad distribution are to be synchronized, the predetermined time interval over which the volume of distributed ads 132 is counted may vary depending on the specific ad campaign use case. In some instances, it may be advantageous or desirable for the predetermined time interval to be a twenty-four hour time interval, resulting in the ad volume being a daily ad volume. However, in other use cases, the predetermined time interval may be shorter or substantially shorter than twenty-four hours, such as a few hours, one hour, or less than sixty minutes. In yet other use cases, it may be advantageous or desirable for the predetermined time interval to be longer than twenty-four hours.
Ad data 134/234 including the ad volume distributed to each of target audiences 150 and 154 during the predetermined time interval of the ad campaign may be generated by ad delivery system 130, and may be received in action 361 by software code 110/210 of system 100, executed by hardware processor 104. As shown in
It is noted that the data flow shown in
Flowchart 360 continues with receiving survey data 146/246 describing a survey volume distributed to each of target audiences 150 and 154 and a survey response volume collected from each of target audiences 150 and 154 during the predetermined time interval (action 362). Action 362 corresponds to receiving survey data 146/246 describing the volume of surveys 142a (i.e., the number of surveys) distributed to exposed-group 151 and control-group 152 of target audience 150 during the same predetermined time interval of the ad campaign over which ad data 134/234 is collected, as well as the volume of surveys 142b distributed to exposed-group 155 and control-group 156 of target audience 154 during that predetermined time interval, and so forth for each target audience. Survey data 146/246 also describes the volume of survey responses 144a (i.e., the number of survey responses) collected from exposed-group 151 and control-group 152 of target audience 150 during the same predetermined time interval, as well as the volume of survey responses 144b collected from exposed-group 155 and control-group 156 of target audience 154 during the same predetermined time interval, and so forth for each target audience.
Survey data 146/246 including the survey volume distributed to each of target audiences 150 and 154 and the survey response volume collected from each of target audiences 150 and 154 during the predetermined time interval may be generated by survey distribution and collection system 140, and may be received in action 362 by software code 110/210 of system 100, executed by hardware processor 104. As shown in
It is reiterated that the data flow shown in
In some implementations, flowchart 360 continues with the optional action of estimating a next ad distribution volume for each of target audiences 150 and 154 for a next predetermined time interval of the ad campaign (action 363). As noted above, the volume of ads 132 distributed to target audience 150 and the volume of ads 132 distributed to target audience 154 may fluctuate over the course of the ad campaign. For example daily fluctuations in traffic on one or both of content distribution platforms 136a and 136b may result in such fluctuations. Consequently, in implementations in which the predetermined time interval referred to by actions 361 and 362 is a day, for example, the volume of ads distributed on one day may be different from the volume of ads distributed the next day, resulting in different next ad distribution volume 138/238. Next ad distribution volume 138/238 may be estimated based on data received from ad delivery system 130, for example, and that estimation may be performed in optional action 363 by software code 110/210 of system 100, executed by hardware processor 104.
According to some implementations, flowchart 360 can conclude with determining next survey sampling rate 128/228 for each of target audiences 150 and 154 for the next predetermined time interval of the ad campaign based at least on ad data 134/234 and survey data 146/246, such that the volume of survey responses 144a and 144b collected from each of target audiences 150 and 154 matches the ad distribution volume to each of target audiences 150 and 154 within a predetermined threshold during each predetermined time interval of the ad campaign (action 364). Action 364 may be performed using survey distribution control module 218 of software code 110/210, executed by hardware processor 104 of system 100. It is noted that in implementations in which flowchart 360 includes optional action 363, the determination of next survey sampling rate 128/228 for each of target audiences 150 and 154 for the next predetermined time interval of the ad campaign may be further based on the estimated next ad distribution volume.
In some implementations, the predetermined threshold within which the survey responses collected from each target population must match may be one or a few percentage points, such as one to five percent, for example. In some implementations, as noted above, survey distribution control module 218 may be implemented as a PID controller. In those implementations, survey distribution control module 218 acts as a closed-loop control module that determines the next survey sampling rate based on one or more negative feedback signals.
Referring to
For example, next survey sampling rate 128/228 may be further determined as the sum of the value determined in action 364 and present error value 222. In some implementations it may be advantageous or desirable to apply a weighting factor to present error value 222 in such a sum. For instance, where NSSRF is the further determined value of next survey sampling rate 128/228, NSSRI is the initial value of next survey sampling rate 128/228 determined in action 364, and PE is present error value 222, NSSRF may be expressed as:
NSSRF=NSSRIαPE,
where α is a tunable weighting factor.
As an alternative to action 365, or in addition to it, flowchart 360 can be extended to include calculating forward cumulative error value 224 for each of target audiences 150 and 154, and the predetermined time interval, based on the difference between a total ad volume distributed to each target audience and a total survey response volume collected from that target audience since the beginning of the ad campaign, and further determining next survey sampling rate 128/228 for each of the target audiences for the next predetermined time interval based on forward cumulative error value 224 since the beginning of the ad campaign (action 366). Action 366 may be performed by software code 110/210, executed by hardware processor 104, and using forward cumulative error module 214 and survey distribution control module 218.
For example, next survey sampling rate 128/228 may be further determined as the sum of the value determined in action 364 and forward cumulative error value 224, or the sum of the value determined in action 365 and forward cumulative error value 224. In some implementations it may be advantageous or desirable to apply a weighting factor to forward cumulative error value 224 in such a sum. For instance, where NSSRF is the further determined value of next survey sampling rate 128/228, NSSRI is the initial value of next survey sampling rate 128/228 determined in action 464, and PE is present error value 222, as noted above, and where FCE is forward cumulative error value 224, NSSRF may be expressed as:
NSSRF=NSSRI+βFCE,
where β is a tunable weighting factor, or as:
NSSRF=NSSRI+αPE+βFCE,
where each of α and β is a tunable weighting factor.
As an alternative to either or both of actions 365 and 366, or in addition to actions 365 and 366, flowchart 360 can be extended to include calculating backward cumulative error value 226 for each of target audiences 150 and 154, and the predetermined time interval, based on the difference between a desired deficit in the survey quota and the actual deficit in the survey quota during the present predetermined time interval, and further determining next survey sampling rate 128/228 for each of the target audiences for the next predetermined time interval based on backward cumulative error value 226 during the present predetermined time interval (action 367). Action 367 may be performed by software code 110/210, executed by hardware processor 104, and using backward cumulative error module 216 and survey distribution control module 218.
For example, next survey sampling rate 128/228 may be further determined as the sum of the value determined in action 364 with backward cumulative error value 226, or the sum of the value determined in action 365 with backward cumulative error value 226, or any value determined in action 366 with backward cumulative error value 226. In some implementations it may be advantageous or desirable to apply a weighting factor to backward cumulative error value 226 in such a sum. For instance, where NSSRF is the further determined value of next survey sampling rate 128/228, NSSRI is the initial value of next survey sampling rate 128/228 determined in action 364, PE is present error value 222, and FCE is forward cumulative error value 224, as noted above, and where BCE is backward cumulative error value 226, NSSRF may be expressed as:
NSSRF=NSSRI+γBCE,
where γ is a tunable weighting factor, or as:
NSSRF=NSSRI+αPE+γBCE,
where each of α and γ is a tunable weighting factor, or as:
NSSRF=NSSRI+βFE+γBCE,
where each of β and γ is a tunable weighting factor, or as:
NSSRF=NSSRI+αPC+βFE+γBCE,
where each of α, β, and γ is a tunable weighting factor.
It is emphasized that flowchart 360 may conclude with action 364 in
Moreover, actions 365, 366, and 367 may be performed in any order, or may be performed substantially concurrently. It is also noted that, in some implementations, hardware processor 104 may execute software code 110/210 to perform actions 361, 362, and 364, or 361, 362, 363, and 364 (hereinafter “actions 361-364”), and/or actions 361, 362, and 364 or actions 361-364 in combination with any one or more of actions 365, 366, and 367, in an automated process from which human involvement is omitted.
S
Control(p,d,t)=SQ-Control=SExposed(p,d,t)/SQ-Exposed≈A(p,d,t)/AT
for every combination of p, d, and t during the ad campaign, as desired.
Thus, the present application discloses systems and methods for synchronizing survey collection with ad distribution during an ad campaign that overcome the drawbacks and deficiencies in the conventional art. In contrast to conventional solutions for distributing and collecting surveys during an ad campaign, the systems and methods disclosed herein advantageously enable real-time responsiveness over the course of an entire campaign. That is to say, by leveraging audience parameters that may include demographics, DMP data, and content data such as a show name or episode number of content being consumed by an audience member, and doing so in real-time, the present concepts ensure that the right ad and/or right survey is reliably served to the right audience member at the right time. Conventional solutions are unable to access and utilize comparable audience parameters in real-time. Those conventional solutions implement an essentially random sampling from the overall audience. Consequently, in the conventional art, there is often a significant mismatch between the audience group being surveyed, and an optimal survey group that is truly representative of the target audience.
From the above description it is manifest that various techniques can be used for implementing the concepts described in the present application without departing from the scope of those concepts. Moreover, while the concepts have been described with specific reference to certain implementations, a person of ordinary skill in the art would recognize that changes can be made in form and detail without departing from the scope of those concepts. As such, the described implementations are to be considered in all respects as illustrative and not restrictive. It should also be understood that the present application is not limited to the particular implementations described herein, but many rearrangements, modifications, and substitutions are possible without departing from the scope of the present disclosure.
