Patent Application
20220058685
Service sectors deal with payers and providers constantly, and payers constantly evaluate providers' performances. One of the biggest and sometimes most important factor is value. In order to evaluate value—performance as it relates to cost, there are various models or measures that aim to provide a score, a value index score, to assist decision makers of payers and providers to better assess performances to meet expectations.
In the advertising technology industry, at a very basic level, a value index may identifying set of features, for example, such as quality, demand, performance, competitiveness, and price for a deal (Agreement between Buyer and Seller). A value index score may be generated to guide the campaign managers and programmatic specialists of the platform to understand whether the deal they are selecting from a particular publisher, advertiser, or a demand-side platform (DSP) combination is optimal or not.
Existing technology applies a process that involves identifying a set of features on the deal performance and provides a prediction in a form of a score based on historical data using Machine Learning (ML) algorithms.
Unfortunately, the technical problem with such approach includes its failure to identify the value of a set of a certain features as a combination, rather than just the value of individual features. The technical problem with such failure is a failure to account for a data structure (and data storage for data to be processed) and a corresponding algorithm to consume or process such combination in the data structure. Moreover, there is no algorithmic expression that accounts for different weights of these combination in the final value index score.
Embodiments of the invention provide a technical solution by generating a value index score based on aggregation from a combination of features as a unit. In one embodiment, instead of generating a value index score based on a collection of features with each feature being a discrete parameter, aspects of the invention generate the value index score while accounting for weights of a combination of features as a unit. Furthermore, embodiments of the invention generate a weight value for each feature and that the weight, not only will it be a factor in the calculation, but also be modifiable in response to other factors of the features.
Moreover, embodiments of the invention further evaluate and aggregate data such as historical performance data from demand side platforms and unity marketplace, which includes data from supply side platforms (SSPs). Such evaluation may adjust the weight to further improve accuracy and aid demand side and supply side for better predictions.
Persons of ordinary skill in the art may appreciate that elements in the figures are illustrated for simplicity and clarity so not all connections and options have been shown to avoid obscuring the inventive aspects. For example, common but well-understood elements that are useful or necessary in a commercially feasible embodiment may often not be depicted in order to facilitate a less obstructed view of these various embodiments of the present disclosure. It will be further appreciated that certain actions and/or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required. It will also be understood that the terms and expressions used herein may be defined with respect to their corresponding respective areas of inquiry and study except where specific meanings have otherwise been set forth herein.
Embodiments of the present invention may now be described more fully with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific exemplary embodiments by which the invention may be practiced. These illustrations and exemplary embodiments may be presented with the understanding that the present disclosure is an exemplification of the principles of one or more inventions and may not be intended to limit any one of the inventions to the embodiments illustrated. The invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Among other things, the present invention may be embodied as methods, systems, computer readable media, apparatuses, or devices. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. The following detailed description may, therefore, not to be taken in a limiting sense.
Referring now to
In one example, the system 100 provides a model for determining value index score of a certain deal or transaction that may be received from a data store 102 or from demand side platform (DSP) data 104. In one example, the system 100 may receive deals may have data 122 ready for scoring. The value index score may provide an indication of a value of a deal. In one embodiment, the system 100 provides one or more sequence of steps in determining the score.
In one example, the system 100 may classify the one or more deals by applying a logistic regression analysis. For example, in any deal, a payer may wish to determine, after receiving quotes from providers for an advertising campaign, how to choose a quote. This is a binary or dichotomous determination, but affected a plurality of factors or features of the quote. Similarly, the provider, in improving its future chances of landing a successful deal with a payer, may wish to know how to close the deal. As such, the decision to accept or reject the deal may be treated as a binary or dichotomous (e.g., Yes or No). In one embodiment, the logistic regression analysis may describe data and explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables.
In one embodiment, the logistic regression analysis may be applied to classify the one or more deals that are received. In one example, each of the deals may include parameters, metrics, etc., that comprise each of the deals.
Referring now to
The data structure 202 further may include a data field 210 for each feature of the deal. For example, the data field 210 may include a name of the feature. In another embodiment the data field 210 may include subfields such as fields 212, 214, and 216. In one example, the field 212 may include one or more parameters of the feature, such as data type, data restrictions, data duration, data source, metrics of the data, etc. In another embodiment, the field 214 may include a weight assigned or calculated by the system 100. In another example, the field 216 may include any historical weight that has previously assigned or calculated for a particular feature.
In another embodiment, the field 210 may further be applicable for an individual metric of a feature. As previously discussed, prior technology fails to account for granular significance at feature or event metrics level. There was also no common data structure to account for such significance. Aspects of the invention provide the common data structure, such as the data structure 202, to store and present the weighted data of each feature or metric such that intelligent analysis may be performed therewith.
In one embodiment, table 1 below illustrates one example of contents or details of DSP performance data:
In one embodiment, the data structure 202 may store the above data as part of a deal when the performance data is one of the many features to be evaluated before providing the value index score of a deal.
As another example, table 2 illustrates some of the contents or details of unity marketplace data:
In one embodiment, the data structure 202 may store the above data as part of a deal when the unity marketplace data is one of the many features to be evaluated before providing the value index score of a deal.
It is to be understood that other data may be stored or included without departing from the scope or spirit of embodiments of the invention. For example, as illustrated in Tables 1 and 2, columns “Available,” “Pending,” and “Future” provide potential growth of the data contents that may become available and therefore may be part of a weighted consideration in determining the score.
Referring back to
In one embodiment, as part of the training, the system 100 may feed the data to a linear regression analysis at 114, a gradient boosting analysis at 116, and a random forest analysis at 118 to refine the data as part of the classification or determining the weight. It is also contemplated that other analysis mechanisms or algorithms may be employed without departing from the scope or spirit of embodiments of the invention. Moreover, as the data is fed into these various analysis, the system 100 may also perform cross data validation and fine tuning or hyper tuning of parameters within the data.
In one embodiment, the system 100 may stack the linear regression, the gradient boosting, and the random forest to further enhance the value of the score.
In one embodiment, the linear regression model or analysis may aim to determine a function that draws a linear relationship between a set of input features and the value a payer or a provider would like to predict. In one embodiment, this generated function may serve as a model for future analysis.
In another embodiment, the random forest analysis may use decision trees as the starting point. However, decision trees tend to and are very prone to overfitting. In one embodiment, in order to achieve higher accuracy, the system 100 may modify the random forest analysis to create a large number of them based on bagging. In one embodiment, bagging may include:
Generate n different bootstrap training sample;
Train algorithm on each bootstrapped sample separately; and
Average the predictions at the end.
In one embodiment, the system 100 may resample the data over and over and for each sample train a new classifier. Different classifiers may over-fit (e.g., reducing variance) the data in a different way, and through voting those differences may be averaged out.
In another embodiment, an additional gradient boosting analysis (GB) at 120 may be used as a boosting method, which may build on weak classifiers. In one embodiment, boosting may include the following:
Draw a random subset of training samples d1 without replacement from the training set D to train a weak learner C1;
Draw second random training subset d2 without replacement from the training set and add 50 percent of the samples that were previously falsely classified/misclassified to train a weak learner C2;
Find the training samples d3 in the training set D on which C1 and C2 disagree to train a third weak learner C3; and
Combine all the weak learners via majority voting.
In another embodiment, the GB 120 may further include historic models to be used to improve the value index score. In one example, as discussed above, some of the training models may be treated as historic models as time passes. In one example, upon the generation of each run or generation of the value index, the ten most recent runs may be used or presented so that the most up-to-date and most accurate value index scores are presented to the user. These more up-to-date scores enable the value index to better learn and adjust its own models as newer data comes in.
In one embodiment, the system 100 may add a classifier at a time, so that the next classifier is trained to improve the already trained ensemble. In another embodiment, for random forest analysis, the system 100 may, each iteration, use the classifier to be trained independently from the rest.
With the above steps and process, in one embodiment, 70% of the data may be for training and 30% of the data may be used for testing the model and derive the final output.
In one embodiment, the system 100 may employ machine learning (ML) or other artificial intelligence (AI) algorithm or models to perform one or more of the analysis. In one embodiment, the system 100 may designate processes under “A” as part of a training stage while “B” as a prediction stage of the processing.
In another embodiment, the system 100, through the recursive and AI processing, the system may export models trained 124 before generating a score at 126.
In one embodiment, as the system 100 receives data from deals and through the analysis methods discussed above, the system 100 may calculate and determine a weight for each of the features in a given deal. In addition, each of the weights may be further modified and updated based on the various models to increase the accuracy of the prediction.
In one embodiment, the system 100 may assign weights to features or metrics of a given deal.
For example, the system 100 includes, based on analysis of the data, a table showing a column 302 for one or more attributes or metrics of a feature or a feature and a column 304 for a weight assigned or calculated for the corresponding attribute or metric or feature.
In one embodiment, the weights may be determined by the logistic regression analysis to classify from machine learning libraries or other AI algorithms. In another embodiment, the GUI 300 may further include a table 306 showing an accuracy rating or value and various controls based on the weights of the attributes after the application of the logistic regression. Moreover, the GUI 300 may also provide a graph 308 showing a distribution of significance level of the attributes. In one embodiment, the graph 308 may be dynamic and correspond to the weights such that as the weights are modified, the distribution of the graph 308 may be modified accordingly.
Referring now to
In another embodiment, in using the data structure 202, the system 100 may further identify proxies or metrics. For example, the following illustrates metrics or features that may be a representation of another element of a deal:
Quality—Viewability;
Performance—(click-through rate) CTR or (pay-per-click) PCC;
Demand—Bid Rate (“BR”);
Competivity—Win Rate (“WR”);
Price—Clearing Price (“CR”)
In one embodiment, the system 100 may provide
In another embodiment, an exemplary prediction function of a value index score may be:
VI=f(V,CTR,BR,WR,CR)
VI=Σ(α1*Quality)+(α2*Performance)+(α3*Competitivity)+(α4*Demand)+(α5*Price)
In another embodiment,
In one embodiment,
In summary,
In one embodiment, using the 4 deals illustrated from
For example, as illustrated in
Patterns:
Turner deals are in the high 8's;
Viacom deals are in the high 6's.
In one embodiment, the message or content of the notification pane 420 may be generated by the system 100. In another embodiment, the message or content of the notification pane 420 may be edited, generated, or modified by a system administrator or an analysis. In another embodiment, the message or content of the notification pane 420 may be composed jointly by the system 100 and a user.
The view 418 further may provide the rules pane 422 showing an exemplary syntax of the rules that produce the message or content in the notification pane 420.
In a further embodiment, in
Evaluation:
Value index range is too wide;
Rule is too generic.
In this example, the view 430 may show the rules previously displayed in the rules pane 422 to a new area 432.
As such, in
In this example, the rules pane 442 may be interactive as well where the user may enter parameters or values for each of the metrics or attributes selected in the view 436. In one embodiment, the system 100 may enable substantially simultaneous execution or evaluation of the rules as the user enters the values for the attributes or metrics in the rules pane 442. In one embodiment, the rules 442 may provide cues (either visually or audibly) to the user if the rules' syntax is not satisfied. The view 440 may then, after the execution of a customized rules provided by the user, provide real-time or simultaneous real-time scores for the deal.
In one embodiment, the value index scores provided on the GUIs may be ranked. In another example, the ranked scores may list the value indexes from the highest rating to the lowest rating.
Moreover,
In
The GUI 502 further includes a table 510 providing headers 512 showing additional details of the deal, such as “format,” “deal ID,”, etc.
Referring now to
Upon selecting the “VALUE INDEX” function 524 on the menu 522, the GUI 502 may provide another menu 526 showing additional sub-functions under the “VALUE INDEX” function. The user may select “COMPETIVITY” function 528 from the menu 526 as one of the features that the user may wish to review. In one example, the functions in the menu 526 may be treated as one or more features of a given deal. Again, it is to be understood that other features may be added without departing from the scope or spirit of embodiments of the invention.
Upon selecting the “COMPETIVITY” function 528, the GUI 502 may provide a view 530 to interact with the user. In this example, the view 530 may provide a graph or a graphical representation showing in simultaneously or substantially simultaneously real-time changes to the feature “COMPETIVITY”. Similarly, the view 530 may provide text confirmation or value of the changes. In one embodiment, the changes may represent changes in the score for the feature “COMPETIVITY.” Once the user is satisfied with the changes, the user may select a “APPLY” button 538 to request the system 100 to execute the changes or a “CLEAR ALL” button 536 to cancel the changes.
Referring now to
In one embodiment, the value index scores provided on the GUIs may be ranked or sorted. In another example, the ranked scores may list the value indexes from the highest rating to the lowest rating. As such, the results in
It is to be understood that the user may add other conditions by repeat some of the illustrated operations above without departing from the scope or spirit of embodiments of the invention.
In one embodiment, a portable computing device 801 may be a mobile device 112 that operates using a portable power source 855 such as a battery. The portable computing device 801 may also have a display 802 which may or may not be a touch sensitive display. More specifically, the display 802 may have a capacitance sensor, for example, that may be used to provide input data to the portable computing device 801. In other embodiments, an input pad 804 such as arrows, scroll wheels, keyboards, etc., may be used to provide inputs to the portable computing device 801. In addition, the portable computing device 801 may have a microphone 806 which may accept and store verbal data, a camera 808 to accept images and a speaker 810 to communicate sounds.
The portable computing device 801 may be able to communicate with a computing device 841 or a plurality of computing devices 841 that make up a cloud of computing devices 811. The portable computing device 801 may be able to communicate in a variety of ways. In some embodiments, the communication may be wired such as through an Ethernet cable, a USB cable or RJ6 cable. In other embodiments, the communication may be wireless such as through Wi-Fi (802.11 standard), Bluetooth, cellular communication or near field communication devices. The communication may be direct to the computing device 841 or may be through a communication network 102 such as cellular service, through the Internet, through a private network, through Bluetooth, etc.
As a result of the system, better information may be provided to a user at a point of sale. The information may be user specific and may be required to be over a threshold of relevance. As a result, users may make better informed decisions. The system is more than just speeding a process but uses a computing system to achieve a better outcome.
The physical elements that make up the remote computing device 841 may be further illustrated in
The database 1025 may be stored in the memory 1010 or 1015 or may be separate. The database 1025 may also be part of a cloud of computing device 841 and may be stored in a distributed manner across a plurality of computing devices 841. There also may be an input/output bus 1020 that shuttles data to and from the various user input devices such as the microphone 806, the camera 808, the inputs such as the input pad 804, the display 802, and the speakers 810, etc. The input/output bus 1020 also may control of communicating with the networks, either through wireless or wired devices. In some embodiments, the application may be on the local computing device 801 and in other embodiments, the application may be remote 841. Of course, this is just one embodiment of the server 841 and the number and types of portable computing devices 841 is limited only by the imagination.
The user devices, computers and servers described herein may be general purpose computers that may have, among other elements, a microprocessor (such as from the Intel Corporation, AMD, ARM, Qualcomm, or MediaTek); volatile and non-volatile memory; one or more mass storage devices (i.e., a hard drive); various user input devices, such as a mouse, a keyboard, or a microphone; and a video display system. The user devices, computers and servers described herein may be running on any one of many operating systems including, but not limited to WINDOWS, UNIX, LINUX, MAC OS, iOS, Android, or Windows (XP, VISTA, etc.). It is contemplated, however, that any suitable operating system may be used for the present invention. The servers may be a cluster of web servers, which may each be LINUX based and supported by a load balancer that decides which of the cluster of web servers should process a request based upon the current request-load of the available server(s).
The user devices, computers and servers described herein may communicate via networks, including the Internet, WAN, LAN, Wi-Fi, other computer networks (now known or invented in the future), and/or any combination of the foregoing. It should be understood by those of ordinary skill in the art having the present specification, drawings, and claims before them that networks may connect the various components over any combination of wired and wireless conduits, including copper, fiber optic, microwaves, and other forms of radio frequency, electrical and/or optical communication techniques. It should also be understood that any network may be connected to any other network in a different manner. The interconnections between computers and servers in system are examples. Any device described herein may communicate with any other device via one or more networks.
The example embodiments may include additional devices and networks beyond those shown. Further, the functionality described as being performed by one device may be distributed and performed by two or more devices. Multiple devices may also be combined into a single device, which may perform the functionality of the combined devices.
The various participants and elements described herein may operate one or more computer apparatuses to facilitate the functions described herein. Any of the elements in the above-described Figures, including any servers, user devices, or databases, may use any suitable number of subsystems to facilitate the functions described herein.
Any of the software components or functions described in this application, may be implemented as software code or computer readable instructions that may be executed by at least one processor using any suitable computer language such as, for example, Java, C++, or Perl using, for example, conventional or object-oriented techniques.
The software code may be stored as a series of instructions or commands on a non-transitory computer readable medium, such as a random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a CD-ROM. Any such computer readable medium may reside on or within a single computational apparatus and may be present on or within different computational apparatuses within a system or network.
It may be understood that the present invention as described above may be implemented in the form of control logic using computer software in a modular or integrated manner. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art may know and appreciate other ways and/or methods to implement the present invention using hardware, software, or a combination of hardware and software.
The above description is illustrative and is not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the pending claims along with their full scope or equivalents.
One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the invention. A recitation of “a”, “an” or “the” is intended to mean “one or more” unless specifically indicated to the contrary. Recitation of “and/or” is intended to represent the most inclusive sense of the term unless specifically indicated to the contrary.
One or more of the elements of the present system may be claimed as means for accomplishing a particular function. Where such means-plus-function elements are used to describe certain elements of a claimed system it will be understood by those of ordinary skill in the art having the present specification, figures and claims before them, that the corresponding structure is a general purpose computer, processor, or microprocessor (as the case may be) programmed to perform the particularly recited function using functionality found in any general purpose computer without special programming and/or by implementing one or more algorithms to achieve the recited functionality. As would be understood by those of ordinary skill in the art that algorithm may be expressed within this disclosure as a mathematical formula, a flow chart, a narrative, and/or in any other manner that provides sufficient structure for those of ordinary skill in the art to implement the recited process and its equivalents.
While the present disclosure may be embodied in many different forms, the drawings and discussion are presented with the understanding that the present disclosure is an exemplification of the principles of one or more inventions and is not intended to limit any one of the inventions to the embodiments illustrated.
The present disclosure provides a solution to the long-felt need described above. In particular, the systems and methods described herein may be configured for improving verification and discovery of merchants or stores that do not accept non-cash payment devices or that do accept non-cash payments devices but differentiate them between local/national issued ones versus foreign issued ones. Further advantages and modifications of the above described system and method will readily occur to those skilled in the art. The disclosure, in its broader aspects, is therefore not limited to the specific details, representative system and methods, and illustrative examples shown and described above. Various modifications and variations can be made to the above specification without departing from the scope or spirit of the present disclosure, and it is intended that the present disclosure covers all such modifications and variations provided they come within the scope of the following claims and their equivalents.
This application is a continuation-in-part of the U.S. nonprovisional application Ser. No. 16/221,514, filed on Dec. 15, 2018 and claims priority to PCT international application dated PCT/US19/066604, filed on Dec. 16, 2019, whose disclosures are incorporated by reference in its entirety herein.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16221514 | Dec 2018 | US |
| Child | 17377364 | US |
