RANKING-BASED METHOD AND SYSTEM FOR EVALUATING CUSTOMER PREDICATION MODELS

Abstract

A method and system perform ranking-based evaluations for regression models that are often appropriate for marketing tasks and are more robust to outliers than traditional residual-based performance measures. The output provided by the method and system provides visualization that can offer insights about local model performance and outliers. Several models can be compared to each other to identify the “best” model and, therefore, the “best” model data for the particular marketing task.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects and advantages will be better understood from the following detailed description of a preferred embodiment of the invention with reference to the drawings, in which:

FIG. 1 is a flow diagram of the steps of the ranking-based method according to the present invention;

FIG. 2 is a visualization of a percent of correctly ranked pairs involving a particular prediction;

FIG. 3 is a visualization of the Area Under the Curve (AUC) values for the model being evaluated; and

FIG. 4 is the approximate lift curve of the cumulative rank.

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION

Referring now to the drawings, and more particularly to FIG. 1, there is shown a flow diagram of the steps of the invention method for performing ranking-based evaluation of the customer prediction models.

The method described here uses several relationships to evaluate the models and related model data. These relationships utilize the following variables:

• • n is a size of a set of customers with known value of y,
  • x is a matrix of customer properties of length n
  • y is a vector of observations of the response (revenue) of length n
  • x_i is a matrix of are customer properties of length n after sorting by y
  • y_i is a vector of observations of the response (revenue) of length n
  • ŷ is a predicted response of the model y=m(x),
  • s_i is the predicted rank for customer of said plurality of customers T is a number of rank order switches, and/or potential customers,
  • i is the integer from between 1 and n indexing sorted customers, also corresponds to the order of observed value y
  • j is an integer indexing customers
  • R is a weighted sum of order switches,
  • {circumflex over (τ)} is a probability of ranking switch,
  • {circumflex over (p)} is a ranking correlation,
  • Ci is a number of observations that are concordant with observation i,
  • 1-αis the confidence interval.

A database 101 contains customer data (x, y) where x are different properties of customers and y is the quantity of interest (e.g., the revenue generated by the customer). It should be noted that y is a vector and x is a matrix of length n and width equal to the number of different customer properties (also called features). The database also contains k models (m₁, . . . , m_k) that for each customer can predict the quality of interest given x_i where ŷ_i=m₁(x).

In function block 110, the customer data (x, y) are sorted in increasing order of y and stored in the database 101. The resulting sorted customer data (x_i, y_i) has the property y_i>y_j if i=j for i=1, . . . , n customers.

In function block 111, all models are applied to the sorted customer properties x to obtain predictions ŷ_i=m₁(x_i) for all customers i from all models l. In function block 112, calculations are made for each model l, the respective predicted rank r_l of the predictions ŷ₁. Note that each r_l is a vector of length n and that the order of the entries in vector r still reflects the order of the true value y. For example, if there are three customers with ordered revenue values $3, $15, $57 for which the model l prediction revenue values: $5.00, $100 and $0, the predicted ranking would be r=2, 3, and 1. Formally, r_i is the rank of customer and/or potential customer i in this order:

s _i =|{j≦n|ŷ _i ≦ŷ _j}|

The invention considers two ranking-based evaluation measures and their interpretations (e.g., ranking order entries in model ranking table, etc.). Function block 113 calculates for each model the number of ranking order switches:

T=Σ _i≦j1{s_i>s_j}  (2)

and the weighted sum of order switches:

R=Σ _i<j(j−i)1(s_i>s_j)  (3)

The first measure simply counts how many of the pairs in the test data are ordered incorrectly by the model m(x). The second measure also considers these incorrect orderings, but weighs them by the difference in their model ranks, that is, a measure of the magnitude of error being committed. The results of each of these steps are stored electronically in the system database 101.

In function block 114, the ranks are transformed using rescaling equations to put them into the range [−1, 1], where 1 corresponds to perfect model performance (T,R=0) and −1 corresponds to making all possible errors, thus attaining perfect reverse ranking. It is easy to verify that max(T)=n(n−1)=2, max(R)=n(n−1)(n+1)=6. The resealing equations are:

$\begin{array}{cc}\hat{\tau }=1-\frac{4T}{n\left(n-1\right)}& \left(4\right)\\ \hat{\rho }=1-\frac{12R}{n\left(n-1\right)\left(n+1\right)}& \left(5\right)\end{array}$

These values are similar to Kendall's τ which measures the strength of the relationship between two variables and Spearman's rank correlation. The moments of {circumflex over (τ)} and {circumflex over (p)} under the relevant null assumptions (τ=0 and p=0, respectively) are calculated and a normal approximation gives a hypothesis testing methodology for the assumption of no correlation. For residual based measures, it is typically not possible to build confidence intervals without parametric assumptions and/or variance estimation. The non-parametric nature of {circumflex over (τ)} allows a general expression for its variance to be written as:

$\mathrm{Var}\left(\hat{\tau }\right)=\frac{8}{n\left(n-1\right)}\left({\pi }_c\left(1-{\pi }_c\right)+2\left(n-2\right)\left({\pi }_{\mathrm{cc}}-{\pi }_c^2\right)\right),$

where π_c=E({circumflex over (τ)})=½+½τ and π_cc are two properties of the ranking function. This is then replaced with the sample means to obtain:

$\begin{array}{cc}\widehat{\mathrm{Var}}\left(\hat{\tau }\right)={\left(\frac{2}{n\left(-1\right)}\right)}^2·2·\left(2\sum _iC_i^2-\sum _iC_i-\frac{\left(2n-3\right)}{n\left(n-1\right)}·{\left(\sum _iC_i\right)}^2\right)\mathrm{where}C_i=\sum _{j<i}1\left\{y_i>y_j\right\}+\sum _{j>i}1\left\{y_i<y_j\right\}& \left(6\right)\end{array}$

is the number of observations that are “concordant” with observation i, that is, that their ranking relative to i in the model data agrees with the ranking by model scores (as plotted in FIG. 3). Function block 114 also calculates the confidence intervals for the values of evaluation measure given its sample value (the non-null case), as they represent the uncertainty in the model evaluation based on a single test set. Since {circumflex over (τ)} is asymptomatically normal, a 1-α confidence interval for {circumflex over (τ)} is:

In function block 115, three graphical representations of the ordered switches are constructed: (1) percent of correctly ranked pairs involving a particular prediction, (2) AUC as a function of cutoff position, and (3) Lift-curve of the cumulative rank. Examples of these graphical representations are shown respectively in FIGS. 2, 3, and 4, respectively.

Starting from the largest model prediction m(x_n), function block 115 calculates in decreasing order for each observation X_i the percentage of correctly ranked pairs (y_i, y_j) over all j≠i. The percentage of correct pairings as a function of the inverse rank is shown in FIG. 2. The area above the curve is the sum of the percent incorrectly ranked pairs, which is equal to 2T/n(n−1). Therefore, the area under the curve equals {tilde over (τ)}. The dashed line corresponds to a locally optimal performance. A perfect model would show a constant performance of 100% correctly ranked pairs. But given that the model is not perfect and makes predictions that are sometimes too large or too small, even a perfect prediction for a particular observation with m(x_i)=y_i will have a number of inversely ranked pairs due to errors of the other predictions. The upper limit of the performance for a given prediction, keeping everything else constant, is therefore not 100% but determined by the model performance of predictions around it. The interpretation of the locally optimal performance is the highest achievable percentage of correctly ranked pairs if m(x_i) could be placed arbitrarily, given all other model predictions.

The performance in a particular region of the graph is characterized by two properties of the plot, 1) the distance of the local optimum from the 100% line, and 2) the distance of the actual performance from the local optimum. A particular region with a performance that on average remains very close to the local optimum has a nearly perfect ranking and is only disturbed by bad predictions that were either larger or smaller than the predictions of the region.

For FIG. 3, the original regression results are transformed into n−1 classification results, where the observations are discretized into a binary class variable c⁽ⁱ⁾(y_j)=1 iff y_j≧y_i for all possible cutoffs 1<i<n.

For each classification c⁽ⁱ⁾ the model performance is evaluated using the area under the ROC curve (AUC,) (FIG. 3). The probabilistic interpretation of the AUC is the probability that a pair of observations with opposite class labels is ranked correctly. Since AUCi only considers pairs with different class labels under cutoff i, the number of pairs used to calculate AUCi is equal to i·(n−i) and the number of incorrectly ranked pairs under this cutoff is therefore:

(1-AUCi)i·(n−i).

In function block 115 of FIG. 1, the total number of times a pair of observations will be assigned opposite class labels across all cutoffs is equal to the rank difference; i.e., a neighboring pair (k; k+1) receives opposite class labels only if the cutoff is equal to k whereas the extreme pair (1, n) will have opposite classes for all n−1 cutoffs. Given the definition in Equation (3) this implies that:

$\sum _i\left(1-{\mathrm{AUC}}_i\right)·i·\left(n-1\right)=R$

Since each AUCi is rescaled by a different factor i·(n−i), a graph of the AUCi as a function of the cutoff i would not have an area equal to {tilde over (p)}. In order to achieve a direct correspondence with {tilde over (p)}, i·(n−i) units are allocated to AUCi by resealing the x-axis accordingly. FIG. 3 shows such a transformation of the x-axis and has an area under the curve of {tilde over (p)}.

The plot in FIG. 4 is very close in spirit to a lift curve. After sorting the model predictions in decreasing order, the cumulative inverse rank of:

$p_i=\sum _{j=1}^i\left(n-s_j+1\right)$

is plotted for increasing cutoffs i in percent. Using the inverse rank emphasizes the model performance on the largest predictions that is shown in the bottom left of the graph. The model performance is bounded above by the optimal ranks

$p_i=\sum _{j=1}^i\left(n-j+1\right)$

and below by the cumulative worst (inverse) ranking

$w_i=\sum _{j=1}^ij.$

The area under the model curve is given as

$\sum _{j=1}^n\left(n-i\right)\left(n-s_i\right)$

shown to be equal to

$n^3-n^2-n-R+\sum _{i=1}^ni^2.$

Once all models have been evaluated through function blocks 111-115, function block 116 stores the results and graphical representations in the system database 101. This information is then displayed for the analyst at display block 117. This display can be provided in any format (e.g., printed report, electronic display on a computer monitor, etc.) specified by the user. The analyst then evaluates the model based on the provided information in function block 118 and selects the model that can provide the suggested “best” list of potential customers for targeted sales and marketing efforts. The database is updated with these recommendations at function block 119 and a final report containing the optimal model and customer rankings is provided as the output 102 to be used by company personnel for targeted sales and marketing efforts.

While the invention has been described in terms of its preferred embodiment, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.

Claims

1. A computer-implemented ranking-based method for evaluating regression models by obtaining predictions y from one or more models to be evaluated on a test set of customers for which the true value y of the quantity of interest has been observed comprising the steps of: storing the test set resides in an electronic database sorted in increasing order such that “large” corresponds to one of a plurality of customers and/or potential customers with largest perceived spending budget, and “small” corresponds to one of a plurality of customers and/or potential customers with smallest perceived spending budget;applying all models to customer data from the test set to obtain predictions for the spending for each model and customer;converting the predictions into ranks and storing for each model in one or more electronic databases as a model ranking table, a number of ranking order switches relative to ranking of the observed customer spending being calculated for each model and ranking order switches being defined as those changes in ranking position of the prediction relative to the order of the true observations y;calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches;transforming the number of ranking switches and weighted sum of ranking order switches into a range of [−1, 1] wherein −1 corresponds to making all possible errors (inverse ranking) and 1 corresponds to a perfect model wherein said number of ranking switches has been transformed to represent a difference between a probability that the ranking of two customers and/or potential customers are in the same order versus the probability that two of the customers and/or potential customers are in different orders from the originally obtained rank;normalizing transformed measures of order switches into a range of [0,1] wherein 1 corresponds to perfect ranking and 0 corresponds inverse ranking;calculating a variance of measures of order switches and determining confidence intervals for each ranking measure;updating the model performance table with the ranking measures and their confidence intervals; andoutputting the ranking measures and confidence levels as well as graphical representations thereof to a domain expert, who will choose based on this information the best model m* of the one or more models evaluated, and storing the predictions of ŷ=m*(x) in the optimal prediction table.

2. The computer-implemented ranking-based method recited in claim 1, wherein the step of calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches comprises the steps of: calculating using computing resources a number of ranking order switches wherein ranking order switches are those changes in ranking order position of said plurality of customers and/or potential customers assuming said ranking obtained in said obtaining step was erroneous, andcalculating using computing resources a weighted sum of ranking order switches wherein said weighted sum of ranking order switches provides a measure of the magnitude of erroneous ranking.

3. The computer-implemented ranking-based method of claim 2, wherein said calculating using computing resources the number of ranking order switches step comprises a relationship: T=Σi<j1{si>sj}

4. The computer-implemented ranking based method recited in claim 2, wherein said calculating using computing resources a weighted sum of order switches step comprising a relationship: R=Σi<j(j−i)1{si>sj}

5. The computer-implemented ranking-based method recited in claim 1, wherein the step of calculating a variance of measures of order switches and determining confidence intervals for each ranking measure comprises the steps of: calculating using computing resources a variance that said measures of said ranking order switches;calculating using computing resources a confidence interval for each measure of said ranking order switches; andconstructing using computing resources graphical representations of said ranking order switches.

6. The computer-implemented ranking-based method recited in claim 5, wherein said calculating using computing resources a variance implements a relationship:

7. The computer-implemented ranking-based method recited in claim 1, wherein the step of outputting the ranking measures and confidence levels comprises the steps of: providing an output that presents an updated ranking of said plurality of customers and/or potential customers that corresponds to ranking from model obtained in said obtaining step with highest confidence interval; and/orproviding an output of a model recommendation that corresponds to a best model of said one or more models evaluated.

8. The computer-implemented ranking-based method recited in claim 1, wherein the step of transforming said number of ranking switches and said weighting sum of order switches step implements a set of relationships:

9. The computer-implemented ranking-based method recited in claim 1, wherein the step of outputting provides an updated ranking of a plurality of customers and/or potential customers that corresponds to ranking from a model obtained in said obtaining step with highest confidence interval.

10. A computer system for implementing a ranking-based method for evaluating regression models by obtaining predictions ŷ from one or more models to be evaluated on a test set of customers for which the true value y of the quantity of interest has been observed comprising: a database storing the test set resides in an electronic database sorted in increasing order such that “large” corresponds to one of a plurality of customers and/or potential customers with largest perceived spending budget, and “small” corresponds to one of a plurality of customers and/or potential customers with smallest perceived spending budget;means for applying all models to customer data from the test set to obtain predictions for the spending for each model and customer and onverting the predictions into ranks and storing for each model in one or more electronic databases as a model ranking table, a number of ranking order switches relative to ranking of the observed customer spending being calculated for each model and ranking order switches being defined as those changes in ranking position of the prediction relative to the order of the true observations y;calculating means for calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches;means for transforming the number of ranking switches and weighted sum of ranking order switches into a range of [−1, 1] wherein −1 corresponds to making all possible errors (inverse ranking) and 1 corresponds to a perfect model wherein said number of ranking switches has been transformed to represent a difference between a probability that the ranking of two customers and/or potential customers are in the same order versus the probability that two of the customers and/or potential customers are in different orders from the originally obtained rank;said calculating means normalizing transformed measures of order switches into a range of [0,1] wherein 1 corresponds to perfect ranking and 0 corresponds inverse ranking and calculating a variance of measures of order switches and determining confidence intervals for each ranking measure;means for updating the model performance table with the ranking measures and their confidence intervals; andoutput means for outputting the ranking measures and confidence levels as well as graphical representations thereof to a domain expert, who will choose based on this information the best model m* of the one or more models evaluated, and storing the predictions of ŷ=m*(x) in the optimal prediction table.

11. The computer system recited in claim 10, wherein said calculating means calculates a number of ranking order switches wherein ranking order switches are those changes in ranking order position of said plurality of customers and/or potential customers assuming said ranking obtained in said obtaining step was erroneous, and a weighted sum of ranking order switches wherein said weighted sum of ranking order switches provides a measure of the magnitude of erroneous ranking.

12. The computer system recited in claim 11, wherein said calculating means implements a relationship: T=Σi<j1{si>sj}

13. The computer system recited in claim 11, wherein said calculating means implements a relationship: R=Σi>j(j−i)1{si>sj}

14. The computer system recited in claim 10, wherein said calculating means calculates a variance that said measures of said ranking order switches, a confidence interval for each measure of said ranking order switches, and graphical representations of said ranking order switches.

15. The computer system recited in claim 14, wherein said calculating means implements a relationship:

16. The computer system recited in claim 10, wherein said output means provides an output that presents an updated ranking of said plurality of customers and/or potential customers that corresponds to ranking from model obtained in said obtaining step with highest confidence interval and/or provides an output of a model recommendation that corresponds to a best model of said one or more models evaluated.

17. The computer system recited in claim 10, wherein said transforming means implements a set of relationships:

18. A computer readable media containing code which implements a ranking-based method for evaluating regression models by obtaining predictions ŷ from one or more models to be evaluated on a test set of customers for which the true value y of the quantity of interest has been observed, the method comprising the steps of: storing the test set resides in an electronic database sorted in increasing order such that “large” corresponds to one of a plurality of customers and/or potential customers with largest perceived spending budget, and “small” corresponds to one of a plurality of customers and/or potential customers with smallest perceived spending budget;applying all models to customer data from the test set to obtain predictions for the spending for each model and customer;converting the predictions into ranks and storing for each model in one or more electronic databases as a model ranking table, a number of ranking order switches relative to ranking of the observed customer spending being calculated for each model and ranking order switches being defined as those changes in ranking position of the prediction relative to the order of the true observations y;calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches;transforming the number of ranking switches and weighted sum of ranking order switches into a range of [−1, 1] wherein −1 corresponds to making all possible errors (inverse ranking) and 1 corresponds to a perfect model wherein said number of ranking switches has been transformed to represent a difference between a probability that the ranking of two customers and/or potential customers are in the same order versus the probability that two of the customers and/or potential customers are in different orders from the originally obtained rank;normalizing transformed measures of order switches into a range of [0,1] wherein 1 corresponds to perfect ranking and 0 corresponds inverse ranking;calculating a variance of measures of order switches and determining confidence intervals for each ranking measure;updating the model performance table with the ranking measures and their confidence intervals;outputting the ranking measures and confidence levels as well as graphical representations thereof to a domain expert, who will choose based on this information the best model m* of the one or more models evaluated, and storing the predictions of ŷ=m*(x) in the optimal prediction table.