LEARNING A DOCUMENT RANKING USING A LOSS FUNCTION WITH A RANK PAIR OR A QUERY PARAMETER

Abstract

A method and system for generating a ranking function to rank the relevance of documents to a query is provided. The ranking system learns a ranking function from training data that includes queries, resultant documents, and relevance of each document to its query. The ranking system learns a ranking function using the training data by weighting incorrect rankings of relevant documents more heavily than the incorrect rankings of not relevant documents so that more emphasis is placed on correctly ranking relevant documents. The ranking system may also learn a ranking function using the training data by normalizing the contribution of each query to the ranking function so that it is independent of the number of relevant documents of each query.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates components of the ranking system in one embodiment.

FIG. 2 is a flow diagram illustrating the processing of the generate document ranker component of the ranking system in one embodiment.

FIG. 3 is a flow diagram that illustrates the processing of the generate labels component of the ranking system in one embodiment.

FIG. 4 is a flow diagram that illustrates the processing of the generate rank pair parameters component of the ranking system in one embodiment.

FIG. 5 is a flow diagram illustrating the processing of the generate query parameters component of the ranking system in one embodiment.

FIG. 6 is a flow diagram that illustrates the processing of the train document ranker component of the ranking system in one embodiment.

FIG. 7 is a flow diagram that illustrates the processing of the rank documents component of the ranking system in one embodiment.

FIG. 8 is a flow diagram that illustrates the processing of the document ranker component of the ranking system in one embodiment.

DETAILED DESCRIPTION

A method and system for generating a ranking function to rank the relevance of documents to a query is provided. In one embodiment, the ranking system learns a ranking function from a collection of queries, resultant documents, and relevance of each document to its query. For example, the queries may be submitted to a web-based search engine to identify the resultant documents that may satisfy the query. The ranking system then determines the relevance of each resultant document to its query. For example, the ranking system may input from a user the relevance of each document to its query. The queries, documents, and relevances are the training data that the ranking system uses to learn the ranking function. The ranking system learns a ranking function using the training data by weighting incorrect rankings of relevant documents more heavily than the incorrect rankings of not relevant documents so that more emphasis is placed on correctly ranking relevant documents. For example, the ranking system may adjust a loss or error function so that more emphasis is placed on minimizing the error in the ranking function's ranking of relevant documents and less emphasis is placed on the error in the ranking function's ranking of not relevant documents. As a result, the ranking function will more correctly rank relevant documents than it does irrelevant documents. In one embodiment, the ranking system may alternatively learn a ranking function using the training data by normalizing the contribution of each query to the ranking function by factoring in the number of resultant documents (e.g., relevant documents) of each query. As a result, the ranking function will reflect contributions made by each query in a way that is independent of the number of resultant documents. The ranking system may either weight the ranking of relevant documents more heavily or normalize the contribution of a query based on number of documents when generating a ranking function, or use both in combination as described below. As a result, the ranking system can generate a ranking function that results in a ranking that is more desired by typical users of a search engine.

In one embodiment, the ranking system generates a ranking function using training data derived from queries and resultant documents that may be collected by submitting the queries to search engines. The ranking system then inputs a ranking of the relevance of each document to its query. For example, the ranking system may prompt a user to indicate the relevance classification, such as relevant, partially relevant, or irrelevant, indicating the relevance of each document to its query. The ranking system generates a feature vector for each document. The feature vector includes features that are useful for determining the relevance of a document to a query. For example, the feature vector may include a count of the number of times a term of the query occurs in the document, the number of terms in the document, and so on. The ranking system generates a label for ordered pairs of documents with different relevance classifications for each query. For example, a pair of documents may include one relevant document (r) and one irrelevant document (i) resulting in two ordered pairs: (r,i) and (i,r). Thus, if a query has 10 documents with 2 documents being relevant, 3 documents being partially relevant, and 5 documents being irrelevant, then the query has 62 pairs (i.e., 2*(2*3+2*5+3*5)). Each ordered pair is also referred to as an instance pair. The ranking system then generates a label for each instance pair indicating whether the ranking of the documents within the instance pair is correct. For example, the ranking of (r,i) is correct assuming the higher ranking document is first in the pair. If so, then the ranking of (i,r) is incorrect.

In one embodiment, the ranking system uses a rank pair parameter for each pair of relevance classifications. The relevance classification pairs (or ranking pairs) are (relevant, partially relevant), (partially relevant, relevant), (relevant, irrelevant), and so on. The rank pair parameter for a ranking pair indicates a weighting for errors in the learning of the ranking function attributable to instance pairs corresponding to that ranking pair. For example, an error in ranking a (relevant, irrelevant) instance pair will be weighted more heavily than an error in ranking a (partially relevant, irrelevant) instance pair because an incorrect ranking of a relevant document is very undesirable whereas the incorrect ranking of a partially relevant document as irrelevant will probably not be noticed by the user. By weighting errors according to the rank pair parameters, the ranking system generates a ranking function that will more likely generate the correct rankings for relevant documents than for not relevant documents generated by switching documents between the relevance classifications of the rank pair. The rank pair parameters may be specified manually or may be generated automatically. In one embodiment, the ranking system generates the ranking pair parameters automatically by calculating an evaluation measure of the perfect ranking of documents for a query and calculating evaluation measures for various not perfect rankings of the documents. The ranking system may perform these calculations for each query and then use the average of the differences between the perfect evaluation measure and the not perfect evaluation measures as the rank pair parameter. The ranking system may use various evaluation measurements such as mean reciprocal rank, winner take all, mean average precision, and normalized discounted cumulative gain.

In one embodiment, the ranking system uses a query parameter for each query to normalize the contribution of the queries to the generation of the ranking function. The ranking system may generate a query parameter for a query based on the number of resultant documents of that query relative to the maximum number of resultant documents of a query of the collection. The ranking system may set the query parameter of a query to the maximum number of resultant documents divided by the number of resultant documents for the query. The ranking system may more specifically set the query parameter of a query to the maximum number of instance pairs of a query divided by the number of instance pairs of the query, which are derived based on the relevance classifications of the pairs of documents.

The ranking system generates the ranking function using various training techniques such as gradient descent or quadratic programming. When gradient descent is used, the ranking system iteratively adjusts weighting parameters for the feature vector used by the ranking function until the error in the ranking function as applied to the training data converges on a solution. During each iteration, the ranking system applies the ranking function with current weighting parameters to each instance pair. If the ranking is incorrect, the ranking function then calculates an adjustment for the current weighting parameters. That adjustment factors in the rank pair parameter and the query parameter as discussed above. At the end of each iteration, the ranking system calculates new current weighting parameters.

The ranking system may represent documents in an input space X ε Rⁿ where n represents the number of features of a feature vector and may represent rankings (or categories) of the documents in an output space of relevance classifications Y={r₁, r₂, . . . , r_q} where q represents the number of ranks (e.g., relevant, partially relevant, and irrelevant). The ranking system may be implemented using a number of ranks selected based on the goals of the ranking system. A total order between the ranks may be represented as r_qr_q−1 . . . r₁, where denotes a ranking relationship. The ranking system learns a ranking function out of a set of possible ranking functions ƒ ε F that each determine the ranking relationship between an instance pair as represented by the following equation:

{right arrow over (x)} _i {right arrow over (x)} _j ƒ({right arrow over (x)}_i)>ƒ({right arrow over (x)})   (1)

where {right arrow over (x)}_i represents the feature vector for document i. The ranking system uses as training data a set of ranked instances S={({right arrow over (x)}_i,y_i)}_i=1^t from the space X×Y. The ranking system may generate a linear or non-linear ranking function. A linear ranking function is represented by the following equation:

ƒ_{{right arrow over (w)}}({right arrow over (x)})=<{right arrow over (w)},{right arrow over (x)}>  (2)

where {right arrow over (w)} denotes a vector of weighting parameters and <·,·> represents an inner product. By substituting Equation 2 into Equation 1, the resulting equation is represented by the following equation:

{right arrow over (x)} _i {right arrow over (x)} _j <{right arrow over (w)},{right arrow over (x)} _i −{right arrow over (x)} _j>>0   (3)

The relationship {right arrow over (x)}_i{right arrow over (x)}_j between instance pairs {right arrow over (x)}_i and {right arrow over (x)}_j is expressed by a new vector {right arrow over (x)}_i−{right arrow over (x)}_j. The ranking system creates the new vector and a label for each instance pair as represented by the following equation:

$\begin{array}{cc}({\overrightarrow{x}}^{\left(1\right)}-{\overrightarrow{x}}^{\left(2\right)},z=\left\{\begin{array}{cc}+1& y^{\left(1\right)}\succ y^{\left(2\right)}\\ -1& y^{\left(2\right)}\succ y^{\left(1\right)}\end{array}\right)& \left(4\right)\end{array}$

where {right arrow over (x)}⁽¹⁾ and {right arrow over (x)}⁽²⁾ represent the first and second documents and y⁽¹⁾ and y⁽²⁾ represent their ranks. From the given training data set S, the ranking system creates a new training data set S′ containing labeled vectors as represented by the following equation:

S′={{right arrow over (x)} _i ⁽¹⁾ −{right arrow over (x)} _i ⁽²⁾ ,z _i}_i=1^l   (5)

where l represents the number of instance pairs. The ranking system then generates an SVM model from the new training data S′ to assign either positive label z=+1 or negative label z=−1 to any vector {right arrow over (x)}_i⁽¹⁾−{right arrow over (x)}⁽²⁾. The constructing of the SVM model is equivalent to solving a quadratic optimization problem as represented by the following equation:

$\begin{array}{cc}\underset{\overrightarrow{w}}{\min }L\left(\overrightarrow{w}\right)=\sum _{i=1}^l{\tau }_{k\left(i\right)}{\mu }_{q\left(i\right)}\left[1-z_i〈\overrightarrow{w},{\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)}〉\right]+\lambda {\overrightarrow{w}}^2& \left(6\right)\end{array}$

where k(i) represents the type of ranks of instance pair i, τ_k(i) represents the rank pair parameter for k(i), q(i) represents the query of instance pair i, μ_q(i) represents the query parameter for q(i), and λ∥{right arrow over (w)}∥² is a regularizer. The ranking system represents a penalty for the i^th pair being incorrectly ranked as the product of τ_k(i) and μ_q(i).

The ranking system can solve for the loss function of Equation 6 using a gradient descent algorithm. The loss function can be represented by the following equation:

$\begin{array}{cc}L\left(\overrightarrow{w}\right)=\sum _{i=1}^ll_i\left(\overrightarrow{w}\right)+\lambda {\overrightarrow{w}}^2,\mathrm{where}l_i\left(\overrightarrow{w}\right)={\tau }_{k\left(i\right)}{{\mu }_{q\left(i\right)}\left[1-z_i〈\overrightarrow{w},{\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)}〉\right]}_{+}& \left(7\right)\end{array}$

Equation 7 can be differentiated with respect to {right arrow over (w)} as represented by the following equation:

$\begin{array}{cc}\frac{\partial L}{\partial \overrightarrow{w}}=\sum _{i=1}^l\frac{\partial l_i\left(\overrightarrow{w}\right)}{\partial \overrightarrow{w}}+2\lambda \overrightarrow{w},\mathrm{where}\frac{\partial l_i\left(\overrightarrow{w}\right)}{\partial \overrightarrow{w}}=\{\begin{array}{cc}0& \mathrm{if}z_i〈\overrightarrow{w},{\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)}〉\ge 1\\ -z_i{\tau }_{k\left(i\right)}{\mu }_{q\left(i\right)}\left({\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)}\right)& \mathrm{if}z_i〈\overrightarrow{w},{\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)}〉<1\end{array}& \left(8\right)\end{array}$

The iteration equations of the gradient descent method may be represented by the following equations:

$\begin{array}{cc}{\overrightarrow{w}}^{\left(k+1\right)}={\overrightarrow{w}}^{\left(k\right)}+{\eta }_k\Delta {\overrightarrow{w}}^{\left(k\right)}\Delta {\overrightarrow{w}}^{\left(k\right)}=-\nabla L\left({\overrightarrow{w}}^{\left(k\right)}\right)=-\sum _{i=1}^l\frac{\partial l_i\left({\overrightarrow{w}}^{\left(k\right)}\right)}{\partial \overrightarrow{w}}-2\lambda {\overrightarrow{w}}^{\left(k\right)}{\eta }_k\text{:}L\left({\overrightarrow{w}}^{\left(k\right)}+{\eta }_k\Delta {\overrightarrow{w}}^{\left(k\right)}\right)=\underset{\eta \ge 0}{\min }L\left({\overrightarrow{w}}^{\left(k\right)}+\eta \Delta {\overrightarrow{w}}^{\left(k\right)}\right)& \left(9\right)\end{array}$

At each iteration, the ranking system reduces the cost function along its descent direction as represented by Equation 8. To determine the step size of each iteration, the ranking system conducts a line search along the descent direction as described by Equations 9. The ranking system may calculate a learning factor η_k to control the learning rate at each iteration k. In one embodiment, rather than calculating each η_k at each iteration, the ranking system uses a fixed learning factor.

The ranking system alternatively can solve for the loss function of Equation 6 using a quadratic programming algorithm. The loss function can be represented as a quadratic optimization problem as represented by the following equation:

$\begin{array}{cc}\underset{\overrightarrow{w}}{\min }M\left(\overrightarrow{w}\right)=\frac{1}{2}{\overrightarrow{w}}^2+\sum _{i=1}^lC_i·{\xi }_i\mathrm{subject}\mathrm{to}{\xi }_i\ge 0,z_i〈\overrightarrow{w},{\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)}〉\ge 1-{\xi }_ii=1,\dots ,l\mathrm{where}C_i=\frac{{\tau }_{k\left(i\right)}{\mu }_{q\left(i\right)}}{2\lambda }.& \left(10\right)\end{array}$

The corresponding Lagrange function can be represented by the following equation:

$\begin{array}{cc}\begin{array}{c}{L}_P=\frac{1}{2}{\overrightarrow{w}}^2+\sum _{i=1}^lC_i·{\xi }_i-\sum _{i=1}^l{\alpha }_i[z_i〈\overrightarrow{w},{\overrightarrow{x}}_i^{\left(1\right)}-\\ {\overrightarrow{x}}_i^{\left(2\right)}〉-\left(1-{\xi }_i\right)]-\sum _{i=1}^l{\mu }_i{\xi }_i\end{array}& \left(11\right)\end{array}$

The goal is to minimize L_p with respect to {right arrow over (w)} and ξ_i. Setting their derivatives to zero results in the following equations:

$\begin{array}{cc}\overrightarrow{w}=\sum _{i=1}^l{\alpha }_iz_i〈{\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)}〉& \left(12\right)\end{array}$ α_i=C_i−μ_i i=1, . . . , l   (13)

along with the positive constraints α_i, μ_i, ξ_i i=1, . . . , l. The substitution of Equations 12 and 13 into Equation 11 can be represented by the following equation:

$\begin{array}{cc}{L}_D=\sum _{i=1}^l{\alpha }_i-\frac{1}{2}\sum _{i=1}^l\sum _{i^{\prime }=1}^l{\alpha }_i{\alpha }_{i^{\prime }}z_iz_{i^{\prime }}〈{\overrightarrow{x}}_i^{\left(1\right)}-{\overrightarrow{x}}_i^{\left(2\right)},{\overrightarrow{x}}_{i^{\prime }}^{\left(1\right)}-{\overrightarrow{x}}_{i^{\prime }}^{\left(2\right)}〉& \left(14\right)\end{array}$

The goal is to maximize L_D subject to the constraints represented by the following equation:

0≦α_i≦C_i i=1, . . . , l   (15)

FIG. 1 is a block diagram that illustrates components of the ranking system in one embodiment. The ranking system 100 may be connected to various document stores 140 via communications link 130. A document store may include a search engine service for searching for documents within the store. Alternatively, a search engine may be external to the document store (e.g., a search engine for searching web pages). The ranking system includes a generation subsystem 110 and a ranking subsystem 120. The generation subsystem is used to learn the ranking function. The generation subsystem includes a generate document ranker component 112 that inputs queries and resultant documents from the query/document store 111 and trains a document ranker. The generate document ranker component invokes a collect queries and resultant documents component 113, a collect ranks for documents component 114, a generate feature vectors component 115, a generate labels component 116, a generate rank pair parameters component 117, a generate query parameters component 118, and a train document ranker component 119, which are described below in more detail. The ranking subsystem component includes a rank documents component 121 and a document ranker component 122. The rank documents component receives a query and resultant documents and generates a ranking of the relevance of the resultant documents to the query. The rank documents component invokes the document ranker component that was trained by the generation subsystem.

The computing devices on which the ranking system may be implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives). The memory and storage devices are computer-readable media that may contain instructions that implement the ranking system. In addition, the data structures and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communications link. Various communications links may be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection.

The ranking system may be implemented on various computing systems or devices including personal computers, server computers, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The ranking system may also provide its services (e.g., ranking of search results using the ranking function) to various computing systems such as personal computers, cell phones, personal digital assistants, consumer electronics, home automation devices, and so on.

The ranking system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. For example, the training component may be implemented on a computer system separate from the computer system that collects the training data or the computer system that uses the ranking function to rank search results.

FIG. 2 is a flow diagram illustrating the processing of the generate document ranker component of the ranking system in one embodiment. The component is invoked to generate a document ranker component that is used to rank the relevance of documents to queries. In block 201, the component collects the queries and resultant documents. The queries and resultant documents may be collected by submitting queries to various search engine services and storing the queries and resultant documents in the query/documents store. In block 202, the component collects ranks for the documents indicating the relevance of each document to its query. The component may collect the ranks from a person who manually determines the ranking of the documents of each query. In block 203, the component generates the feature vectors for each document of each query. In block 204, the component invokes the generate labels component to generate a label indicating whether each instance pair represents a correct relationship between the documents of the instance pair. In block 205, the component invokes the generate rank pair parameters component to generate a rank pair parameter for each rank pair. In block 206, the component invokes the generate query parameters component to generate a query parameter for each query. In block 207, the component invokes the train document ranker component to train a document ranker to rank documents and then completes.

FIG. 3 is a flow diagram that illustrates the processing of the generate labels component of the ranking system in one embodiment. The component loops selecting each query and selecting each instance pair for the selected query and sets its label to indicate whether the ranking of the pair is correct. In block 301, the component selects the next query. In decision block 302, if all the queries have already been selected, then the component returns, else the component continues at block 303. In block 303, the component selects the next instance pair for the selected query. In decision block 304, if all the instance pairs have already been selected, then the component loops to block 301 to select the next query, else the component continues at block 305. In block 305, if the ordering of the documents of the instance pair is correct, then the component continues at block 306 to set the label to positive one, else the component continues at block 307 to set the label to negative one. The component then loops to block 303 to select the next instance pair for the selected query.

FIG. 4 is a flow diagram that illustrates the processing of the generate rank pair parameters component of the ranking system in one embodiment. The component generates a rank pair parameter for each possible pair of ranking or relevance classifications. The rank pair parameters are used to weight a loss function so that relevant documents are more correctly ranked than not relevant documents. In block 401, the component selects the next rank pair. In decision block 402, if all the rank pairs have already been selected, then the component returns, else the component continues at block 403. In blocks 403-410, the component loops selecting each query and calculating the difference between a perfect ranking of the documents of the query and various random selected rankings of the documents. In block 403, the component selects the next query. In decision block 404, if all the queries have already been selected, then the component continues at block 411, else the component continues at block 405. In block 405, the component calculates an evaluation measure of the perfect ranking. In block 406, the component randomly selects a document from each rank of the selected rank pair. In decision block 407, if enough not perfect rankings have already been selected to provide an accurate rank pair parameter, then the component loops to block 403 to select the next query, else the component continues at block 408. The component may determine that enough have already been selected when the average difference converges on a solution. In block 408, the component switches the ranks of the selected documents. In block 409, the component calculates an evaluation measurement for the switched ranking. In block 410, the component calculates the difference between the switched and perfect evaluation measures and then loops to block 406 to select documents from each rank. In block 411, the component sets the rank pair parameter for the selected rank pair to the average of the differences as calculated in block 410. One skilled in the art will appreciate that many different techniques can be used to generate the rank pair parameters. For example, a user could manually indicate an evaluation measurement, a user could directly indicate the rank pair parameters, and so on.

FIG. 5 is a flow diagram illustrating the processing of the generate query parameters component of the ranking system in one embodiment. The component generates a query parameter for each query to normalize the contribution of each query derived from the number of resultant documents of the query such as the number of instance pairs in the various ranks. In block 501, the component selects the next query. In decision block 502, if all the queries have already been selected, then the component continues at block 508, else the component continues at block 503. In block 503, the component selects the next instance pair of the selected query. In decision block 504, if all the instance pairs of the selected query have already been selected, then the component continues at block 506, else the component continues at block 505. In block 505, the component increments a count of instance pairs for the selected query and then loops to block 503 to select the next instance pair. In decision block 506, if the total count of the instance pairs for the selected query is greater than the maximum count encountered so far, then the component continues at block 507, else the component loops to block 501 to select the next query. In block 507, the component sets the maximum count to the count of the selected query and then loops to block 501 to select the next query. In block 508, the component sets a query parameter for each query to the maximum count divided by the count for that query and then returns. One skilled in the art will appreciate that various techniques may be used to calculate a query parameter. For example, the query parameter may vary nonlinearly based on the number of document or instance pairs, may vary based on the number of relevant documents, and so on.

FIG. 6 is a flow diagram that illustrates the processing of the train document ranker component of the ranking system in one embodiment. The train document ranker component implements the gradient descent algorithm as described above. In block 601, the component initializes the weighting parameters or weights for the feature vectors to the zero vector. In decision block 602, if a termination condition has been satisfied, then the component returns the weights, else the component continues at block 603. In block 603, the component initializes a change in weights to the zero vector and initializes an index. In blocks 604-607, the component loops calculating a change in the feature vector attributed to incorrectly ranked instance pairs. In block 604, the component selects the next instance pair. In decision block 605, if all the instance pairs have already been selected, then the component continues at block 608, else the component continues at block 606. In decision block 606, if the current weights result in an incorrect ranking, then the component continues at block 607, else the component continues at block 604 to select the next instance pair. In block 607, the component calculates an adjustment to the weights factoring in the rank pair parameter and the query parameter for the selected instance pair and then loops to block 604 to select the next instance pair. In block 608, the component adjusts the value of the change in weights based on the regularizer. In block 609, the component sets the new current value of the weights to the old current value plus the change in weights adjusted by a learning factor. The component then loops to block 602 to determine whether the training is complete, for example, when the adjustments converge to zero or after a fixed number of iterations. One skilled in the art will appreciate many different training techniques can be used to train a ranking function such as adaptive boosting, neural networks, and so on.

FIG. 7 is a flow diagram that illustrates the processing of the rank documents component of the ranking system in one embodiment. The component is passed a query and resultant documents and generates a ranking for the documents. In block 701, the component selects the next document pair. In decision block 702, if sufficient document pairs have already been selected, then the component returns, else the component continues at block 703. In block 703, the component generates a feature vector for the documents of the selected document pair. In block 704, the component invokes the document ranker component to generate a rank for the selected documents relative to each other. In block 705, the component sets the rank of the selected document pair to the returned rank and then loops to block 701 to select the next document pair. At each iteration, the component may select the higher ranked document of the selected pair from the last iteration and use that as one of the documents of the pair for this iteration. In this way, the component will converge on selecting the most relevant documents. The other document may be selected randomly from documents that have not yet been selected. Alternatively, the component may use Equation 2 to generate a score for each document indicating the relevance of the document to the query. The component can then rank the documents according to their scores.

FIG. 8 is a flow diagram that illustrates the processing of the document ranker component of the ranking system in one embodiment. The component is passed feature vectors for a pair of documents and returns a score of the correct relative ranking of the documents. In block 801, the component selects the next feature of the feature vectors. In decision block 802, if all the features have already been selected, then the component returns the score, else the component continues at block 803. In block 803, the component increments the score by the weight of the selected feature times the difference in the values of the selected feature of the passed feature vectors. The component then loops to block 801 to select the next feature.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. The ranking system may be used to rank a variety of different types of documents. For example, the documents may be web pages returned as a search result by a search engine, scholarly articles returned by a journal publication system, records returned by a database system, news reports of a news wire service, and so on. Accordingly, the invention is not limited except as by the appended claims.

Claims

1. A computer system for generating a ranking function to rank relevance of a document to a query, comprising: a collection of queries, resultant documents, and relevance of each document to its query; anda component that trains a ranking function using the documents and the relevances by weighting incorrect rankings of relevant documents more heavily than the incorrect ranking of not relevant documents so that the ranking function more correctly ranks relevant documents than it does not relevant documents.

2. The computer system of claim 1 including a component that uses the generated ranking function to rank the relevance of documents to a query.

3. The computer system of claim 1 including a component that generates training data from queries, documents, and relevances, the training data comprising feature vectors and labels, each feature vector being derived from a query and a resultant document, each label indicating correct ordering of relevance of a pair of resultant documents of a query.

4. The computer system of claim 1 wherein a different weighting is used for each rank pair where a rank pair represents a combination of two different relevance classifications.

5. The computer system of claim 4 wherein the relevance classifications are relevant, partially relevant, and irrelevant.

6. The computer system of claim 4 including a component that generates the weightings by comparing an evaluation measure of incorrect rankings of documents to an evaluation measure of correct rankings of the documents.

7. The computer system of claim 6 wherein the evaluation measure is selected from the group consisting of mean reciprocal rank, winner take all, mean average precision, and normalized discounted cumulative gain.

8. The computer system of claim 6 wherein the weighting is set to an average of differences between the evaluation measure of the correct rankings and the evaluation measure of the incorrect rankings.

9. The computer system of claim 1 wherein the component that trains the ranking function uses a gradient descent algorithm.

10. A computer system for generating a ranking function to rank relevance of a document to a query, comprising: a collection of queries, resultant documents, and relevance of each document to its query; anda component that trains a ranking function using the queries, resultant documents, and relevances by normalizing a contribution of each query to the ranking function by factoring in instance pairs of resultant documents of a query so that the contributions of queries are independent of the number of instance pairs of a query.

11. The computer system of claim 10 including a component that uses the generated ranking function to rank the relevance of documents to a query.

12. The computer system of claim 10 including a component that generates training data from queries, documents, and relevances, the training data comprising feature vectors and labels, each feature vector being derived from a query and a resultant document, each label indicating correct ordering of relevance of a pair of resultant documents of a query.

13. The computer system of claim 12 wherein each pair of documents includes documents from different relevance classifications.

14. The computer system of claim 13 wherein the relevance classifications are relevant, partially relevant, and irrelevant.

15. The computer system of claim 10 wherein the contribution of a query is inversely proportional to the number of instance pairs for the query to the maximum number of instance pairs of a query.

16. The computer system of claim 10 wherein the component that trains the ranking function weights incorrect rankings of relevant documents more heavily than the incorrect ranking of not relevant documents so that the ranking function more correctly ranks relevant documents than it does not relevant documents.

17. A computer system for ranking relevance of a document to a query, comprising: a component that ranks relevance of a document to a query using feature weights to apply to features of the document, the feature weights being learned based on training data of features and relevances of documents to queries by weighting incorrect ranking of relevant documents more heavily than the incorrect ranking of not relevant documents so that relevant documents are more correctly ranked than not relevant documents;a component that identifies features of resultant documents of a query; anda component that provides the features of the resultant documents to the component that ranks the resultant documents of the query.

18. The computer system of claim 17 wherein the component that ranks learns feature weights by normalizing a contribution of each query by factoring into the contribution the number of relevant documents of a query so that a query with more relevant documents makes a contribution similar to a query with fewer relevant documents.

19. The computer system of claim 17 wherein an incorrect ranking of a relevant document as an irrelevant document is weighted more heavily than an incorrect ranking of a relevant document as a partially relevant document.

20. The computer system of claim 17 wherein the component that provides the features provides features for pairs of resultant documents.