Patent Application
20200210438
The following disclosures are submitted under 35 U.S.C. 102(b)(1)(A): Roitman, Haggai. (2018). “Enhanced Performance Prediction of Fusion-based Retrieval.” 195-198. 10.1145/3234944.3234950.
The present invention relates to information retrieval, and more specifically, to query performance prediction for information retrieval systems.
Information retrieval systems seek to provide relevant information from a collection of information resources for a user's information needs. For a given query and corpus of documents, an information retrieval technique may return a ranked list indicating the relevance of the documents to the query. Different information retrieval techniques may return different ranked lists for the same query. Fusion-based retrieval has been used to improve the quality of information retrieval results by combining the ranked list outputs of multiple retrieval techniques into a single fused list, which is returned as the final retrieval outcome.
Query performance prediction (QPP) techniques may be used to assess information retrieval performance in the absence of relevance judgments. In the fusion-based retrieval setting, one commonly employed QPP approach treats the various base ranked lists, generated by different retrieval techniques prior to the fusion of such lists, as reference lists and combines QPP estimates for the reference lists according to their symmetric similarity to the fused list. The higher the predicted query performance according to a given reference list is and the more similar that list is to the fused list, the better the query performance is assumed to be.
According to one embodiment, a computer-implemented method is provided for predicting query performance. The method generally includes receiving one or more initial ranked lists that are generated using respective information retrieval techniques in response to a query and a final ranked list that is generated based, at least in part, on the one or more initial ranked lists. The method further includes determining a likelihood that the final ranked list includes information relevant to the query based, at least in part, on (i) a respective likelihood that each of the one or more initial ranked lists includes information relevant to the query, and (ii) a respective asymmetric co-relevance between each of the one or more initial ranked lists and the final ranked list.
Further embodiments provide a non-transitory computer-readable medium that includes instructions that, when executed, enable a computer to implement one or more aspects of the above method, and a computer system programmed to implement one or more aspects of the above method.
Embodiments are disclosed that enhance query performance prediction (QPP) in information retrieval systems. The symmetric list similarity measures used by traditional QPP techniques in the fusion-based retrieval setting (and other settings) do not properly account (if at all) for relevance-dependent aspects of the relationship between a given (base) reference list generated using an information retrieval technique and a final fused list generated using a fusion technique. Such a relationship is actually asymmetric, rather than symmetric, as there is a “cause and effect” based relationship between the reference lists and the fused list generated from the reference lists. Embodiments more properly model the asymmetric relationship of reference and fused lists using an asymmetric co-relevance model that estimates, assuming a reference list includes relevant information, the odds that the fused list will be observed. In particular, the asymmetric co-relevance between a reference list and a fused list may be determined by adjusting a symmetric co-relevance of the reference list and the fused list using an odds ratio between the symmetric co-relevance of the reference list and the fused list to the reference list's own relevance. In the fusion-based retrieval setting, the QPP techniques disclosed herein that model asymmetric co-relevance relationships between reference lists and a fused list may be used to select a subset of reference lists to fuse from a set of reference lists and/or to select a fusion technique to use from multiple available fusion techniques.
For example, the information retrieval application 120 may be a web search engine that receives a user query over the Internet, generates a ranked list of web pages (or other relevant information on the Internet such as images, videos, etc.), and returns a portion of the list containing the highest ranked web pages (or other relevant information) as a web page that can be loaded by a web browser and displayed via a display device. Other example applications of information retrieval systems include recommender systems, desktop search, enterprise search, information filtering, news search, federated search, etc.
In the fusion-based retrieval setting, instead of obtaining only a single ranked list, the quality of information retrieval results is improved by combining the ranked list outputs obtained using multiple retrieval techniques.
More formally, let q denote a query and C denote a corpus of documents. In a fusion-based retrieval setting, query q is evaluated over corpus C using several different retrieval techniques. Let ={L1, L2, . . . , Lm} denote a set of m(>1) ranked lists, where each ranked list Li(∈
contains the top-k documents d(∈ C) with the highest retrieval scores SL(d|q) according to the i-th retrieval technique (i ∈{1,2. . . , m}), A fusion technique F takes the ranked lists
as input and returns an aggregated (top-k) ranked list, also referred to herein as the fused list and denoted LF.
, described above, are treated as the reference lists 210-1 to 210-m and p(LF|q,r) is estimated as follows:
{circumflex over (p)}(LF|q, r) p(LF|Li, r)p(Li|q, r). (1)
Here, every base (ranked) list Li ∈ in equation (1) may be assumed to be a pseudo-effective reference list, the term p(Li|q,r) entails a generic prediction approach based on each base ranked list Li's own performance prediction, and the term p(LF|Li,r) captures a relationship between the (presumably) effective base ranked list Li and the fused list LF.
Traditionally, the p(LF|Li,r) term in equation (1) was estimated according to the symmetric similarity sim(Li,LF) between the base and fused ranked lists Li and LF, and list similarity was further estimated based on either document identities (e.g., overlap), their (retrieval) scores (e.g., correlation) or relative ranks within both ranked lists. By contrast, embodiments disclosed herein rely on the fact that p(LF|Li,r) actually “encodes” an asymmetric co-relevance relationship between the base and fused ranked lists Li and LF. Such a relationship captures the fact that a given “relevant” reference list Li serves as an informative evidence source of a given observed fused list LF. Traditional reference-list based QPP techniques, which model the relationship solely according to sim(Li,LF), do not properly account for such a “cause and effect” based relationship between the reference Li and fused LF lists. That is, the asymmetric co-relevance relationship between the base and fused ranked lists Li and LF accounts for the order in which these lists are generated and attempts to determine, given that a base ranked list Li is effective (i.e., assuming that that the base ranked list Li has some relevant information), what the likelihood is of observing the fused list LF.
The following is a derivation of an estimator for the asymmetric co-relevance likelihood p(LF|Li,r), according to an embodiment. Using a Bayesian interpretation of the likelihood p(LF|Li,r) gives:
Here, the term p(Lf|Li) captures the relevance-independent part of the relationship between the base and fused ranked lists Li and LF, which corresponds to the list similarity sim(Li, LF) estimate in traditional reference-list based QPP techniques. Using this fact, the following “adjusted” similarity-based estimator may be derived:
{circumflex over (p)}(LF|Li, r) Or(LF, Li)sim(Li, LF), (3)
where:
Or(LF, Li) adjusts sim(Li,LF) according to the odds-ratio between the symmetric (joint) co-relevance of both ranked lists to list Li's own relevance, i.e., the odds of containing relevant information given that both the base ranked list Li and the fused list LF are observed, divided by the odds of containing relevant information given that only the base ranked list Li is observed. Intuitively, the higher the odds are, the more informative the base ranked list Li(∈) is assumed to be with respect to the determination of the relevancy of the fused list LF. As a result, higher odds are assumed to imply that higher quality should be predicted for fused list LF based on its similarity with base ranked list Li. That is, the lower the denominator and higher the numerator in equation (4), the more the base ranked list is informative in its relationship to the fused list in terms of explaining that the fused list will include relevant information. Conversely, if the odds are low (the denominator is higher relative to the numerator), then the base ranked list is not informative enough to explain that the fused list is relevant. Intuitively, if the starting point is a base ranked list that is not able to explain whether a given document is relevant or not, then the final fused list cannot rely on such a base ranked list as a source of relevant information.
The term p(r|LF, Li) in equation (4) captures the symmetric co-relevance relationship between the base ranked and fused lists Li and LF. This term may be estimated given the documents in LF as:
{circumflex over (p)}(r|LF, Li)Σd∈L
Then, after noting that
and modifying equation (5) accordingly, the result of such a modification can be inserted back into equation (4), giving the following derivation of Or(LF, Li):
Equation (6) reveals that Or (LF, Li) basically adjusts the similarity sim(Li, LF) with respect to each observed document d ∈LF, where the likelihood ratio
in equation (6) weights each document d ∈ LF based on whether or not the reference list Li should “attest” for d's relevancy. On the one extreme, p(d,r|Li)<<p(d|Li)p(r|Li), which may imply that there is a low confidence that list Li should serve as the “evidence” source for document d's relevancy. As a result, in such a case, the relevant contribution of sim(Li, LF) should be attenuated to the performance prediction of LF. On the other extreme, p(d, r|Li) >>p(d|Li)p(r|Li), which would (marginally) boost sim(Li, LF) and imply that list Li is informative and, therefore, should be used to predict the performance of LF.
For any given list L(∈∪{LF}), the term p(d|L) in equation (6) captures the (general) relationship between a given document d and list L. In one embodiment, p(d|L) in equation (6) is estimated as the Bhattacharyya similarity between unsmoothed language models: {circumflex over (p)}(d|L)
Σw√{square root over ({circumflex over (p)}(w|θd){circumflex over (p)}(w|θL))}. A language model gives the likelihood of observing a word in a text. The words in a text, such as a concatenation of all documents in a list to obtain a meta-document representing the list, can be counted in order to obtain a language model (which is a count of words divided by the total number of words), such as a language model of the list. Although described herein primarily with respect to Bhattacharyya similarity, other measures of similarity may be used in alternative embodiments, such as Cosine similarity or other similarity measures from the f-divergence similarity family to which Bhattacharyya similarity belongs.
Next, for any given list L(∈), the term p(r|L) in equation (6) measures a likelihood that list L contains any relevant information and may be estimated as the entropy of its unsmoothed language model: {circumflex over (p)}(r|L)
−Σw{circumflex over (p)}(w|θL)log{circumflex over (p)}(w|θL). That is, the information retrieval application 120 may go over the lexicon of the unsmoothed language model (including possible words) and measure entropy using such a formula.
Finally, the term p(d,r|Li) in equation (6) measures a likelihood that a relevant document is observed in list Li. Note that p(d,r|Li) ∞p(Li, r|d), meaning the term p(d,r|Li) may be estimated as the information gained by generating the relevance model induced from list L (denoted
from document d's (∈LF) unsmoothed language model to generating the term from the background (corpus) model, i.e.,
A relevance model is a language model that gives higher probability to words that are more relevant to a query, i.e., the relevance model not only counts words but gives a bias to words that are more relevant to the query. A list may be assumed to contain relevant information and be represented by its relevance model, and the question is then whether the relevance model for the list can be encoded using a document's own language model. Assuming that the entire corpus of documents is a background language model, the information retrieval application 120 may create a metadocument by concatenating all of the documents in the corpus. The information gained by generating the relevance model induced from list L from document d′s unsmoothed language model to generating the term from the background (corpus) model is then a difference between entropies (in equation (7)), one entropy given the document model and the other entropy given the background corpus model. A document is able to generate a relevant list if its own language model is more capable of explaining the relevance model than the background corpus model.
may be sum-normalized. Let
Σim
NZ(d|L)·SCombSUM(d|q, L), where NZ(d|L) is the number of lists in L that contain document d. For the Borda and RFF techniques, let rankL
Σim#{d′ ∈Li; rankL
At step 420, the information retrieval application 120 determines a respective likelihood that each base ranked list includes information relevant to the query. That is, the p(Li|q,r) term in equation (1) is determined. Any technically feasible QPP technique, such as the Clarity or weighted information gain (WIG) techniques, may be used to predict the query performance of a base ranked list. The Clarity technique estimates query performance according to the divergence between the relevance model induced from a given ranked list and the background model induced from the corpus of documents. The WIG technique estimates query performance according to the difference between the average retrieval score of documents in a given ranked list and that of the corpus of documents. Other QPP techniques may also be used to predict the query performance of the base ranked list, such as the Normalized Query Commitment (NQC) or Query-Feedback (QF) techniques.
At step 430, the information retrieval application 120 determines a respective asymmetric co-relevance between each base ranked list and the fused list. That is, the p(LF|Li,r) term in equation (1) is determined. In one embodiment, the respective asymmetric co-relevance between each base ranked list and the fused list is determined by adjusting a symmetric co-relevance of the base ranked list and the fused list using an odds ratio between the symmetric co-relevance of the base ranked list and the fused list to the base ranked list's own relevance, according to equation (3) in which the respective asymmetric co- relevance is {circumflex over (p)}(LF|Li,r) Or(LF,Lisim(Li, LF). The symmetric co-relevance of the base ranked list and the fused list may be determined using any feasible technique, such as by simply measuring the documents that the base ranked list and the fused list include or using a statistical measure that examines the difference between lengths of documents. In one embodiment, the symmetric co-relevance of the base ranked list and the fused list may be determined using the Rank-Biased Overlap (RBO) measure. The information retrieval application 120 may determine the odds ratio used to adjust the symmetric co-relevance according to equation (6) as
with p(d|L)
measuring a relationship between document d and list L and being estimated as, e.g., a Bhattacharyya similarity between unsmoothed language models {circumflex over (p)}(d|L) Σw√{square root over ({circumflex over (p)}(w|θd){circumflex over (p)}(w|θL))}.or any other suitable similarity; p(r|L) measuring a likelihood that list L contains any relevant information and being estimated as, e.g., an entropy of its unsmoothed language model {circumflex over (p)}(r|L)
−Σw{circumflex over (p)}(w|θL) log {circumflex over (p)}(w|θL); and p(d,r|Li) measuring a likelihood that a relevant document is observed in list Li and being estimated as, e.g., information gained by generating a relevance model induced from list L, denoted as
from document d's unsmoothed language model to generating the relevence model from a background corpus model
At step 440, the information retrieval application 120 determines a likelihood that the fused list is relevant to the query based on the respective likelihood that each base ranked list includes information relevant to the query determined at step 420 and the respective asymmetric co-relevance between each base ranked list and the fused list determined at step 430. The likelihood that the fused list is relevant to the query may be estimated according to equation (1) as {circumflex over (p)}(LF|q, r) ΣL
At step 520, the information retrieval application 120 processes the received query using multiple information retrieval techniques to obtain base ranked lists and, optionally, multiple fusion techniques to obtain fused lists that each combine a subset of (which may include all of) the base ranked lists. Once again, any technically feasible information retrieval and fusion (e.g., CombSum, CombMNZ, Borda, RFF, etc.) techniques may be used by the information retrieval application 120 to process the received query.
At step 530, the information retrieval application 120 determines a predicted query performance of each of the fused lists obtained at step 520, while accounting for asymmetric list co-relevance. In one embodiment, the information retrieval application 120 may perform the method 400, described above with respect to
At step 540, the information retrieval application 120 selects a subset of base ranked lists and/or a fusion technique used to generate one of the fused lists that is associated with a highest query performance prediction. That is, the information retrieval application 120 optimizes which base ranked lists are combined and/or which fusion technique is used to generate the final fused list that is returned in order to maximize the predicted query performance. If the fusion technique is fixed, then each subset of base ranked lists that is chosen will produce a different fused list, and the information retrieval application 120 may use QPP techniques disclosed herein to select a subset of base ranked lists that gives a highest predicted quality. Alternatively, the fusion technique may be changed while not changing the base ranked lists being combined. For each different fusion technique (e.g., CombMNZ, Comb SUM, RRF, etc.), the information retrieval application 120 may again use QPP techniques disclosed herein to select the fusion technique that gives a highest predicted quality. In yet another embodiment, a hybrid option may be used where both the subset of base ranked lists and the fusion technique are changed, and information retrieval application 120 uses QPP techniques disclosed herein to select the best (subset of base ranked lists and fusion technique) configuration.
At step 550, the information retrieval application 120 causes the fused list that is associated with the highest determined likelihood to be output to a user via a display device. Continuing the example in which the information retrieval application 120 is a web search engine that processes a search query received over the Internet by generating fused lists and predicts query performances of those fused lists, the information retrieval application 120 may serve a web page including a portion of the fused list (e.g., including a given number of highest ranked documents in the fused list) to a user's computing device, thereby causing the web page including the fused list to be loaded by a web browser and displayed to the user via a display device.
The CPU 605 retrieves and executes programming instructions stored in the memory 620. Similarly, the CPU 605 stores and retrieves application data residing in the memory 620. The interconnect 617 facilitates transmission, such as of programming instructions and application data, between the CPU 605, I/O device interface 610, storage 630, network interface 615, and memory 620. The CPU 605 is included to be representative of a single CPU, multiple CPUs, a CPU having multiple processing cores, one or more graphics processing units (GPUs), a combination of such processors, or any other processor(s). And the memory 620 is generally included to be representative of a random access memory. The storage 630 may be a disk drive storage device. Although shown as a single unit, the storage 630 may be a combination of fixed and/or removable storage devices, such as magnetic disk drives, flash drives, removable memory cards or optical storage, network attached storage (NAS), or a storage area-network (SAN). Further, the system 600 is included to be representative of a physical computing system as well as virtual machine instances hosted on a set of underlying physical computing systems. Further still, although shown as a single computing system, one of ordinary skill in the art will recognized that the components of the system 600 shown in
As shown, the memory 620 includes an operating system 621 and the information retrieval application 120. The operating system 621 may be, e.g., Linux® or Windows®. The information retrieval application 120 is configured to process queries using a fusion-based query performance prediction technique that accounts for asymmetric list co-relevance. In one embodiment, the information retrieval application 120 may receive a query; process the received query using multiple information retrieval techniques to obtain base ranked lists and, optionally, multiple fusion techniques to obtain fused lists that each combine a subset of the base ranked lists; determine a predicted query performance of each of the fused lists, while accounting for asymmetric list co-relevance; select a subset of base ranked lists and/or a fusion technique used to generate one of the fused lists that is associated with a highest query performance prediction; and cause the fused list that is associated with the highest determined likelihood to be output to a user via a display device, according to the method 500 described above with respect to
Although discussed above primarily with respect to fusion-based retrieval, the asymmetric list co-relevance described herein may also be adapted to model asymmetric list co-relevance relationships for query performance prediction in other retrieval settings that include reference lists and are multi-stepped in their nature. Examples of such retrieval settings include the reranking, cascade-retrieval, and diversification settings. Reranking may include generating a base ranked list and reranking the documents therein (as opposed to fusing the base ranked list with other base ranked lists) to generate a final ranked list. Cascade retrieval may include using ranking functions to progressively prune documents and refine the rank order of non-pruned documents. Diversification may include generating a base ranked list and modifying the base ranked list by pushing to the top of the list not only documents containing relevant information but also documents that are different from each other.
Advantageously, techniques are disclosed herein for query performance prediction that account for asymmetric co-relevance relationships between ranked lists. Experience has shown that, in the fusion-based retrieval setting, the QPP techniques disclosed herein are more accurate and consistent (less sensitive to the fusion technique used to generated a fused list) than traditional QPP techniques that assume symmetric list co-relevance. Further, the QPP techniques disclosed herein may be used in the fusion-based retrieval setting to select a subset of reference lists to fuse from a set of reference lists and/or to select a fusion technique to use from multiple available fusion techniques. As a result, a fused list that is associated with a highest QPP score for a given query may be generated and a portion thereof output to a user via a display device.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
In the foregoing, reference is made to embodiments presented in this disclosure. However, the scope of the present disclosure is not limited to specific described embodiments. Instead, any combination of the foregoing features and elements, whether related to different embodiments or not, is contemplated to implement and practice contemplated embodiments. Furthermore, although embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the scope of the present disclosure. Thus, the foregoing aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s). Likewise, reference to “the invention” shall not be construed as a generalization of any inventive subject matter disclosed herein and shall not be considered to be an element or limitation of the appended claims except where explicitly recited in a claim(s).
Aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.”
The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
Embodiments of the invention may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as a computing capability that provides an abstraction between the computing resource and its underlying technical architecture (e.g., servers, storage, networks), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows a user to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in “the cloud,” without regard for the underlying physical systems (or locations of those systems) used to provide the computing resources.
Typically, cloud computing resources are provided to a user on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g. an amount of storage space consumed by a user or a number of virtualized systems instantiated by the user). A user can access any of the resources that reside in the cloud at any time, and from anywhere across the Internet. In context of the present invention, a user may access applications (e.g., the information retrieval application 120) or related data available in the cloud. For example, the information retrieval application 120 could execute on a computing system in the cloud. In such a case, the information retrieval application 120 could process queries using a fusion based query performance prediction technique that accounts for asymmetric list co-relevance and store the results at a storage location in the cloud, or transmit the results to a user's computing device. Doing so allows a user to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).
While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.
