Patent Grant
9110975
The present disclosure relates to ranking of search results.
Internet search engines aim to identify documents or other items that are relevant to a user's needs and to present the documents or items in a manner that is most useful to the user. Such activity often involves a fair amount of mind-reading—inferring from various clues what the user wants. Certain clues may be user specific. For example, knowledge that a user is making a request from a mobile device, and knowledge of the location of the device, can result in much better search results for such a user.
Clues about a user's needs may also be more general. For example, search results can have an elevated importance, or inferred relevance, if a number of other search results link to them. If the linking results are themselves highly relevant, then the linked-to results may have a particularly high relevance. Such an approach to determining relevance, generally associated with the GOGGLE® PageRank technology, is premised on the assumption that, if authors of web pages felt that another web site was relevant enough to be linked to, then web searchers would also find the site to be particularly relevant. In short, the web authors “vote up” the relevance of the sites.
Other various inputs may be used instead of, or in addition to, such techniques for determining and ranking search results. For example, user reactions to particular search results or search result lists may be gauged, so that results on which users often click will receive a higher ranking. The general assumption under such an approach is that searching users are often the best judges of relevance, so that if they select a particular search result, it is likely to be relevant, or at least more relevant than the presented alternatives.
Systems, methods, and apparatus including computer program products for ranking search results of a search query are described. In general, particular inputs may be generated or analyzed to affect the presentation of search results. For example, such inputs may increase the relevance that a system will assign to a particular result in a particular situation, thus boosting the score or other indicator of relevance for the result (and perhaps the relevance of the result in the context of a particular query). Such an approach may benefit a user by providing them with search results that are more likely to match their needs. As a result, users can learn more using the internet, can find more relevant information more quickly, and will thus achieve more in their work or elsewhere, and will be more likely to use such a system again. A provider of such services may also benefit, by providing more useful services to users, and by thereby inducing more traffic to their search services. Such additional traffic may provide an operator with additional revenue, such as in the form of advertising that accompanies the searching and the delivery of search results. In general, one aspect of the subject matter described in this specification can be embodied in a method that includes for a first document identified as a search result of a user user-submitted query, generalizing the user-submitted query by forming one or more variants of the user-submitted query to generate one or more other queries, each of the one or more other queries being different from the user-submitted query. A generalized quality of result statistic is generated for the first document from respective data associated with each of the other queries, each respective data being indicative of user behavior relative to the first document as a search result for the associated other query. The generalized quality of result statistic is provided as the quality of result statistic input to the document ranking process for the first document and the user-submitted query. Other embodiments of this aspect include corresponding systems, apparatus, and computer program products.
These and other embodiments can optionally include one or more of the following features. The data can be aggregated click data. The data associated with each of the other queries can be weighted. The quality of result statistic can be modified based on a level of certainty for each of the generated one or more queries. A variant of the user-submitted query is formed by performing one or more of: replacing one or more terms in the user-submitted query with synonyms, changing an order of one or more terms in the user-submitted query, or replacing one or more query terms in user-submitted query with stems. Particular embodiments of the described subject matter can be implemented to realize one or more of the following advantages. User queries can be generalized to other queries in order to take advantage of user behavior data associated with the other queries. Patterns of user interaction can be identified that indicate a user has refined a query to obtain a desired document. Based on the patterns of user interaction, user behavior data associated with the refined query can be propagated to earlier queries in the user's session. Data associated with prior queries indicative of user behavior can be combined to create a quality of result statistic that can be input to a document ranking A document's rank can be boosted in a result list for a user query in which there is no model data. User behavior statistics associated with a query having a small amount of aggregated user behavior data can take advantage of the user behavior data associated with a similar query.
The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims.
FIG. 5E1 is a diagram of generalization of a query through removal of optional terms.
FIG. 5E2 is a flow chart of a method for optional term generalization.
Like reference numbers and designations in the various drawings indicate like elements.
A user 1002 (1002a, 1002b, 1002c) can interact with the system 1000 through a client device 1004 (1004a, 1004b, 1004c) or other device. For example, the client device 1004 can be a computer terminal within a local area network (LAN) or wide area network (WAN). The client device 1004 can include a random access memory (RAM) 1006 (or other memory and/or a storage device) and a processor 1008. The processor 1008 is structured to process instructions within the system 1000. In some implementations, the processor 1008 is a single-threaded processor. In other implementations, the processor 1008 is a multi-threaded processor. The processor 1008 can include multiple processing cores and is structured to process instructions stored in the RAM 1006 (or other memory and/or a storage device included with the client device 1004) to display graphical information for a user interface.
A user 1002a can connect to the search engine 1030 within a server system 1014 to submit a query 1015. When the user 1002a submits the query 1015 through an input device attached to a client device 1004a, a client-side query signal 1010a is sent into a network 1012 and is forwarded to the server system 1014 as a server-side query signal 1010b. Server system 1014 can be one or more server devices in one or more locations. A server device 1014 includes a memory device 1016, which can include the search engine 1030 loaded therein. A processor 1018 is structured to process instructions within the device 1014. These instructions can implement one or more components of the search engine 1030. The processor 1018 can be a single-threaded processor or a multi-threaded processor, and can include multiple processing cores. The processor 1018 can process instructions stored in the memory 1016 related to the search engine 1030 and can send information to the client device 1004, through the network 1012, to create a graphical presentation in a user interface of the client device 1004 (e.g., a search results web page displayed in a web browser).
The server-side query signal 1010b is received by the search engine 1030. The search engine 1030 uses the information within the user query 1015 (e.g. query terms) to find relevant documents. The search engine 1030 can include an indexing engine 1020 that actively searches a corpus (e.g., web pages on the Internet) to index the documents found in that corpus, and the index information for the documents in the corpus can be stored in an index database 1022. This index database 1022 can be accessed to identify documents related to the user query 1015. Note that, an electronic document (which for brevity will simply be referred to as a document) does not necessarily correspond to a file. A document can be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files. Moreover, a document can be stored in a memory without having first been stored in file.
The search engine 1030 can include a ranking engine 1052 to rank the documents related to the user query 1015. The ranking of the documents can be performed using traditional techniques for determining an information retrieval (IR) score for indexed documents in view of a given query. The relevance of a particular document with respect to a particular search term or to other provided information may be determined by any appropriate technique. For example, the general level of back-links to a document that contains matches for a search term may be used to infer a document's relevance. In particular, if a document is linked to (e.g., is the target of a hyperlink) by many other relevant documents (e.g., documents that also contain matches for the search terms), it can be inferred that the target document is particularly relevant. This inference can be made because the authors of the pointing documents presumably point, for the most part, to other documents that are relevant to their audience.
If the pointing documents are in turn the targets of links from other relevant documents, they can be considered more relevant, and the first document can be considered particularly relevant because it is the target of relevant (or even highly relevant) documents. Such a technique may be the determinant of a document's relevance or one of multiple determinants. The technique is exemplified in the GOGGLE® PageRank system, which treats a link from one web page to another as an indication of quality for the latter page, so that the page with the most such quality indicators wins. Appropriate techniques can also be taken to identify and eliminate attempts to cast false votes so as to artificially drive up the relevance of a page.
To further improve such traditional document ranking techniques, the ranking engine 1052 can receive an additional signal from a rank modifier engine 1056 to assist in determining an appropriate ranking for the documents. The rank modifier engine 1056 provides one or more measures of relevance for the documents, which can be used by the ranking engine 1052 to improve the search results' ranking provided to the user 1002. The rank modifier engine 1056 can perform one or more of the operations described further below to generate the one or more measures of relevance.
The search engine 1030 can forward the final, ranked result list within a server-side search results signal 1028a through the network 1012. Exiting the network 1012, a client-side search results signal 1028b can be received by the client device 1004a where the results can be stored within the RAM 1006 and/or used by the processor 1008 to display the results on an output device for the user 1002a.
In various implementations, the time (T), also known as “click data”, can be measured as the time between the initial click through to the document result until the time the user comes back to the main page and clicks on another document result. In general, an assessment is made about the time (T) regarding whether this time indicates a longer view of the document result or a shorter view of the document result, since longer views are generally indicative of quality or relevance for the clicked through result. This assessment about the time (T) can further be made in conjunction with various weighting techniques.
The components shown in
A user can submit a query 5106a “celica”, for example, to a search engine through a graphical user interface 5109, as might be presented on a personal computer, a mobile telephone, or other device. A query includes one or more terms. For example, a query of a recipe database can include the terms “fast”, “fiber” and “switch”. In addition to dictionary words, terms can include special characters, numbers, mathematical expressions, Boolean expressions, slang terms, names, titles, images, sounds, videos, other suitable query terms, and combinations of these. Moreover, query terms can be in more than one language.
In response to the user selecting the search button 5122, for example, a search engine returns a ranking or result list 5108 which is an ordered list of references to documents that satisfy the query 5106a. The result list 5108 includes a set of document references URL A, URL B, URL C 5110a, and URL D. The result list 5108 can contain the text of the URL itself, a short description of the information found within each document, a snapshot of the portion of the document which contains the query, other suitable information, or a combination of these. If a user selects (e.g., clicks) URL C 5110a from the result list 5108, for example, the selected URL can cause the user interface 5109 (e.g., a web browser) to present the associated document 5112. Once the user has finished viewing the document, the user can navigate back to the result list 5108.
In various implementations, the model 5102 can be built as users interact with search engines. For example, a timer can track how long a user views or “dwells” on the document 5112. The amount of time 5104 is referred to as “click data”. For example, a longer time spent dwelling on a document, termed a “long click”, can indicate that a user found the document to be relevant for their query. A brief period viewing a document, termed a “short click”, can be interpreted as a lack of document relevance. In various implementations, the click data 5114 is a count of each click type (e.g., long, medium, short) for a particular query 5106 and document 5110 combination. Aggregated click data from model queries for a given document can be used to create a quality of result statistic for that document and can be used to enhance a ranking of that document. By way of illustration, a quality of result statistic can be a weighted average of the count of long clicks for a given document and query.
A search engine or other process creates a record 5116 in the model 5102 for documents that are selected by users in response to a query. Each record 5116 within the model 5102 (herein referred to as a tuple: <document, query, data>) is at least a combination of a query 5106 submitted by users, a document reference 5110 selected by users in response to that query, and an aggregation of click data 5114 for all users that select the document reference 5110 in response to the query 5106. The aggregate click data can be viewed as an indication of document relevance. In various implementations, model data can be location-specific (e.g. country, state, etc) or language-specific. For example, a country-specific tuple would include the country from where the user query originated from in whereas a language-specific tuple would include the language of the user query. Other extensions of model data are possible.
If a user submits the query 5106c “Toyota celica” to a user interface 5156, rather than the query 5106a “celica”, there is no model 5102 information for this query as of yet. In this case, the search engine would return a document ranking without the benefit of user behavior data. However, if the search engine recognized that the query 5106c “Toyota celica” is topically the same as the query 5106a “celica” (or vice versa) this would allow the search engine to take advantage of the user behavior data 5114a in the model 5102 for the query “celica” 5106a. This is an example of generalizing a query.
User behavior data is combined from the matching queries to create a quality of result statistic for the document (step 5204). In further implementations, the quality of result statistic can be modified based on a level of certainty in the generalization. For example, the statistic might be down-weighted or demoted based on belief in the accuracy of the match. The quality of result statistic is then provided as input to enhance a document ranking (step 5206).
For one or more documents in the initial ranking 5254, the rank modifier 5256 determines a quality of result statistic for a given document based on model 5102 data for the user query 5106 and the given document. If the user query 5106 does not match a query in the model 5102 for the given document, a query generalizer component 5260 provides one or more generalized forms of the query 5106 to the rank modifier 5256 which the rank modifier 5256 can use to obtain model 5102 data for the document. The generalized queries are considered topically similar to the user query 5106 and will potentially match records in the model 5102 for the given document.
Although this diagram illustrates separate functional components, components can be divided into more components or integrated into fewer components. Moreover, components can interact on a single computing device or a plurality of computing devices connected by one or more networks or communication channels.
One generalization technique is to remove one or more so-called “stop words” from the user query. In various implementations, stop words are common words that typically carry no inherent meaning by themselves or whose removal does not change the topicality of the query. Examples of stop words are adverbs, conjunctions, and forms of the verb “be”. In various implementations, the fewest possible number of stop words are removed from the user query in order to obtain a match with a query associated with the selected document in the model. If there is a match (step 5308), the quality of result statistic is determined (step 5306) based on data from one or more matching tuples and additional documents can be considered for generalization (step 5302). Otherwise, a different generalization technique can be attempted.
The user query is can also be generalized by removing one or more optional terms from the user query. In various implementations, optional terms are terms that do not change the concept of a query if they are removed from the query. For example, in
Another generalization technique is to replace one or more terms in the user query with synonyms. In various implementations, a synonym is a term that is topically similar to a term in the user query. For example, “car” is a synonym of “auto”, and “photograph” is a synonym of “picture”. A synonym of a query term can be determined using, for instance, the technique described in U.S. patent application Ser. No. 11/096,726, entitled DETERMINING QUERY TERM SYNONYMS WITHIN QUERY CONTEXT, filed Mar. 31, 2005, the contents of which are incorporated herein in their entirety. In various implementations, the fewest possible number of terms in the user query are replaced synonyms order to obtain a match with a query associated with the selected document in the model. If there is a match (step 5310), the quality of result statistic is determined (step 5306) based on data from one or more matching tuples and additional documents can be considered for generalization (step 5302). Otherwise, a different generalization technique can be attempted.
Another generalization technique is to replace one or more terms in the user query with their stems. In various implementations, a stem is a different pluralization or form of speech of a term in the user query. For example, the term “matches” contains the stem term “match”. In various implementations, the fewest possible number of user query terms are replaced with their stems in order to match a query associated with a document in the model. If there is a match (step 5312), the quality of result statistic is determined (step 5306) based on data from one or more matching tuples and additional documents can be considered for generalization (step 5302). Otherwise, a different generalization technique can be attempted.
Another generalization technique is to reorder one or more terms in the user query. That is, this technique creates a generalized query that is a permutation of the user query. A permutation is only applicable to a user query comprised of more than one term. For example, the query “discount airfare Orlando” can be rearranged to read “Orlando discount airfare”, “discount Orlando airfare”, etc. If a permutation of the user query matches one or more tuples within the model (step 5314), the data from the tuple(s) is used to determine the quality of result statistic (step 5306) and additional documents can be considered for generalization (step 5302). In various implementations, permutations of other generalized queries can also be attempted. Otherwise, a different generalization technique can be attempted.
Another generalization technique is to match part of the user query with part of a query associated with the selected document in the model. A partial query match occurs when one or more user query terms match a one or more terms of one or more model queries for a selected document. For example, the query “election candidates Seattle Wash.” is a partial query match of the query “mayoral candidates Seattle”, because both queries contain the terms “candidates” and “Seattle”. If any portion of the user query is found to be a match for a portion of one or more model queries (step 5316), the data from the one or more matching tuples used to determine the quality of result statistic (step 5306) and additional documents can be considered for generalization (step 5302). If none of the generalization techniques produces a matching query, the process can be repeated for other documents (step 5302). Although generalizations occur in a particular sequence in this illustration, other sequences are possible including sequences with fewer or more generalization techniques. Moreover, one or more of the generalization techniques can be performed in parallel rather than serially.
In various implementations, a user query can be generalized in a plurality of ways such that the “best” generalization is determined. For a given user query and document, a plurality of generalizations (including combinations of generalizations) are determined for the user query. The generalized query with the highest score is considered the best match for the user query.
In various implementations, a plurality of generalizations can be considered by computing a so-called match belief score for each query in a model for a given document. The match belief score is based on the types of generalizations required to match a user query to one in the model for the document in question. The match belief score is a measure of the similarity between the user query and a query associated with the document in the model. For example, the match belief can be demoted for any generalization required to match the two queries. Therefore, a match that only requires a synonym substitution will be considered better than one that requires a synonym substitution and a stop word removal, for example. Generally speaking, the more generalizations that are applied, the further the match belief score is degraded. After the match belief scores are computed for all queries, the one with the highest score is chosen as the best match for the input query.
In various implementations, the match belief score can be used to down-weight or demote data associated with the chosen query, the quality of result statistic, or both. For example, if the match belief score is low, these values can be down weighted to reflect a low level of confidence in the matched query, and vice versa.
By way of illustration, TABLE 1 lists queries associated with a document in the model and corresponding match belief scores given the user query “pictures of fall color”. A match belief score between 0.0 and 1.0 is determined for each query associated with the document (e.g., 0.0 for no match, 1.0 for an exact match).
FIG. 5E1 is a diagram of generalization of a query through removal of optional terms. As described above, optional terms are terms whose removal from a query will not change the query's concept. For example, if the query 5407 “Toyota celica hatchback” has the optional terms “Toyota” and “hatchback”, the query 5407 can be generalized by removing one or both of the terms. Generalized query 5403 “celica hatchback” was created by removing the optional term “Toyota” where as generalized query 5405 “celica” was created by removing both optional terms.
FIG. 5E2 is a flow chart of a method for optional term generalization. If a user query does not match a <document, query, data> tuple in the model, the query generalizer component 5260 (or another process) is invoked to remove one or more optional terms from the user query (step 5409). Next, it is determined whether or not the generalized user query matches a tuple in the model (step 5411). In various implementations, as few optional terms as possible are removed from the user query to obtain a match. If no match is found, one or more additional or alternative optional terms can be removed from the user query (step 5409). Otherwise, a quality of result statistic is determined based on one or more matching tuple's data (step 5413). Optionally the quality of result statistic can be modified to reflect a level of certainty in the generalized query (step 5417). For example, a level of certainty can be based on similarity in meaning between the modified query and the user query as determined using a probabilistic model such as a Bayesian belief network or through other techniques. Other techniques for determining a level of certainty are possible.
As described above, a model contains tuples that each include at least a query, a document selected in response to the query, and data indicative of user behavior. For example, document 5404 is associated with queries 5406, 5408 and 5410 which could be represented as three tuples in a model, each having the same document reference (5404), a different query (5406, 5408 or 5410), and associated user behavior data. By way of illustration, if a user query 5402 “Jack in the box” was submitted to a search engine and, an initial ranking contained document 5404, there is no tuple for that document/query combination. However, by removing one or more stop words from the user query 5402, a match is possible. In this way, model coverage for generalized queries is increased. Initially, the stop word “the” is removed from the user query 5402 to create the generalized query “jack in box”. The generalized query matches query 5406. As such, the user behavior data associated with the tuple <document (5404), query (5406), data> can be used to determine a quality of result statistic for document 5404. If removal of the stop word “the” from the user query 5402 did not create a match, for example, an alternate stop word could be removed instead, such as “in” to create the generalized query 5408 “jack the box”. If this did not create a match, both stop words could be removed to create the generalized query 5410 “jack box”. In this way, stop word generalization attempts to find a match in the model for a query and a document by removing the fewest number of stop words from the user query.
T1:<document1,query1,data1>
is first created in a model (or sometime afterwards) such as in response to a user selecting a document in a result list (e.g., 5108), a new tuple can automatically be created:
T1′:<document1,query1′,data1>.
The new tuple T1′ has the same document and data as T1, but the query1′ is a version of query1 with all of the stop words removed. In addition, each time data1 is modified in T1 the modification can be automatically propagated to data1 associated with T1′. Preemptively creating versions of tuples without stop words means there will be a match of the non-stop word versions of queries.
Method 5450 captures this idea. Initially, all stop words are removed from a user query to create a modified user query (step 5452), such as query1′ described above. The modified user query is stored in the model along with the user query (e.g., query1) for the same document (e.g., document1; step 5454). User behavior data (e.g., data1) associated with the user query is then forwarded to the modified user query in the model for the same document (step 5456).
By way of illustration, if a first user submits the first query “motorcycling in the Black Hills of South Dakota”, a tuple is created within the model containing the user query, and a second tuple is created containing a modified version of the user query: “motorcycling Black Hills South Dakota”. When a second user submits the second query “motorcycling through Black Hills South Dakota”, the second query can be reduced to the same modified query “motorcycling Black Hills South Dakota”. This provides the second user with the opportunity to benefit from the user behavior data gathered from the previous user's query, even though the users did not submit identical queries. By combining user behavior data, a level of certainty in a quality of result statistic is improved.
In various implementations, model data for a query and a non stop word version of the query can be merged in the model if their data is similar. This can provide space savings in the model. A technique for doing so is provided by method 5453. Two queries are selected from the model, where one of the queries is a non stop word version of the other (step 5455). It is then determined whether the two queries' user behavior data is similar (step 5457). A metric can be used to determine similarity of the data. For example, the metric can be the normal of a vector of aggregated user behavior data. If the queries have similar data, the queries are merged into a single query (e.g., the non stop word version of the user query; step 5459). Finally, if there are more queries in the model to consider (step 5411), the process continues at step 5455. This method can be performed on a periodic or as needed basis. In various embodiments, the approach outlined in
For example, a user query 5502 containing the terms “car stores” can be generalized to different variant forms. The term “stores” in the user query 5502 can be reduced to its stem “store” to produce a variant query 5506, “car store”. The term “car” in the user query 5502 can be substituted for a synonym “auto”, resulting in a variant query 5508 “auto stores”. The synonym “auto” can be kept and the term “stores” can be reduced to its stem form, creating a variant query 5510, “auto store”. Finally, the user query can be reordered to create a variant query 5512, “stores car”. Other variants are possible.
When a match is found for the generalized user query (step 5554), a quality of result statistic is determined based on the one or more matching tuples' user behavior data (step 5556). The quality of result statistic is then optionally modified (e.g., demoted or down-weighted) based on a level of certainty in the variant's similarity in meaning to the original query terms (step 5558).
In a first form of partial query matching, the user query terms are a subset of the model query terms. For example, a user query “tea party” is a subset of a model query “Wonderland tea party”. Alternatively, model query terms can be a subset of user query terms. For example, a user query “photos of the golden gate bridge” includes all terms found within a model query “golden gate bridge”. A subset of the user query can also match a subset of a model query. For example, a user query “lions tigers” and a model query “tigers bears” share the subset of terms “tigers”. Other forms of partial query matching are possible.
In various implementations, in determining a partial query match the edit distance between a user query and a model query is calculated. The edit distance between a user query and a model query can be determined based on Levenshtein distance. The edit distance is the minimum number of operations needed to transform one query into the other, where an operation is an insertion, deletion, or substitution of a term. In some implementations, the edit distance can be determined at the character or semantic concept level. For example, the query “child's metal wagon” has an edit distance of three from the query “red wagon”, because the terms “child's” and “metal” would have to be dropped, as well as the term “red” added to the first query, so that the two queries would be identical. Edit distance can be used to determine a level of certainty in a model query being a good indicator of relevance for a user query.
For example, a set of queries 5606, 5610, 5612, 5616, 5620, and 5624 are associated with the document 5604 in a model. The query 5606 “credit card gold”, is a superset of the user query 5602 “credit card”. An edit distance 5608 between the model query 5606 and the user query 5602 of one is calculated by counting the term “gold” which only occurs within the model query 5606. There are no common terms between the user query 5602 “credit card” and the model query 5610 “paper boy”. Therefore, the model query 5610 is not a partial match. The model query 5612 “credit card platinum gold” is a superset of the user query 5602 “credit card”. There is an edit distance 5614 of two between the model query 5612 and the user query 5602 because both the terms “gold” and “platinum” only occur within the model query 5612.
A subset of the terms within the model query 5616 “credit rating” matches a subset of the terms within the user query 5602. An edit distance 5618 between the model query 5616 and the user query 5602 is calculated as two, because the term “rating” would need to be both removed from the model query 5616 and replaced with the term “card” for the two queries to match. The model query 5620 “card” is a subset of the user query 5602 “credit card”. An edit distance 5622 of one exists between them because the term “credit” would have to be added to the model query 5620 to cause the two queries to match. There are no common terms between the model query 5624 “san Francisco chronicle” and the user query 5602 “credit card” therefore the model query 5624 does not qualify as a partial match.
The exponential decay models the loss of the semantic context when terms are missing and can prevent an overestimate of the relevance of loose matches. In various implementations, a down-weighted dw click fraction ƒ is defined as:
dƒ(nq,uq)=ƒ(nq,uq)×(1+ed(nq,uq))k
where nq is a partial match model query, uq is the user query, ed is a function returning the edit distance between nq and uq, and k is a constant greater than one. The function ƒ is a so-called click fraction which is a weighted average of click data for a given tuple. By way of illustration, ƒ is defined as:
where #wc is the sum of weighted clicks for the query-document pair, and #c is the total number of clicks for the query-document pair. However, other definitions off are possible. Each down-weighted click fraction represents a relative contribution of a partial query match that covers the entire user query or a subset of the user query. A vector T containing an element for each term t in the user query can store the sum of all down-weighted click fractions of partial matches related to t (step 5656):
T[t]=Σdƒ(nq,uq) where tεnq
The values stored in T are normalized and placed in a vector V (step 5658). For example, each value T[t] can be normalized by the sum of all the terms within the user query.
The final click fraction ƒƒ for the document can be calculated as follows:
where a is a constant factor that determines how much the final click fraction is reduced (step 5660).
In various implementations, if the user finds the document they are looking for in response to a refined query (e.g., as indicated by a long click) within a user session, the click data is propagated to the user behavior data for the original query associated with that document. In various implementations, a user session is defined by time as well as an indicator that the user has moved onto to a topically different or unrelated query. For example, if a user's initial query was “Toyota celica” and their subsequent query is “best recipes”, a new session is defined for the query “best recipes” since this query is topically unrelated to the original query. By way of further illustration, if the first query and the second query have query terms in common, it is likely that the two are unrelated. For example, if a user submits a first query 5802 “supercar” to a user interface 5808 at a first time, a search engine responds with a result list 5804. At a later time, the user submits a refined query 5806 to the user interface 5808. This time, the result list 5810 contains a document 5810b that the user finds relevant. The user behavior data for refined query and document 5810b combination is shared with the original query 5802 and document 5810b combination.
The memory 6016 is a computer readable medium such as volatile or non volatile that stores information within the system 6050. The memory 6016 can store processes related to the functionality of the search engine 1030, for example. The storage device 6052 is capable of providing persistent storage for the system 6050. The storage device 6052 can include a floppy disk device, a hard disk device, an optical disk device, or a tape device, or other suitable persistent storage mediums. The storage device 6052 can store the various databases described above. The input/output device 6054 provides input/output operations for the system 6050. The input/output device 6054 can include a keyboard, a pointing device, and a display unit for displaying graphical user interfaces.
The computer system shown in
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Embodiments of the invention can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the invention, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
While this specification contains many specifics, these should not be construed as limitations on the scope of the invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of the invention. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Thus, particular embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. Moreover, the server environment, which is configured to provide electronic search service and employ the ranking systems and techniques described, need not be implemented using traditional back-end or middleware components. The server environment can be implemented using a program installed on a personal computing apparatus and used for electronic search of local files, or the server environment can be implemented using a search appliance (such as GOGGLE® in a Box, provided by Google Inc. of Mountain View, Calif.) installed in an enterprise network.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5265065 | Turtle | Nov 1993 | A |
| 6014665 | Culliss | Jan 2000 | A |
| 6076051 | Messerly et al. | Jun 2000 | A |
| 6078916 | Culliss | Jun 2000 | A |
| 6182068 | Culliss | Jan 2001 | B1 |
| 6327590 | Chidlovskii et al. | Dec 2001 | B1 |
| 6363378 | Conklin et al. | Mar 2002 | B1 |
| 6480843 | Li | Nov 2002 | B2 |
| 6490575 | Berstis | Dec 2002 | B1 |
| 6539377 | Culliss | Mar 2003 | B1 |
| 6671681 | Emens et al. | Dec 2003 | B1 |
| 6701309 | Beeferman et al. | Mar 2004 | B1 |
| 6738764 | Mao et al. | May 2004 | B2 |
| 6792416 | Soetarman et al. | Sep 2004 | B2 |
| 6816850 | Culliss | Nov 2004 | B2 |
| 6944612 | Roustant et al. | Sep 2005 | B2 |
| 7085761 | Shibata | Aug 2006 | B2 |
| 7113939 | Chou et al. | Sep 2006 | B2 |
| 7146361 | Broder et al. | Dec 2006 | B2 |
| 7222127 | Bem et al. | May 2007 | B1 |
| 7231399 | Bem et al. | Jun 2007 | B1 |
| 7243102 | Naam et al. | Jul 2007 | B1 |
| 7379951 | Chkodrov et al. | May 2008 | B2 |
| 7426507 | Patterson | Sep 2008 | B1 |
| 7451487 | Oliver et al. | Nov 2008 | B2 |
| 7516146 | Robertson et al. | Apr 2009 | B2 |
| 7526470 | Karnawat et al. | Apr 2009 | B1 |
| 7533130 | Narayana et al. | May 2009 | B2 |
| 7693818 | Majumder | Apr 2010 | B2 |
| 7747612 | Thun et al. | Jun 2010 | B2 |
| 7783632 | Richardson et al. | Aug 2010 | B2 |
| 7801885 | Verma | Sep 2010 | B1 |
| 7809716 | Wang et al. | Oct 2010 | B2 |
| 7818320 | Makeev | Oct 2010 | B2 |
| 7836058 | Chellapilla | Nov 2010 | B2 |
| 7844589 | Wang et al. | Nov 2010 | B2 |
| 7849089 | Zhang et al. | Dec 2010 | B2 |
| 7853557 | Schneider et al. | Dec 2010 | B2 |
| 7877404 | Achan et al. | Jan 2011 | B2 |
| 7895177 | Wu | Feb 2011 | B2 |
| 7953740 | Vadon et al. | May 2011 | B1 |
| 7974974 | Tankovich et al. | Jul 2011 | B2 |
| 7987185 | Mysen et al. | Jul 2011 | B1 |
| 8024330 | Franco et al. | Sep 2011 | B1 |
| 8027439 | Zoldi et al. | Sep 2011 | B2 |
| 8037042 | Anderson et al. | Oct 2011 | B2 |
| 8037043 | Zoeter et al. | Oct 2011 | B2 |
| 8051061 | Niu et al. | Nov 2011 | B2 |
| 8065296 | Franz et al. | Nov 2011 | B1 |
| 8073263 | Hull et al. | Dec 2011 | B2 |
| 8073772 | Bishop et al. | Dec 2011 | B2 |
| 8086599 | Heymans | Dec 2011 | B1 |
| 8156111 | Jones et al. | Apr 2012 | B2 |
| 20020034292 | Tuoriniemi et al. | Mar 2002 | A1 |
| 20020103790 | Wang et al. | Aug 2002 | A1 |
| 20020138478 | Schwartz et al. | Sep 2002 | A1 |
| 20030009399 | Boerner | Jan 2003 | A1 |
| 20030018707 | Flocken | Jan 2003 | A1 |
| 20030028529 | Cheung et al. | Feb 2003 | A1 |
| 20030037074 | Dwork et al. | Feb 2003 | A1 |
| 20030167252 | Odom et al. | Sep 2003 | A1 |
| 20030195877 | Ford et al. | Oct 2003 | A1 |
| 20030220913 | Doganata et al. | Nov 2003 | A1 |
| 20040006740 | Krohn et al. | Jan 2004 | A1 |
| 20040034632 | Carmel et al. | Feb 2004 | A1 |
| 20040083205 | Yeager | Apr 2004 | A1 |
| 20040093325 | Banerjee et al. | May 2004 | A1 |
| 20040158560 | Wen et al. | Aug 2004 | A1 |
| 20040186996 | Gibbs et al. | Sep 2004 | A1 |
| 20040199419 | Kim et al. | Oct 2004 | A1 |
| 20040215607 | Travis, Jr. | Oct 2004 | A1 |
| 20050015366 | Carrasco et al. | Jan 2005 | A1 |
| 20050021397 | Cui et al. | Jan 2005 | A1 |
| 20050060290 | Herscovici et al. | Mar 2005 | A1 |
| 20050060310 | Tong et al. | Mar 2005 | A1 |
| 20050060311 | Tong et al. | Mar 2005 | A1 |
| 20050071741 | Acharya et al. | Mar 2005 | A1 |
| 20050102259 | Kapur | May 2005 | A1 |
| 20050102282 | Linden | May 2005 | A1 |
| 20050125376 | Curtis et al. | Jun 2005 | A1 |
| 20050160083 | Robinson | Jul 2005 | A1 |
| 20050198026 | Dehlinger et al. | Sep 2005 | A1 |
| 20050222987 | Vadon | Oct 2005 | A1 |
| 20050240576 | Piscitello et al. | Oct 2005 | A1 |
| 20050240580 | Zamir et al. | Oct 2005 | A1 |
| 20050256848 | Alpert et al. | Nov 2005 | A1 |
| 20060047643 | Chaman | Mar 2006 | A1 |
| 20060069667 | Manasse et al. | Mar 2006 | A1 |
| 20060074903 | Meyerzon et al. | Apr 2006 | A1 |
| 20060089926 | Knepper et al. | Apr 2006 | A1 |
| 20060095421 | Nagai et al. | May 2006 | A1 |
| 20060106793 | Liang | May 2006 | A1 |
| 20060173830 | Smyth et al. | Aug 2006 | A1 |
| 20060195443 | Franklin et al. | Aug 2006 | A1 |
| 20060200476 | Gottumukkala et al. | Sep 2006 | A1 |
| 20060200556 | Brave et al. | Sep 2006 | A1 |
| 20060253428 | Katariya et al. | Nov 2006 | A1 |
| 20070005575 | Dai et al. | Jan 2007 | A1 |
| 20070005588 | Zhang et al. | Jan 2007 | A1 |
| 20070038659 | Datar et al. | Feb 2007 | A1 |
| 20070061211 | Ramer et al. | Mar 2007 | A1 |
| 20070081197 | Omoigui | Apr 2007 | A1 |
| 20070106659 | Lu et al. | May 2007 | A1 |
| 20070112730 | Gulli et al. | May 2007 | A1 |
| 20070130370 | Akaezuwa | Jun 2007 | A1 |
| 20070156677 | Szabo | Jul 2007 | A1 |
| 20070192190 | Granville | Aug 2007 | A1 |
| 20070208730 | Agichtein et al. | Sep 2007 | A1 |
| 20070214131 | Cucerzan et al. | Sep 2007 | A1 |
| 20070233653 | Biggs et al. | Oct 2007 | A1 |
| 20070260597 | Cramer | Nov 2007 | A1 |
| 20070266021 | Aravamudan et al. | Nov 2007 | A1 |
| 20070266439 | Kraft | Nov 2007 | A1 |
| 20070288450 | Datta et al. | Dec 2007 | A1 |
| 20080010143 | Kniaz et al. | Jan 2008 | A1 |
| 20080027913 | Chang et al. | Jan 2008 | A1 |
| 20080052219 | Sandholm et al. | Feb 2008 | A1 |
| 20080052273 | Pickens | Feb 2008 | A1 |
| 20080059453 | Laderman | Mar 2008 | A1 |
| 20080077570 | Tang et al. | Mar 2008 | A1 |
| 20080082518 | Loftesness | Apr 2008 | A1 |
| 20080091650 | Fontoura et al. | Apr 2008 | A1 |
| 20080114624 | Kitts | May 2008 | A1 |
| 20080114729 | Raman et al. | May 2008 | A1 |
| 20080114750 | Saxena et al. | May 2008 | A1 |
| 20080140699 | Jones et al. | Jun 2008 | A1 |
| 20080162475 | Meggs et al. | Jul 2008 | A1 |
| 20080183660 | Szulczewski | Jul 2008 | A1 |
| 20080189269 | Olsen | Aug 2008 | A1 |
| 20080208825 | Curtis et al. | Aug 2008 | A1 |
| 20080228442 | Lippincott et al. | Sep 2008 | A1 |
| 20080256050 | Zhang et al. | Oct 2008 | A1 |
| 20080313168 | Liu et al. | Dec 2008 | A1 |
| 20080313247 | Galvin | Dec 2008 | A1 |
| 20090012969 | Rail et al. | Jan 2009 | A1 |
| 20090055392 | Gupta et al. | Feb 2009 | A1 |
| 20090157643 | Gollapudi et al. | Jun 2009 | A1 |
| 20090182723 | Shnitko et al. | Jul 2009 | A1 |
| 20090287656 | Bennett | Nov 2009 | A1 |
| 20100131563 | Yin | May 2010 | A1 |
| 20100205541 | Rapaport et al. | Aug 2010 | A1 |
| 20100241472 | Hernandez | Sep 2010 | A1 |
| 20110064795 | Tosi et al. | Mar 2011 | A1 |
| 20110295844 | Sun et al. | Dec 2011 | A1 |
| Entry |
|---|
| Agichtein, Eugene et al., “Improving Web Search Ranking by Incorporating User Behavior Information”, SIGIR '06, Aug. 6-11, 2006, Seattle, WA, pp. 19-26. |
| Agichtein, Eugene et al., “Learning User Interaction Models for Predicting Web Search Result Preferences”, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle Washington, (2006), pp. 3-10. |
| Boyan, Justin et al., “A Machine Learning Architecture for Optimizing Web Search Engines”, School of Computer Science, Carnegie Mellon University, May 10, 1996, to appear in AAAI Workshop in Internet-Based Information Systems, Oregon, 1996, pp. 1-8. |
| Cutrell, Edward et al., “Eye tracking MSN Search: Investigating snippet length, target position and task types”, Jan. 2007, pp. 1-13. |
| Joachims, Thorsten, “Optimizing Search Engines using Clickthrough Data”, Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Edmonton, Alberta, Canada (2002), pp. 133-142. |
| Kelly, Diane et al., “Implicit Feedback for Inferring User Preference: A Bibliography”, SCILS, Rutgers University, New Brunswick, NJ 08901, 11 pages. |
| U.S. Appl. No. 12/331,872, Le et al. |
| U.S. Appl. No. 13/476,875, filed May 21, 2012, Ranking Search Results Based on Similar Queries, Lopatenko et al. |
| U.S. Appl. No. 12/717,475, filed Mar. 4, 2010, Deriving Site Reputation and Focus From User Feedback Statistics, Thakur, et al. |
| U.S. Appl. No. 12/623,276, filed Nov. 20, 2009, Modifying Scoring Data Based on Historical Changes, Kim et al. |
| Bar-Llan et al., “Presentation Bias is Significant in Determining User Preference for Search Results—A User Study”; Journal of the American Society for Information Science and Technology, vol. 60, Issue 1 (p. 135-149), Sep. 2008, 15 pages. |
| Bar-Llan et al.; “Methods for comparing rankings of search engine results”; Computer Networks: The International Journal of Computer and Telecommunications Networking, Jul. 2006, vol. 50, Issue 10, 19 pages. |
| Boldi, et al.; The Query-flow Graph: Model and Applications; CKIM '08, Oct. 26-30, Napa Valley, California, USA, pp. 609-617. |
| Burke, Robin, Integrating Knowledge-based and Collaborative-filtering Recommender Systems, AAAI Technical Report WS-99-01, Compilation copyright © 1999, AAAI (www.aaai.org), pp. 69-72. |
| Craswell, et al.; “Random Walks on the Click Graph”; Jul. 2007; SIGIR '07, Amsterdam, the Netherlands, 8 pages. |
| Diligenti, et al., Users, Queries and Documents: A Unified Representation for Web Mining, wi-iat, vol. 1, 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, 2009, pp. 238-244. |
| Hofmann, Thomas, Latent Semantic Models for Collaborative Filtering, ACM Transactions on Information Systems, vol. 22, No. 1, Jan. 2004, pp. 89-115. |
| Google News archive, Jul. 8, 2003, Webmasterworld.com, [online] Retrieved from the Internet http://www.webmasterwolrd.com/forum3/15085.htm [retrieved on Nov. 20, 2009] 3 pages. |
| Gr{hacek over (c)}ar, Miha, User Profiling: Collaborative Filtering, SIKDD 2004, Oct. 12-15, 2004, Ljubljana, Slovenia, 4 pages. |
| Joachims, T., Evaluating retrieval performance using clickthrough data. Proceedings of the SIGIR Workshop on Mathematical/Formal Methods in Information Retrieval; Aug. 12-15, 2002; Tampere, Finland, 18 pages. |
| Joachims et al., “Search Engines that Learn from Implicit Feedback”; Aug. 2007, IEEE Computer Society. |
| Linden, Greg et al., Amazon.com Recommendations: Item-to-Item Collaborative Filtering, [online ], http://computer.org/internet/, IEEE Internet Computing, Jan.-Feb. 2003, IEEE Computer Society, pp. 76-80. |
| U.S. Appl. No. 11/556,143, filed Nov. 2, 2006, in Office Action mailed Jan. 25, 2010, 14 pages. |
| U.S. Appl. No. 11/556,143, filed Nov. 2, 2006, in Office Action mailed Jul. 6, 2010, 20 pages. |
| U.S. Appl. No. 11/556,143, filed Nov. 2, 2006, in Office Action mailed Apr. 20, 2011, 18 pages. |
| McDonnell, Philip, A., “Time Based Ranking,” U.S. Appl. No. 11/870,893, filed Oct. 11, 2007, to be published by the USPTO in a file wrapper, 42 pages. |
| Nicole, Kristen, Heeii is StumbleUpon Plus Google Suggestions, [online], Retrieved from the Internet http://mashable.com/2007/05/15/heeii/, 11 pages. |
| Lemire, Daniel, Scale and Translation Invariant Collaborative Filtering Systems, Published in Information Retrieval, 8(1), pp. 129-150, 2005. |
| U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Feb. 8, 2010, 31 pages. |
| U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Feb. 25, 2009, 21 pages. |
| U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Sep. 10, 2009, 23 pages. |
| U.S. Appl. No. 11/685,095, filed Mar. 12, 2007, in Office Action mailed Apr. 13, 2011, 31 pages. |
| Radlinski, et al., Query Chains: Learning to Rank from Implicit Feedback, KDD '05, Aug. 21-24, 2005, Chicago, Illinois, USA, 10 pages. |
| U.S. Appl. No. 11/556,086, filed Nov. 2, 2006, in Office Action mailed Jun. 23, 2010, 21 pages. |
| Schwab, et al., Adaptivity through Unobstrusive Learning, 2002, 16(3), pp. 5-9. |
| Stoilova, Lubomira et al., GiveALink: Mining a Semantic Network of Bookmarks for Web Search and Recommendation, LinkKDD '05, Aug. 21, 2005, Chicago, IL, USA, 8 pages. |
| W3C, URls, URLs and URNs: Classification and Recommendations 1.0, Report from the joint W3C/IETF UR1 Planning Interest Group, Sep. 21, 2001, 8 pages. |
| Xiao, et al., Measuring Similarity of Interests for Clustering Web-Users, ADC, 2001, pp. 107-114. |
| Xie et al., Web User Clustering from Access Log Using Belief Function, K-CAP '01, Oct. 22-23, 2001, Victoria, British Columbia, Canada, pp. 202-208. |
| Yu et al., Selecting Relevant Instances for Efficient and Accurate Collaborative Filtering, CIKM '01, Nov. 5-10, 2001, Atlanta, Georgia, pp. 239-246. |
| Zeng et al., Similarity Measure and Instance Selection for Collaborative Filtering, WWW '03, May 20-24, 2003, Budapest, Hungary, pp. 652-658. |
| Zeng, et al., “Learning to Cluster Web Search Results”, SIGIR '04, Proceedings of the 27th Annual International ACM SIGIR conference on research and development in information retrieval, 2004. |
| Joachims, “Evaluating Search Engines Using Clickthrough Data”, Cornell University, Department of Computer Science, Draft, Feb. 19, 2002, 13 pages. |
| Jansen et al., “An Analysis of Web Documents Retrieved and Viewed”, School of Information Sciences and Technology, The Pennsylvania State University, the 4th International Conference on Internet Computing, Las Vegas, Nevada, pp. 65-69, Jun. 23-26, 2003, 5 pages. |
| U.S. Appl. No. 13/310,901, filed Dec. 5, 2011, Kim et al., Refining Search Results. |
| U.S. Appl. No. 13/608,278, filed Sep. 10, 2012, Kuramochi et al., Collecting Image Search Event Information. |
| U.S. Appl. No. 13/620,528, filed Sep. 14, 2012, Kim et al., Refining Search Results. |
| U.S. Appl. No. 13/953,162, filed Jul. 29, 2013, Kim et al. |
| U.S. Appl. No. 14/143,622, filed Dec. 30, 2013, Kim et al. |
| U.S. Appl. No. 12/825,461, filed Jun. 29, 2010, Kim et al. |
| U.S. Appl. No. 13/617,019, filed Sep. 14, 2012, Tong et al. |
| U.S. Appl. No. 13/898,363, filed May 20, 2013, Tong et al. |
