Web content is shifting from text to multimedia as the amount of multimedia documents on the Web grows at increasing rates. As a result, images on the internet can be a significant source of multimedia information. To access and utilize this abundant image-based information efficiently and effectively, web images should be properly annotated for indexing.
Systems and methods for bipartite graph reinforcement modeling to annotate web images are described. In one aspect the systems and methods implement bipartite graph reinforcement modeling operations to identify a set of annotations that are relevant to a Web image. The systems and methods annotate the Web image with the identified annotations. The systems and methods then index the annotated Web image. Responsive to receiving an image search query from a user, wherein the image search query comprises information relevant to at least a subset of the identified annotations, the image search engine service presents the annotated Web image to the user.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In the Figures, the left-most digit of a component reference number identifies the particular Figure in which the component first appears.
Conventional image-annotation techniques are typically based on image content analysis, and generally do not utilize any rich textual information associated with web images. In contrast, commercial image search engines index web images generally use text surrounding web images in hosting web page(s) when attempting to locate images relevant to a search query. These implementations, however, are based on underling assumptions that text in web pages is related to semantic content of hosted images. Although such approaches may work well in some cases, these conventional techniques may result in images being indexed with inaccurate, irrelevant, and/or incomplete annotations. Systems and methods for bipartite graph reinforcement modeling (“BGRM”) to annotate web images for image search, which are described below with respect to
More particularly, the systems and methods extract two sets of candidate annotations for a target Web image for annotation, where the target image is hosted by a web page. At first, initial set of candidate annotations is extracted from the related textual information such as texts surrounding or otherwise associated with the target image. Using visual and textual information associated with the initial candidates, the systems and methods rank each of the initial candidates in view of the target image. Because such surrounding or other text may be noisy and incomplete, it may not entirely describe semantic content of target image. To account for any such limitations of the initial candidate annotations, the systems and methods extract a second, extending set of candidate annotations from a large-scale image database. To this end, each initial candidate annotation/word is submitted as a query to an image search engine to mine the large-scale database to obtain corresponding search results. These searches are based on an assumption that if certain images in the database are visually similar to the target image and semantically related to the candidate annotations, the textual descriptions associated with these other web images may correlate to the target image.
The systems and methods mine the search results for each initial candidate annotation by clustering the search results. These clustering operations result in a set of clusters, each with a cluster name. With some exceptions (e.g., duplicate word removal, etc.) the cluster names are the extending candidate annotations. The systems and methods determine ranking values for each of the extending candidates using a calculated average similarity between images in the corresponding cluster and the target image. Treating the initial and extending candidates as disjoint graph vertices, the systems and methods model the candidate annotations as a bipartite graph. Edges between disjoint vertices represent calculated and weighted relationships between respective ones of the initial and candidate annotations. In view of these relationships, the initial ranking values associated with respective ones of the initial and extending annotations are then re-ranked/modeled using a ranking reinforcement algorithm. The systems and methods annotate the target image with a select set of the top re-ranked annotations for indexing and image search.
These and other aspects for the systems and methods for bipartite graph reinforcement modeling (“BGRM”) to annotate web images are now described in greater detail.
Although not required, the systems and methods for bipartite graph reinforcement modeling to annotate web images, according to one embodiment are described in the general context of computer-program instructions executed by a computing device such as a personal computer. Program modules generally include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. While the systems and methods are described in the foregoing context, acts and operations described hereinafter may also be implemented in hardware.
In this implementation, for example, program modules 108 include bipartite graph reinforcement modeling module (“BGRM”) 112 and “other program modules” 114 such as an Operating System (OS) to provide a runtime environment, device drivers, other applications, etc. BGRM 112, given a web image, generates two sets of candidate words/annotations—initial annotation candidates (“initial candidates”) 116 and extending annotation candidates (“extending candidates”) 118. In one implementation, web image(s) are retrieved from respective web page(s) hosted by web server 120 (e.g., web server 120-1). Web server 120 is coupled to computing device 102 across communication network 122 such as the Internet. In this implementation, for example, for the given web image (a “target image”) hosted by a web page, BGRM 112 extracts initial annotation candidates 116 using information provided by conventional annotation extraction techniques and/or from text surrounding the web image in the web page hosting the web image. Such information includes, for example, file name, ATL text, URL, titles, description, surrounding text, etc. After stop word removal and stemming, BGRM 112 analyzes visual and textual information associated with each of the initial candidates 116 to rank the textual information to reflect semantic similarity to the target image. In one implementation, for example, BGRM 112 ranks each initial candidate word using a conventional ranking technique such as tf*idf, etc. Associated/surrounding text, however, may be noisy and incomplete, not entirely describing semantic content of web image(s) proximal to the text.
To address assumed limitations of the initial candidates 116, BGRM 112 extends the initial candidates 116 with extending candidates 118. Extending candidates 118 are mined from a large-scale image database. In this implementation, the large-scale image database is the set of web images associated with web pages hosted by web servers 120. To this end, each initial candidate word is submitted as a query to image search engine 124 to mine the database and obtain search results 126. Such queries are shown as a respective portion of “other program data” 128. These searches are based on an assumption that if certain images in the database are visually similar to the target image (i.e., the web image being annotated) and semantically related to the queried words, the textual descriptions associated with these other web images may also correlate to the target image.
BGRM 112 mines search results 126 for each word in the initial candidates 116 by clustering the search results, and thereby, obtaining a set of cluster names. These cluster names are the extending candidates 118. BGRM 112 ranks the extending candidates based on the average similarity between images in the corresponding cluster and the target image for annotation. BGRM 112 then models the initial and extending candidates as a bipartite graph 130 by treating the initial and extending candidates as respective sets of disjoint graph vertices. BGRM 112 evaluates relationships between respective ones of the vertices (candidate annotations) in the graph 130 to identify annotations that pertain to semantically similar concepts. In view of these relationships, BGRM 112 re-ranks the initial and extending annotations 116 and 118 using a ranking reinforcement algorithm that leverages identified relations between vertices in the bipartite graph 130. BGRM 112 annotates the target image with the top ranked annotations from the initial and the extending annotations to generate annotated web image 132. Conventional image indexing application(s) are then used to index annotated web image(s) 132 to present the user (e.g., via display device 134) with image(s) and/or text relevant to a search query.
The following sections provide further exemplary details of how system 100 annotates web image(s) with top ranked annotations, according to one or more embodiments.
BGRM 112 determines extending annotations 118 using a search-based method. Specifically, each initial annotation 116 and its corresponding image are submitted in a query to an image search engine 124 to find semantically related and visually related images (i.e., search results 126). This is a text-based image search. BGRM 112 extracts and identifies more annotations from the search results. For one target web image I, word qi in initial candidate annotation words Q is used to query the image search engine to find semantically related images. This process is applied for each initial candidate annotation identified for I (i.e., the target image for annotation). Then, from the semantically related images provided by the search results 126, BGRM 112 identifies visually related images by determining content-based image similarity between the target image and the images found on the web. In one implementation, for example, the search result (SR) of word w in image I (the target image for annotation) is represented by SR(I, w)={(im1, sim1, de1), (im2, sim2, de2), . . . , (iml, siml, del)}. Where im is the web image obtained by querying I and w, sim is the visual similarity between im and I, and de is the textual description of im. I gives the total number of images in the search result 126. Extending words are extracted by mining the search result (SR) 126 for each word in initial annotations 116 using a conventional search result clustering (SRC) algorithm (e.g., Hua-Jun Zeng, Qi-Cai He, Wei-Ying Ma, et al., “Learning to Cluster Web Search Results”, SIGIR, 2004). In this implementation, SRC clusters documents by ranking salient phrases. SRC first extracts salient phrases and calculates several properties, such as phrase frequencies, and combines the properties into a salience score based on a pre-learned regression model. SRC is capable of generating highly readable cluster names. In this implementation, the cluster names are used as respective extending candidate annotations 118. For each target image for annotation, SRC is used to cluster the descriptions in the search result. The cluster names are merged and duplicate words are discarded. Remaining words in the cluster names are extending candidate annotations X (118). In general, a precise initial word may propagate more precise extending words. Additionally, because the visual information also essentially extends words, an imprecise initial candidate/word may also result in more precise extending words as compared to the initial word.
After acquiring initial candidate annotations 116 for each target web image, a ranking value is defined for each initial candidate using both visual and textual information to measure how consistent it is with the corresponding target web image. The visual consistence of an initial word may be indicated by the visual similarities between images in its search results and the target image. These scores are utilized as the visual ranking value of an initial word. In this implementation, BGRM 112 first sorts the visual similarity scores in descending order. Then, BGRM 112 calculates the average of top K visual similarity scores as the ranking value. And for each initial word qi associated with the target image I, a visual ranking value rankv(qi|I) is calculated, for example, as follows:
Where sim(.) is the image similarity scores ranked in the descending order. To estimate the textual consistence, the similarity of keywords is computed within a web image first by checking how frequently one keyword appears in the search result of another.
For a target image I, BGRM 112 counts the frequency Feqqk(qi) of the initial word qi appearing in textual descriptions of images in the search result of the initial word qk and the frequency Feqqi(qk) of qk appearing in the search result of qi. Feqqk(qi) and Feqqi(qk) reflect the local relation of qi and qk, so the similarity between them is defined, for example, as follows:
sim(qi,qk|I)=Feqqk(qi)+Feqqi(qk) (2)
Generally speaking, the more common a keyword is the more chance it will associate with other keywords. However this kind of associations has lower reliability. Therefore, counts are weighted according to the uniqueness of each keyword, i.e. setting a lower weight to frequent keywords and a higher weight to unique keywords. Finally, the similarities of initial words in the target image I are calculated by modifying equation 2 as follows:
Where N(qi) is the number of the word qi occurring in the descriptions of training images, and ND is the total number of images in the training dataset. In this implementation, several million training images were collected from some photo sharing web sites. A text-based image search system such as that described herein was built using these training images. Each initial annotation is used to query this search system and get its search result. It can be appreciated that such training images can be acquired in other manners.
This approach measures textual similarity between keywords in a local way. It considers not only similarity between words, but also takes into account their relations to a corresponding image. When two words are closely related to the target image for annotation and always appear together in the web page, textual similarity between the two words is high. Conventional methods typically consider only relation between two words. Compared with the traditional methods, the local textual similarity measure is more suitable for the annotation ranking task of system 100. After calculating the textual similarity, BGRM 112 determines the textual ranking value for the initial word qi (rankt(qi|I)) as a normalized summation of the local textual similarities between qi and the other initial words in image I. For example, in one implementation:
In equation 4, the denominator is the normalization factor.
After obtaining above two types of initial word rankings, BGRM 112 normalizes the initial rankings into [0, 1]. These normalized rankings are then fused using a weighted linear combination scheme to define the ranking value of an initial word qi, as shown for example and equation 5:
F
0(qi|I)=a×rankv(qi|I)+(1−a)×rankt(qi|I) (5)
Referring to equation 5, a is the weight ranging from 0 to 1. In one implementation, and because in web-based approaches, text features are generally more effective than image features, the value of parameter a is less than 0.5. In other implementations, the value is greater than or equal to 0.5.
Ranking values for extending annotations 118 are defined differently than how rankings for initial candidates 116 are defined. In this implementation, an extending candidate is actually the name of a search result cluster. In view of this, the ranking value of an extending candidate is estimated by average similarity between images in a corresponding cluster (for which the extending candidate is a cluster name) and the target image for annotation. If member images of a cluster are relevant to the query, the concepts learned from this cluster are likely to represent the content of the query image (i.e., an image associated with an initial annotation 116). Considering uniqueness of each keyword, such ranking values are also weighted using the textual information to define the ranking score, as shown for example, in equation 6.
C
0(xi|I)=v(xi)log(ND/N(xi)) (6)
Referring to equation 6, xi is an extending word of image I and v(xi) is the average member image similarity, and C0 is the initial ranking value of an extending annotation.
This section describes exemplary BGRM 112 operations to construct bipartite graph 130. Initial and extending candidate annotations 116 and 118 are heterogeneous annotations for web images. As described above, initial words (116) represent direct description of the corresponding target image for annotation, while extending words (118) are mined in a large-scale image database. As such, extending words indirectly describe the target image by propagating descriptions from semantically and visually related images. Additionally, extending words with the same initial word tend to be similar to each other. So similarities between extending words are partly determined by their initial word. Similarities between initial words do not have such characteristics. Therefore, initial and extending words are not re-ranked using a unified measure. However, initial and extending words may have close relations. For example, if an initial word is precise, its corresponding extending words are likely to also be precise, and vice versa. In view of such characteristics, BGRM 112 narrows the candidate annotations 116 and 118 for a web image using a bipartite graph. In the mathematical field of graph theory, a bipartite graph is a graph whose vertices can be divided into two disjoint sets V1 and V2 such that every edge connects a vertex in V1 and one in V2; that is, there is no edge between two vertices in the same set.
To construct bipartite graph G 128, BGRM 112 treats initial and extending candidate annotations 116 and 118 as two disjoint sets of graph vertices. Vertices from different disjoint sets are connected by BGRM 112 using edges with proper weights. The weight of an edge is defined using the relations between initial and extending words 116 and 118. In this implementation, BGRM 112 sets a nonzero weight to an edge only if the relation of two vertices is close enough—as determined by a configurable threshold. For two vertices qi, xj of an edge, they have close relation if: (1) xj is extended by qi; or (2) qi is quite similar to xj. The weight of the edge is calculated, for example, as follows:
Referring to equation 7, ωij is the weight, s(.) is the textual similarity between words, and s(.|th) is textual similarity with a pre-defined threshold th.
Suppose, for example, that the initial weight ωij is equal to 0. Equation 7 shows that if xj is extended by qi, the weight ωij will be added by 1. If similarity s(qi, j) between them is above a pre-defined threshold th, ωij will be added by s(qi, xj). s(.) can have various definitions as long as it can represent the relationship between words. In this implementation, for example, BGRM 112 calculates textual similarity using a conventional technique (e.g., Pucher, M., “Performance Evaluation of WordNet-based Semantic Relatedness Measures for Word Prediction in Conversational Speech”, 6th International Workshop on Computational Semantics, Tilburg, Netherlands, 2005). In this implementation, BGRM 112 groups synonyms with the same meaning together to form a synset called a concept. Various concepts are linked to one another, providing a determination of semantic connections between sets of synonyms. In this implementation, the Jiang and Conrath Measure (JNC) can be used to measure semantic distance between two concepts (Jiang, J. and Conrath, D., “Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy”, International Conference on Research in Computational Linguistics, 1997). Given two words wi and wj, BGRM 112 finds associating concepts ci and cj, and determines maximum similarity among all possible corresponding concept pairs as the semantic similarity (i.e., relationship(s)) for the two words.
Initial and extending candidates 116 and 118 are re-ranked with a reinforcement learning algorithm in view of the identified relationships between the two disjoint sets of vertices (i.e., annotations). In this implementation, for example, an exemplary reinforcement algorithm on G is shown in Equation (8). BGRM 112 iterates this equation until convergence.
Referring to equation 8, C0 and F0 are initial ranking value vectors of the extending and initial candidate annotations, respectively. For purposes of exemplary illustration, such vectors are their respective portion of “other program data” 128. In equation 8, L is an adjacency matrix of G (i.e., bipartite graph 130), and L is both row-normalized and column normalized using Equation (9). LT is the transpose of L. C and F indicate the new C0 and F0 after iterations. Respective subscripts indicate iteration times. Parameters α, β are weights ranging from 0 to 1. These weights determine the degree the model relies on the propagated relations between the vertices. The first row of equation 8 addresses the extending annotation 118 ranking update. The second row in equation 8 is directed to re-ranking initial annotations 116. Because LTF reveals that initial word rankings are propagated to extending word rankings via their link relations, the first row in equation 8 indicates that system 100 utilizes information provided by initial words to reinforce information provided by extending words, while the system maintains a certain confidence on the initial values. Additionally, LC reveals extending word rankings via corresponding link relations to reinforce initial word rankings.
L=Dr−1WDc−1 (9)
Referring to equation 9, W is the original adjacency matrix of G; Dr is the diagonal matrix with its (i;i)-element equal to the sum of the i-th row of W; Dc is the diagonal matrix with its (i;i)-element equal to the sum of the i-th column of W.
Equation 8 shows, in each iteration, C is first reinforced and F is then reinforced by updated C. In this implementation, equation 8 shows a stronger reliance on initial word ranking than extending word 118 ranking. Such reliance impacts selection of α and β. In this implementation, for example, the greater value of β is chosen to show the more confidence on the initial ranking for initial words 116.
We now show that the sequences {Cn} and {Fn} of equation 8 converge to provide a final result. Results can also be obtained without iterations by using the convergence form of equation 8. By the iteration equation 8, we have the following:
Where γ=(1−α)(1−β). Since 0<α, β<1, and the eigenvalues of LLT and LTL are in [−1, 1], for they are normalized symmetric matrix based on row-normalized and column-normalized L:
Where C* and F* is the converged C and F, respectively. Now BGRM 112 computes C and F directly without iterations.
In this implementation, and because the ranking values of two sets of candidates 116 and 118 are defined differently, they are treated differently. Three exemplary strategies to determine the final annotations to annotate a target Web image are now described.
In one implementation, for example, using a “top N” strategy BGRM 112 selects a fixed number of annotations with highest ranking values. Actually, top m1 initial words 116 and top m2 extending words 118 are selected (N=m1+m2). Empirically, one can use cross-validation to set suitable m1, m2 values.
In one implementation, for example, BGRM 112 implements a “threshold” strategy to select final annotations from both initial 116 and extending 118 words. If the ranking value of a candidate word is above a threshold, the candidate is chosen. In one implementation, for example, a dynamic threshold dth is used for this purpose, as shown in equation 11.
dth=1/Num(annotations)×η (11)
Where Num(annotations) is the number of one set of candidate annotations of the target image. Thus, the dynamic threshold is determined individually for initial and extending candidates. Note that for one set of annotations within the target image, their ranking values are normalized to ensure that sum of them is 1. So, 1/Num(annotations) means the average ranking value. η is the weight. Therefore, equation 11 expresses that if a candidate's ranking value is larger than the corresponding weighted average ranking value, BGRM 112 selects the candidate.
In one implementation, for example, BGRM 112 determines the number of final annotations according to the number of initial candidates. This implementation, the “threshold” strategy is first utilized to remove imprecise annotations from initial ones. Then extending words with high ranking values are appended to make sure that the number of final annotations is equal to that of the initial candidates. By using the modified threshold strategy, BGRM 112 achieves selection of dynamic numbers of final annotations (cf. top N strategy) with only one parameter estimation (cf. threshold strategy). After selecting final annotations, BGRM 112 merges their ranking values using a unified measure. In one implementation, for example, higher ranking values are provided to initial words. Denote wi as a selected annotation of an image I. An exemplary final annotation ranking function is shown in equation 12. For initial words Q (116), BGRM 112 linearly shrinks the ranking values F to be within [t1, 1]; for extending words X (118), their ranking values C are shrunk to be with in [0, t2], where 0<t1≦t2<1. In this implementation, for example, t1=t2=0.5, although other values could also be used.
Referring to
Operations of block 314 index the annotated Web image 132 using conventional indexing operations to provide a search engine 124 with information to present a user with relevant search results. In one implementation, such an indexing service is local to computing device 102. In another implementation, such an indexing service is remote to computing device 102. Operations of block 316 evaluate the indexed information associated with the annotated Web image to provide a user with relevant search results to a search query.
Although the above sections describe bipartite graph reinforcement modeling to annotate web images in language specific to structural features and/or methodological operations or actions, the implementations defined in the appended claims are not necessarily limited to the specific features or actions described. Rather, the specific features and operations to preserve validity of upper data layers for software fuzzing using lower data layers are disclosed as exemplary forms of implementing the claimed subject matter.