Patent Application
20080021928
The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the claims.
The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. This includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing computer readable media now known or later developed.
Most image search and retrieval systems are text-driven wherein a user inputs keywords as queries, and a computer system, or more specifically, a search engine, processes the queries by searching through test associated with images in an image database. The search engine returns “relevant” images which are associated with textual information that matches the keywords. In this type of scheme, it is not possible for an un-annotated image to be retrieved, even if the image content is highly relevant to the keywords. Hence, providing relevant textual information for un-annotated images can facilitate search and retrieval operations for the images.
One embodiment of the present invention provides a technique that automatically annotates an image by generating text relevant to the image by using the vast image base that is available on the web. More specifically, one embodiment of the present invention extracts image features from a given image, and then searches through images on the Internet to identify a set of images which contain similar images features. Next, the text surrounding these images is obtained. The technique subsequently identifies common keywords in the obtained text which are used to annotate the given image. Note that this technique requires no human intervention, and utilizes many existing image-processing techniques to perform steps such as image feature extraction and identifying similar images on the web.
The process starts with receiving an image, and a request to provide relevant textual information to the image (step 100). Note that the image may be formatted and stored in any digital format, including but are not limited to: JPEG, GIF, BMP, TIFF, PDF, PS, EMF, MNG, PNG, PSD, SWF and WMF. In addition, the image can also be obtained from still frames of a digital video. For example, we can extract an image from a single frame in a MPEG video and subsequently annotate this image.
Next, the process partitions the image into tiles of a specific size (step 102), and subsequently extracts image features from the tiles (step 104). Note that typical image features which can be extracted from these tiles can include, but are not limited to: shapes, colors, and textures. For example, several color features for a given tile in the image can be extracted, wherein the color features can include, but are not limited to: color histograms, a color histogram layout, and color moments. Furthermore, a number of image-processing techniques can be used to extract specific image features, wherein these image-processing techniques can include, but are not limited to: a direct cosine transform (DCT) technique, a principal component analysis (PCA) technique, and a Gabor wavelet technique. In one embodiment of the present invention, the image features can also include orientation histograms.
Also note that the present invention is not limited to features which are associated with tiles. The present invention can generally be used with any type of image feature, and is not limited to image features associated with tiles. For example, the present invention can be used with image features which are not associated with tile boundaries, such as attributes of a color histogram for the entire image.
The process then matches the extracted image features against known image features in an image-feature library (step 106). In one embodiment of the present invention, the known image features in the library were previously obtained using a similar image extraction technique. For example, the known image features can be represented by DCT coefficients if a DCT technique is used to extract the image features in step 104. In one embodiment of the present invention, matching the extracted image features against the known image features involves matching each extracted image feature to the nearest known image features in the library.
Next, the process combines the matched known-image-features to form image-feature combinations for the image (step 108). More specifically, the process generates a set of image-feature combinations for the image partitioned at the specific tile sizes.
In one embodiment of the present invention, steps 102 to 108 are repeated iteratively for different tile sizes, wherein the tile sizes can range from only a number of pixels per tile up to a single tile containing the entire image. Specifically, for each tile size, the process partitions the image into tiles of that tile size and then extracts image features from the corresponding tiles. The process then matches extracted image features to the known image features and subsequently uses the known image features to produce a set of image-feature combinations for the image. Hence, the resulting image-feature combinations from step 108 comprise image-feature combinations obtained for different image tile sizes. Note that using different tile sizes facilitates capturing image features on multiple scales. Instead of matching to known image features, the system can alternatively restrict the values of the image features to fall into a limited set of discrete values.
The process next identifies multiple other images which have similar image features and/or image-feature combinations (step 110). Specifically, the process searches through a large number of images from various sources. These image sources can include, but are not limited to, images stored in organized image databases, images embedded in webpages on the Internet, images associated with videos on the Internet, or any image that is linked to a web address on the Internet. Note that the difference between images on the Internet and the images in the organized image databases is that the images on the Internet are typically not classified and are typically uncorrelated.
Determining if a particular image includes similar image features to the original image involves processing the particular image in the same manner as the original image to generate image features for the particular image.
While comparing the image features and/or image-feature combinations in the original image with another image, various probability models can be applied. For example, these probability models can include anything from a simple counting technique, which counts the number of similar or identical features, to a complex “multivariate distribution technique,” which assumes a multivariate normal distribution of features when comparing two images. (For a discussion of multivariate normal distributions, please see “http://en.wikipedia.org/wiki/Multivariate_gaussian_distribution”.)
After the multiple similar images are identified, the process obtains text surrounding these images (step 112). In one embodiment of the present invention, the text is obtained around the image tag in the HTML file that includes the link to the image. In another embodiment, the text directly surrounding the actual image in a page is obtained. It is assumed that the text found in the vicinity of a given image on the web has high probability being relevant to the content of the image. As a result, step 112 uses keywords from the text surrounding each identified image as candidate keywords for annotating the original image. Note that some of these candidate keywords may not be relevant to the content of the original image.
The process then expands the set of keywords associated with each identified image by adding synonyms for the set of keywords (step 114). In one embodiment of the present invention, intelligent thesaurus tools are used to add synonyms for each keyword in the set. For example, keywords “sea” and “ocean” may appear in two sets of keywords for two identified images, respectively. After expanding the keywords this way, both images will be associated with both keywords.
Next, the process performs comparisons between sets of keywords for those identified images, to identify intersecting keywords (step 116). Note that adding synonyms to the keywords increases the probability of identifying such intersections.
Finally, the process takes the intersecting keywords between the multiple identified images, and annotates the original image with the intersecting keywords (step 118). Note that the relevance of the intersecting keywords to the original image can be improved by identifying more images in step 110. Because the Internet is an enormously large image database, the probability of finding better matched images increases as more and more images are searched and examined. Furthermore, the intersecting keywords obtained from more images become more accurate as well.
During operation, computer system 200 receives an image 202 and a request to annotate image 202 with relevant textual information.
Computer system 200 includes a processor 201 and a memory 204. Memory 204 can include any type of memory that can store software modules that implement functions described in
Note that computer system 200 interfaces with Internet 214 to access images on Internet 214.
Note that the present invention can also be used to annotate videos which do not have titles or descriptions, thereby facilitating search and retrieval operation for these videos. Specifically, a video can be partitioned into a set of representative frames, and each frame can be processed as an image. Using the above-described process, each of these representative frames can be annotated with relevant keywords. One can choose to use other existing techniques to analyze the annotations for these frames in order to generate a set of common annotations for the entire video.
The foregoing descriptions of embodiments of the present invention have been presented only for purposes of illustration and description. They are not intended to be exhaustive or to limit the present invention to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.
