The invention relates to digital image processing that automatically classifies images and more particularly relates to additive clustering of images lacking individualized capture date-time information.
With the widespread use of digital consumer electronic capturing devices such as digital cameras and camera phones, the size of consumers' image collections continue to increase very rapidly. Automated image management and organization is critical for easy access, search, retrieval, and browsing of these large collections.
A method for automatically grouping images into events and sub-events based on date-time information and color similarity between images is described in U.S. Pat. No. 6,606,411 B1, to Loui and Pavie (which is hereby incorporated herein by reference). An event-clustering algorithm uses capture date-time information for determining events. Block-level color histogram similarity is used to determine sub-events. This method has the shortcoming that clustering very large image sets can take a substantial amount of time. It is especially problematic if events and sub-events need to be recomputed each time new images are added to a consumer's image collection, since additions occur a few at a time, but relatively often. Another problem is that consumers need to be able to merge collections of images distributed across multiple personal computers, mobile devices, image appliances, network servers, and online repositories to allow seamless access. Recomputing events and subevents after each merger is inefficient.
The event-clustering algorithm described in U.S. Pat. No. 6,606,411 B1 has the limitation that it uses date-time information from digital camera capture metadata. This is problematic if images to be added to a database lack correct date-time information. Examples of such images include scanned images, digital image CDs from film capture, stills from video camcorders, or images from digital cameras with incorrect date-time settings. In many cases, the images have an associated date-time that relates to origination of a digital file after scanning or other processing, rather than date-time of image capture.
Many methods based on content-based image classification have been proposed for images where no metadata is available. In PCT Patent Application WO 01/37131 A2, published on May 25, 2001, visual properties of salient image regions are used to classify images. In addition to numerical measurements of visual properties, neural networks are used to classify some of the regions using semantic terms such as “sky” and “skin”. The region-based characteristics of the images in the collection are indexed to make it easy to find other images matching the characteristics of a given query image. U.S. Pat. No. 6,240,424 B1, issued May 29, 2001, discloses a method for classifying and querying images using primary objects in the image as a clustering center. Images matching a given unclassified image are found by formulating an appropriate query based on the primary objects in the given image. U.S. Pat. No. 6,477,269 B1, issued Nov. 5, 2002, discloses a method that allows users to find similar images based on color or shape by using an example query. It is known to provide image retrieval from image databases using a variety of techniques. U.S. Pat. No. 6,480,840, to Zhu and Mehrotra, issued on Nov. 12, 2002, discloses content-based image retrieval using low-level features such as color, texture and color composition.
These content-based methods have the shortcoming of not considering another type of information commonly available with images, chronological order. Images are commonly stored on media in chronological order. For example, images on a Kodak PictureCD™ derived from film capture are in order of capture. Filenames are often created for images using a numerical sequence or other sequence that results in a chronology. For example, some captured digital images have numerical suffixes in the filename that indicate order of generation.
It would thus be desirable to provide methods and systems, in which new images are additively clustered in a database, without reclustering the entire database.
It is further desirable to provide methods and systems, in which chronological order can be considered in additive clustering.
The invention is defined by the claims. The invention, in broader aspects, provides a method and system for combining new images into a database having chronologically ordered images classified into event groups based upon a time difference threshold, and into subgroups based upon a similarity measure. In the method and system, new images are ordered into clusters based upon assessed image features. A representative image is selected in each cluster. A database segment chronologically overlapping the new images is designated and a set of database images similar to each representative image are identified in the segment. Different subgroups including one or more retrieved images are associated with each of cluster to provide matched subgroups. The new images are assigned to matched subgroups associated with respective clusters.
It is an advantageous effect of the invention that an improved methods and systems are provided, in which new images are additively clustered in a database, without reclustering the entire database.
It is a further effect of the invention that an improved methods and systems are provided, in which chronological order is maintained in additive clustering, even when capture date/time information is missing.
The above-mentioned and other features and objects of this invention and the manner of attaining them will become more apparent and the invention itself will be better understood by reference to the following description of an embodiment of the invention taken in conjunction with the accompanying figures wherein:
In the method, images are additively clustered into preexisting event groups and subgroups of a database of images. The groups are organized on the basis of date-time information. The subgroups of the database are based upon a similarity measure other than date-time information. The term “date-time” is used herein to refer to time information in the form of actual dates and times or elapsed dates and times relative to a common standard.
In the following description, some embodiments of the present invention will be described in terms that would ordinarily be implemented as software programs. Those skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. Because image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance with the present invention. Other aspects of such algorithms and systems, and hardware and/or software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein may be selected from such systems, algorithms, components, and elements known in the art. Given the system as described according to the invention in the following, software not specifically shown, suggested, or described herein that is useful for implementation of the invention is conventional and within the ordinary skill in such arts.
As used herein, the computer program may be stored in a computer readable storage medium, which may comprise, for example; magnetic storage media such as a magnetic disk (such as a hard drive or a floppy disk) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any other physical device or medium employed to store a computer program.
The present invention may be implemented in computer hardware. Referring to
Referring to
A compact disk-read only memory (CD-ROM) 124, which typically includes software programs, is inserted into the microprocessor based unit for providing a means of inputting the software programs and other information to the microprocessor based unit 112. In addition, a floppy disk 126 may also include a software program, and is inserted into the microprocessor-based unit 112 for inputting the software program. The compact disk-read only memory (CD-ROM) 124 or the floppy disk 126 may alternatively be inserted into externally located disk drive unit 122, which is connected to the microprocessor-based unit 112. Still further, the microprocessor-based unit 112 may be programmed, as is well known in the art, for storing the software program internally. The microprocessor-based unit 112 may also have a network connection 127, such as a telephone line, to an external network, such as a local area network or the Internet. A printer 128 may also be connected to the microprocessor-based unit 112 for printing a hardcopy of the output from the computer system 110.
Images may also be displayed on the display 114 via a personal computer card (PC card) 130, such as, as it was formerly known, a PCMCIA card (based on the specifications of the Personal Computer Memory Card International Association), which contains digitized images electronically embodied in the card 130. The PC card 130 is ultimately inserted into the microprocessor-based unit 112 for permitting visual display of the image on the display 114. Alternatively, the PC card 130 can be inserted into an externally located PC card reader 132 connected to the microprocessor-based unit 112. Images may also be input via the compact disk 124, the floppy disk 126, or the network connection 127. Any images stored in the PC card 130, the floppy disk 126 or the compact disk 124, or input through the network connection 127, may have been obtained from a variety of sources, such as a digital camera (not shown) or a scanner (not shown). Images may also be input directly from a digital camera 134 via a camera docking port 136 connected to the microprocessor-based unit 112 or directly from the digital camera 134 via a cable connection 138 to the microprocessor-based unit 112 or via a wireless connection 140 to the microprocessor-based unit 112.
The output device provides a final image that has been subject to the transformations. The output device can be a printer or other output device that provides a paper or other hard copy final image. The output device can also be an output device that provides the final image as a digital file. The output device can also includes combinations of output, such as a printed image and a digital file on a memory unit, such as a CD or DVD.
The present invention can be used with multiple capture devices that produce digital images. For example,
The microprocessor-based unit 112 provides the means for processing the digital images to produce pleasing looking images on the intended output device or media. The present invention can be used with a variety of output devices that can include, but are not limited to, a digital photographic printer and soft copy display. The microprocessor-based unit 112 can be used to process digital images to make adjustments for overall brightness, tone-scale, image structure, etc. of digital images in a manner such that a useful image is produced by an image output device. Those skilled in the art will recognize that the present invention is not limited to just these mentioned image processing functions.
The general control computer shown in
It should also be noted that the present invention can be implemented in a combination of software and/or hardware and is not limited to devices, which are physically connected and/or located within the same physical location. One or more of the devices illustrated in
The present invention may be employed in a variety of contexts and environments. Exemplary contexts and environments particularly relevant to combining images from different modalities include, without limitation, medical imaging, remote sensing, and security imaging related to transport of persons and goods. Other exemplary contexts and environments particularly relevant to modalities capturing visible light include, without limitation, wholesale digital photofinishing (which involves exemplary process steps or stages such as film or digital images in, digital processing, prints out), retail digital photofinishing (film or digital images in, digital processing, prints out), home printing (home scanned film or digital images in, digital processing, prints out), desktop software (software that applies algorithms to digital images), other digital fulfillment (such as digital images in—from media or over the web, digital processing, with images out—in digital form on media, digital form over the web, or printed on hard-copy prints), kiosks (digital or scanned input, digital processing, digital or scanned output), mobile devices (e.g., PDA or cell phone that can be used as a processing unit, a display unit, or a unit to give processing instructions), and as a service offered via the World Wide Web.
Referring now to
The manner in which the database has been classified into event groups is not critical and can be automatic using an event-clustering algorithm or manual or a combination of both. The same applies to the classification of event groups into subgroups. Manual classification can be used to determine the time difference threshold heuristically followed by automatic classification using that threshold. In a particular embodiment, the database images have been automatically classified into events and sub-events based on date-time information and color similarity between images as described in U.S. Pat. No. 6,606,411 B1, to Loui and Pavie.
The database supports content-based image retrieval using the same feature or features on which the similarity measure is based. In a particular embodiment, color matching based on histograms computed in each block of images divided into small blocks, as described in U.S. Pat. No. 6,351,556 issued on Feb. 16, 2002 to Loui and Pavie (which is hereby incorporated herein by reference), is used to compute similarity between images. This similarity measure has also been used to determine sub-event boundaries in the automatic event clustering method described in U.S. Pat. No. 6,606,411 B 1, to Loui and Pavie. Alternatively, low-level features such as color, texture and color composition can be used for computing similarity. Color and texture representations and a procedure for similarity-based retrieval is disclosed in U.S. Pat. No. 6,480,840, to Zhu and Mehrotra, issued on Nov. 12, 2002 (which is hereby incorporated herein by reference). In this patent, the dominant colors of an image are determined and each dominant color is described by attribute sets that include color range, moments and distribution within a segment. Texture is described in terms of contrast, scale, and angle. Similarity scores between two images are computed as a weighted combination of the similarity of the underlying features.
The new images, when received, lack capture date-time information or have date-time information that is deemed untrustworthy. The new images can come from one source or multiple sources. For example, the new images can be on a PictureCD obtained by scanning film negatives during photofinishing, image files from scanned prints, or edited digital images, in which capture meta-data has been inadvertently erased. The new images can be from a digital camera that has the date and/or time set incorrectly. This can be determined automatically or manually. For example, date-time information, which indicates a date-time in the future, can be automatically ignored.
Referring to
The chronological order of the new images is first ascertained. This can be done by extracting metadata associated with the new images. For example, automatically-generated dates of file origination can be provided with the new images or the new images may have unique filenames inclusive of metadata defining a relative chronological order. A chronological order can also be ascertained based upon a physical order of images in storage media or by accepting user input designating the order. The ordering of the input images is maintained throughout the addition process.
After the new images are received, one or more image features of each of the new images are determined. The determination can be manual, semi-automatic, or automatic. It is highly preferred that the feature or features used are the same as those upon which the similarity measure is based. In a particular embodiment, the features are block-level color histograms described in U.S. Pat. No. 6,351,556 issued on Feb. 16, 2002 to Loui and Pavie.
The new images are ordered/grouped into clusters based upon the image features. This ordering/grouping can be performed in a variety of different ways, but it is highly preferred that the grouping be on the same basis as the classification of event groups into subgroups in the database. This use of the same classification procedure helps to ensure against the possibility of anomalous classifications, which could result from the use of different classification procedures. In a particular embodiment, the clustering is on the basis of color similarity in neighboring images as described in U.S. Pat. No. 6,351,556 to Loui and Pavie, maintaining the ordering of the images.
A representative image in each of the clusters is selected. The representative image has a high similarity to other images in the cluster and can be selected on the basis of the feature or features used for the grouping of new images. The representative image can have a value of a similarity measure that is at or near an average of the same measure for all of the members of the cluster. Alternatively, the representative image can be the image at the mid-point of the chronologically ordered group. Yet another alternative is manual selection of representative images of each cluster.
A segment of the database is designated that chronologically overlaps the new images. The segment can be the entire database, but is preferably limited to a portion of the database. The temporal length of the segment can be limited based upon any date-time information available that relates to the new images. The new images, in most cases, will have a last date-time, which can be used to exclude from the segment a portion of the database between the new image last date-time and a most recent database image date-time. The last date-time can be a date of creation of scanned digital images from prints or film, the date of receipt of the new images by email, or the last file creation date, for example. The last date-time can also be supplied by a user, for example as a response to an automatically generated question. (The user can supply the last date-time implicitly, for example, by choosing to use the method rather than adding images at the chronological end of the database.)
In the method, the last date-time is no later than the most recent database image date-time. If the last date-time is later, then the method is not used or the new images are culled until a last date-time earlier than the most recent database image date-time can be established. Culling can be done on the basis of user input.
A chronologically ordered segment of the database can include all images earlier than the new image last date-time, but to reduce computational requirements, it is preferred that a beginning date-time of the new images is also ascertained. If the beginning date-time is later than the least recent chronological end of the database, then the segment can then be shortened by limiting the segment to database images having date-times on or between the beginning date-time and the last date-time. The beginning date-time can be ascertained in the same manner as the last date-time.
A representative image from each cluster is used to query the database for similar images. The ordering of the query images is maintained with most recent first. A set of one or more of the database images similar to each of the representative images are identified in the segment, resulting in a set of one or more retrieved images for each representative image as shown in
It is possible for representative images to each match a retrieved image of a different subgroup in the same chronological order as the new clusters, such that association of clusters and subgroups is trivial, but this is unlikely to be the case. In general, the problem of assigning the new clusters to the existing subgroups while maintaining the chronological ordering of the clusters needs to be solved algorithmically to resolve conflicts. It is currently preferred that the algorithm either choose ((201) in
The decision ((205) in
Other alternatives are possible, such as balancing these two goals or providing alternative outputs for the user to decide between.
Referring now to
The sub-events corresponding to the top m images within the database segment, from each of the N retrieved sets are sorted by decreasing time-stamp (most recent first). The value of m is an integer that is preferably between 3 and 20, for example, five. N is the number of new images in a cluster.
The greedy algorithm is used to assign the new groups to existing sub-events ensuring the criterion that more recent sub-events are given a higher priority over older sub-events when assigning the new groups. This algorithm ensures that the shortest start to end time (time starting from the first to the last sub-event where a new group is added) that maintains the ordering of the new images is selected. The greedy algorithm for assigning the new images to existing sub-events is as follows:
The second alternative has a dynamic programming algorithm that maximizes the match score, based upon an assumption that the match score is significant, in addition to meeting the ordering constraints.
The assignment that maximizes the image similarity score while maintaining the order of the new images is selected using the following algorithm following the dynamic programming paradigm.
The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.
Number | Name | Date | Kind |
---|---|---|---|
5749077 | Campbell | May 1998 | A |
6081251 | Sakai et al. | Jun 2000 | A |
6202073 | Takahashi | Mar 2001 | B1 |
6240424 | Hirata | May 2001 | B1 |
6249316 | Anderson | Jun 2001 | B1 |
6272279 | Yokoyama et al. | Aug 2001 | B1 |
6278446 | Liou et al. | Aug 2001 | B1 |
6285995 | Abdel-Mottaleb et al. | Sep 2001 | B1 |
6324545 | Morag | Nov 2001 | B1 |
6347313 | Ma et al. | Feb 2002 | B1 |
6351556 | Loui et al. | Feb 2002 | B1 |
6477269 | Brechner | Nov 2002 | B1 |
6480840 | Zhu et al. | Nov 2002 | B2 |
6606411 | Loui et al. | Aug 2003 | B1 |
6754675 | Abdel-Mottaleb et al. | Jun 2004 | B2 |
6865297 | Loui et al. | Mar 2005 | B2 |
6950989 | Rosenzweig et al. | Sep 2005 | B2 |
6993180 | Sun et al. | Jan 2006 | B2 |
7039239 | Loui et al. | May 2006 | B2 |
7076503 | Platt et al. | Jul 2006 | B2 |
7130864 | Lin et al. | Oct 2006 | B2 |
7154513 | Nishiyama | Dec 2006 | B2 |
7197127 | Stevens | Mar 2007 | B2 |
7242809 | Hunter et al. | Jul 2007 | B2 |
7243101 | Itou | Jul 2007 | B2 |
7313574 | Paalasmaa et al. | Dec 2007 | B2 |
7643686 | Kraus et al. | Jan 2010 | B2 |
7653249 | Loui et al. | Jan 2010 | B2 |
20020183966 | Mishra et al. | Dec 2002 | A1 |
20020188602 | Stubler et al. | Dec 2002 | A1 |
20030009469 | Platt et al. | Jan 2003 | A1 |
20030059107 | Sun et al. | Mar 2003 | A1 |
20030084065 | Lin et al. | May 2003 | A1 |
20030120642 | Egilsson et al. | Jun 2003 | A1 |
20030147558 | Loui et al. | Aug 2003 | A1 |
20040208377 | Loui et al. | Oct 2004 | A1 |
20050027712 | Gargi et al. | Feb 2005 | A1 |
20060074973 | Platt et al. | Apr 2006 | A1 |
20060104520 | Kraus et al. | May 2006 | A1 |
20060126944 | Loui et al. | Jun 2006 | A1 |
20060153460 | Kim et al. | Jul 2006 | A1 |
20060173746 | Cooper et al. | Aug 2006 | A1 |
Number | Date | Country |
---|---|---|
WO 0137131 | May 2001 | WO |
Number | Date | Country | |
---|---|---|---|
20060200475 A1 | Sep 2006 | US |