Patent Application
20220004573
This invention pertains to multimedia authoring methods, systems, software, and product distribution media. A particular embodiment of the present invention automatically generates one or more view-based representations of a user's stored media files.
In recent years, there has been a proliferation of low cost and high quality digital capture devices such as digital cameras, smart phones, and camera phones. These devices are capable of recording both still and motion (or video) digital images. Many of these devices incorporate wireless network access; touch screen interfaces; low-cost internal, removable, and network-accessible image storage; and various sensors to monitor the motion, orientation, and location of the device. These features have enhanced the opportunities for picture taking and have reduced the effort and expense of photography to the point where users are now amassing vast collections of digital images. Viewing the media assets in these ever-growing collections has become increasingly difficult due to the sheer volume of content. In addition, with collections that include thousands of still image and video files, selecting and presenting desired subsets of the collections becomes a formidable task for the average consumer interested in generating a photo product, such as a collage, photo album, calendar, poster or the like.
A variety of commercial offerings exist in the marketplace for automatically taking a set of multimedia assets and creating a specific photo product, such as an 8×10 photo album. Some of these offerings are even capable of choosing a subset of the assets if the entire set is too large for the desired photo product. However, these solutions typically require the consumer to start the process all over if they wish to go from one output modality, or output format, to another output modality, such as from an 8×10 photobook to an 8×12 photobook. Typically, the consumer must specify at the beginning of the process the desired output product form, or modality. Even if these solutions automate or partially automate the process of identifying a subset of assets and determining how the assets should be broken up into pages—if required by the desired output modality—changing the product form, or modality, will require redoing much of the work in determining what assets go onto what pages. Moreover, current solutions typically do not reflect the creativity that consumers often like to see represented in photobook creations. Commercial offerings typically only order photos sequentially, either based on chronology or upload order. However, consumers often like to create photobooks that may only loosely follow chronological order. For example, a manually created photobook may often have one or more pages that are dedicated to a particular theme, wherein multimedia assets associated with the theme are accumulated at various times.
There remains a need in the art for a system that is capable of generating a multiplicity of representations from a set of multimedia objects, each with a potentially different form. In addition, a system is needed that is capable of generating representations that include thematic groupings in addition to the traditional purely sequential groupings.
The present invention is related to a method of creating one or more view-based representations from a set of multimedia assets comprising the steps of: using a processor to analyze the multimedia assets and corresponding metadata to generate derived metadata; applying a selected method to analyze recorded and derived metadata; ordering and grouping the set of multimedia objects according to the analyzed metadata; prioritizing the assets of the set of multimedia assets according to the analyzed metadata; selecting an output modality; and generating a view-based representation of the set of multimedia assets according to the ordering, grouping, prioritization data, and selected output modality. The method may further comprise the step of saving the ordering, grouping, and prioritization data independently of the elected output modality. The grouping consists of one or more of the following techniques: chronology, frequent itemset mining, face detection, face recognition, location clustering, object detection, object recognition, event detection, and event recognition. The view-based representation may comprise computing an emphasis score for members of the set of multimedia objects.
The selected modality includes a multimedia presentation, an interactive multimedia presentation, a photobook, a printed collage, a virtual collage, or a digital slide show. The ordering, grouping, and prioritization may be persisted independently of the view modality. The views in the view based representation of the set of multimedia objects for the selected modality are determined by the ordering and grouping. An aspect of the present invention also provides a graphic user interface that incorporates a visual representation of the ordering and grouping of the set of multimedia objects according to said analysis. The graphic user interface provides a description or indication of size for the groupings.
The present invention also provides a method of generating one or more view-based representations of an output modality for multimedia assets comprising: a processor analyzing recorded metadata and user-provided metadata associated with multimedia assets of a multimedia collection; the processor generating derived metadata based on the recorded metadata and the user-provided metadata; the processor organizing the media assets in one or more groupings; the processor prioritizing the media assets in each grouping; the processor prioritizing the one or more groupings; the processor generating a storyboard; mapping the storyboard as one or more view-based representations according to a selected output modality; and rendering on a display the one or more view-based representations.
It is to be understood that the attached drawings are for purposes of illustrating the concepts of the invention and may not be to scale.
The invention is inclusive of combinations of the aspects of the present invention described herein. References to “a particular aspect” and the like refer to features that are present in at least one embodiment of the invention. Separate references to “an aspect” or “particular aspects” or the like do not necessarily refer to the same aspect or aspects; however, such aspects are not mutually exclusive, unless so indicated or as are readily apparent to one of skill in the art. The use of singular or plural in referring to the “method” or “methods” and the like is not limiting. It should be noted that, unless otherwise explicitly noted or required by context, the word “or” is used in this disclosure in a non-exclusive sense.
According to one aspect of the present invention, all metadata from the assets in multimedia collection 110 (including captured metadata, user-provided metadata, and derived metadata) is stored in a single file according to the Resource Description Framework (RDF) data model defined by the World Wide Web Consortium (W3C). Some or all of the derived metadata may also be recorded in the media assets themselves; for example, the XMP format defined by Adobe® provides a way to store RDF-based metadata within media assets. Some derived metadata may be associated with a set of media assets, as opposed to a specific media asset. A feature of the present invention supports metadata associated both with specific media assets, as well as sets of assets corresponding to higher level abstractions, such as events, places, and people.
Metadata, in general, encompasses data that is stored and associated with a media asset. As mentioned previously, metadata includes recorded or previously recorded metadata, which is recorded by the capture device—e.g., capture time, date, and location provided by a digital camera. Metadata also encompasses user-provided metadata, such as that provided via a user interface located on the image capture device or via an image editing application interface that may be accessed through a remote computer. Lastly, metadata encompasses derived metadata, such as metadata computed by face detection or event classification algorithms that are applied to media assets post-capture.
Derived metadata can be generated at the time of storing media asset files on a computer. It can also be generated or derived automatically without a user's knowledge by programmed operation of appropriate metadata-generating software. Such software may be capable of generating many types of metadata based on extrapolating existing metadata (i.e., recorded metadata and user-provided metadata) or making inferences from the same. For example, a family tree may be inferred given sufficient existing metadata for known family members who are depicted in stored image media collections. Derived metadata includes metadata deduced from existing metadata of any type. All metadata, whether input by a user, provided by a recording apparatus, or derived by a computer system, can be used by the one embodiment of the system to generate additional metadata based on inferences that can be determined from existing metadata.
EXIF data (Exchangeable Image File format for digital still cameras: EXIF Version 2.2, JEITA CP-3451, Japan Electronics and Information Technology Industries Association, April 2002) is an example of metadata generated by a recording device, which is stored with the captured media asset. For example, a digital camera might include various camera settings associated with an image such as f-stop, speed, and flash information. Such camera-generated data, or recorded metadata, may also include GPS data indicating geographic location related to where an image was captured.
Temporal event clustering of still images and videos is generated by automatically sorting, segmenting, and clustering an unorganized set of media assets into separate temporal super events, events, and sub-events, as described in detail in U.S. Pat. No. 6,606,411, entitled “A Method For Automatically Classifying Images Into Events,” issued on Aug. 12, 2003, and U.S. Pat. No. 6,351,556, entitled “A Method For Automatically Comparing Content Of Images For Classification Into Events,” issued on Feb. 26, 2002. In an embodiment of the present invention, temporal event clustering is performed as part of generating metadata. The prior art references listed in this paragraph and the following paragraphs are hereby incorporated by reference in their entirety.
A growing number of digital cameras, and many smartphones, are capable of recording the latitude and longitude of the image capture location when an image is captured. Such location-based metadata can be very powerful especially when used in concert with other metadata sources. For example, the U.S. Geological Survey's Board on Geographical Names maintains the Geographic Names Information System, which provides a means to map latitude and longitude coordinates to commonly recognized feature names and types. Exemplary location types include “church,” “park,” and “school.” Location metadata may also be used to aid in clustering assets. One embodiment of the present invention can use a system like the Geographic Names Information System to generate derived metadata in the form of location types upon analyzing recorded metadata in the form of latitude and longitude values recorded by the image capture device. For example, a media asset captured in New York's Central Park could have recorded metadata in the form of latitude and longitude values (40.7789° N and 73.9675° W). Metadata generator 120 analyzes the recorded metadata (the latitude and longitude values) to generate derived metadata for that asset in the form of the name of the location—i.e., Central Park. Location-based data as described in U.S. Patent Application Publication No. US 2006/0126944, entitled “Variance-Based Event Clustering,” filed on Nov. 17, 2004, can include cell tower locations, GPS coordinates, and network router locations.
Scene classifiers identify or classify a scene into one or more scene types (e.g., beach, indoor, etc.) or one or more activities (e.g., running, swimming, eating, etc.). Scene classifying methods may also be applied to media assets to generate derived metadata. Details of such methods are described in U.S. Pat. No. 6,282,317, entitled “Method for Automatic Determination of Main Subjects in Photographic Images”; U.S. Pat. No. 6,697,502, entitled “Image Processing Method for Detecting Human Figures in a Digital Image Assets”; U.S. Pat. No. 6,504,951, entitled “Method for Detecting Sky in Images”; U.S. Patent Application Publication No. US 2005/0105776, entitled “Method for Semantic Scene Classification Using Camera Metadata and Content-Based Cues”; U.S. Patent Application Publication No. US 2005/0105775, entitled “Method of Using Temporal Context for Image Classification”; and U.S. Patent Application Publication No. US 2004/003746, entitled “Method for Detecting Objects in Digital Image Assets.”
Materials class features provide an indication of various materials (e.g., water, grass, snow etc.) included in the digital image. In one embodiment, the materials class features are determined using a supervised learning approach, such as that described by Carneiro et al. in the article entitled “Supervised Learning of Semantic Classes for Image Annotation and Retrieval” (IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, pp. 394-410, 2007). Determining materials class features is yet another aspect of the present invention that may be implemented to generate further derived metadata. Another method for determining materials class features that can be used in accordance with the present invention is described by Boutell et al. in the article “Learning Multi-label Scene Classification” (Pattern Recognition, Vol. 37, pp. 1757-1771, 2004). This method by Boutell et al. is designed to determine materials class features for digital images that can contain multiple materials classes, such as water, sand, and blue sky which might appear together in a beach scene. The method for detecting sky in an image as described in U.S. Pat. No. 6,504,951, to Luo et al., entitled “Method for Detecting Sky in Images” can also be used to determine materials class features in some embodiments.
Low-level image features provide an indication of the visual content of an image. Low-level image features are those computed directly from the pixel content of the image, such as, but not limited to, the colors, the textures, or the edge information. Low-level image features are useful for determining the similarity of appearance between two images. To ensure that the detected colors match the perceived color regions in the digital image, spatially coherent color regions in the digital image are identified. U.S. Pat. No. 6,480,840, to Zhu et al., entitled “Method and Computer Program Product for Subjective Image Content Similarity-based Retrieval,” describes determining color features from an image based on the assumption that significantly-sized, coherently-colored regions of an image are perceptually significant. Accordingly, a coherent color histogram is first computed for an image that provides an indication of the number of pixels of a particular color that belong to coherently colored regions. Dominant colors are then determined that occupy a significant proportion (according to a predefined threshold) of the overall pixels. Color classifiers characterize the dominant colors present in an image. U.S. Pat. No. 6,480,840 also describes the determination of texture features that can be used for low-level features in accordance with the present invention.
Content-based Image Retrieval (CBIR) techniques retrieve images from a database that are similar to an example (or query) image, as described in detail in U.S. Pat. No. 6,480,840. The algorithms that are used to enable CBIR may also be used to generate metadata indicating image similarity based on low-level image features. This concept can be extended to portions of images or Regions of Interest (ROI). The query for similar images can target either whole prospective similar images or a portion (ROI) of prospective similar images. In other words, images retrieved using CBIR techniques match the query image either as a whole image match or contain an ROI matching a corresponding ROI in the query image. In the context of the current invention, CBIR may be used to identify assets that are similar to some other identified asset.
Facial features in images are detectable and indicate the presence of human faces. Methods for detecting human faces are well known in the art of digital image processing. For example, a face detection method that can be used in accordance with the present invention is described by Jones et al. in the article entitled “Fast Multi-view Face Detection” (Mitsubishi Electric Research Laboratories, TR2003-96, 2003). Facial features that can be computed from various face detection methods include the number of faces in the image and the approximate sizes and locations of faces in the digital image. One feature of the present invention optimizes the methods for detecting faces and analyzing facial features to generate derived metadata.
In some embodiments, the facial images can be analyzed to determine additional information about the people appearing in the images, such as estimated age and gender of the persons in the image. A method for assigning a face to an age category may be implemented in the present invention, where the details of such a method are described in U.S. Pat. No. 5,781,650, entitled “Automatic Feature Detection and Age Classification.” The method uses ratios of facial measurements that are known to change with age due to bone structure development. Also, U.S. Pat. No. 7,362,919, entitled “Method for Generating Customized Photo Album Pages and Prints Based on People and Gender Profiles” describes a method for determining the gender of a detected face using facial measurements and facial textures that are known to be different between the genders. Face detectors are also described in U.S. Pat. No. 7,110,575, entitled “Method for Locating Faces in Digital Color Images,” issued on Sep. 19, 2006; U.S. Pat. No. 6,940,545, entitled “Face Detecting Camera and Method,” issued on Sep. 6, 2005; and U.S. Patent Application Publication No. US 2004/0179719, entitled “Method and System for Face Detection in Digital Image Assets,” filed on Mar. 12, 2003.
There are also commercially available software products (from companies such as Omron, Luxand, ArcSoft etc.) and open-source software products (such as OpenCV, Intel Perceptual Computing and bob) for face detection and characterization. These products can be used for obtaining face-based metadata features, such as a count of the number of faces in the image, approximate sizes of faces, locations of faces, age and gender of faces, and people recognized as the same person across images.
Face recognition is the identification or classification of a face to an example of a person or a label associated with a person based on facial features as described in U.S. Pat. No. 8,315,463, entitled “User Interface for Face Recognition,” issued Nov. 20, 2012; U.S. Pat. No. 7,711,145, entitled “Finding Images With Multiple People or Objects,” issued May 4, 2010; and U.S. Patent Application Publication No. U.S. 2007/0098303, entitled “Determining a Particular Person from a Collection,” published on May 3, 2007.
Face clustering is a form of face recognition wherein faces are grouped by similarity. With face clustering faces that appear to represent the same person are associated together and given a label, but the actual identity of the person is not necessarily known. Face clustering uses data generated from facial detection and feature extraction algorithms to group faces that appear to be similar. This selection may be triggered based on a numeric confidence value. The output of the face clustering algorithm is new metadata: namely, a new object representing the face cluster is created. Each media asset containing a face that is part of the face cluster receives a metadata item indicating that the asset contains a person as represented by the face cluster. In some embodiments, the system may be able to associate the face cluster with an identified person, either by the user providing the facial identification or by consulting some auxiliary data source that provides exemplar faces for a given person.
Detected events may be classified into a semantic category such as birthday, wedding, etc., as described in detail in U.S. Patent Application Publication No. U.S. 2007/0008321, entitled “Identifying Collection Images With Special Events,” filed on Jul. 11, 2005. In one embodiment of the present invention, detecting events serves to expand the library of derived metadata stored in metadata repository 130. Media assets can be associated with one another because they depict the same event. Such media assets may be classified together as depicting the same event because they share the same location, setting, or activity per a unit of time, and are intended to be related, according to the expected intent of the user or group of users. Within each event, media assets can also be clustered into separate groups of relevant content called sub-events. While media assets may be generally classified as depicting the same event because they share the same setting or activity, media assets in a sub-event share more specific similar content within the event. For example, assets may be classified in a sub-event because they were captured at roughly the same time and optionally have some measure of visual similarity.
In U.S. Pat. No. 8,611,677, issued Dec. 17, 2013, Das et al. have described a method for classifying images or videos in a multimedia collection into one of several event categories using a combination of time-based and content-based features. Video concept classification is described in U.S. Pat. No. 8,135,221, which issued on Mar. 13, 2012, to Jiang et al. Multiple events themselves may also be clustered into larger groups called super-events. U.S. Pat. No. 8,611,677 also describes a method for computing super-events from the output of event clustering using a density-based clustering approach. As an example of the different event categories, consider a weekend vacation trip with morning, afternoon, and evening activities each day. Each of the different activities might constitute a separate sub-event, with the sub-events of a day collectively constituting an event and the entire weekend constituting a super-event. In view of this example, it should be understood that the exact grouping will depend on the capture time of the various assets and how they are clustered.
An image value index (IVI) is defined as a measure of the degree of importance (significance, attractiveness, usefulness, or utility) that an individual user might associate with a particular asset. An IVI score can be a stored rating entered by a user as metadata. User-provided metadata in the form of importance rankings, or significance rankings, are described in detail in U.S. Pat. No. 8,135,684, entitled “Value Index from Incomplete Data”; and in U.S. Pat. No. 8,330,830, entitled “Camera User Input Based Image Value Index.” Automatic IVI algorithms can utilize image features, such as sharpness, lighting, and other indications of quality. Camera-related metadata (exposure, time, date etc.), image understanding (skin or face detection and size of skin/face area), or behavioral measures (viewing time, magnification, editing, printing, or sharing) can also be used to calculate an IVI for any particular media asset. Various types of IVI metrics are described in the paper entitled “Multidimensional Image Value Assessment and Rating for Automated Albuming and Retrieval,” 15th IEEE International Conference on Image Processing, 2008, by Loui et al. In particular, this paper describes a technical IVI metric based upon technical characteristics of an image as well as a facial IVI metric based upon characteristics of detected faces in an image.
Video key frame extraction is the process of extracting key-frames and/or a salient shot, scene, or event, and the associated audio to provide a summary or highlight of a video sequence. Video key frame extraction is described in detail in U.S. Pat. No. 7,889,794, entitled “Extracting Key Frame Candidates from Video Clip.”
The growing prominence of social platforms such as Facebook® and Instagram® provide new sources of user-provided metadata, based upon how media objects are used and shared. For example, Facebook® is a widely used social network that provides a means for users to upload and share images and video objects, with optional titles or captions. The fact that a user has even shared a media object indicates that the object has greater value to the user. Facebook® further provides a means for people with whom media objects are shared to comment on the object or to express interest in the object by “liking” it. In addition, Facebook® users can tag people within images. All this information about media assets and their usage is accessible to applications via the Facebook® platform API. Other social platforms such as Flickr® and Instagram® provide comparable APIs.
U.S. Pat. No. 8,625,904 describes the computation of various types of metadata and the discretization of feature values into a finite number of levels. This patent also describes a frequent pattern mining step that is used to identify recurring patterns in a metadata database. In one embodiment of the present invention, similar pattern recognition algorithms, or pattern mining methods, are used to identify recurring patterns in the metadata stored in metadata repository 130. The frequent pattern mining step identifies a set of frequent itemsets, where each of the frequent itemsets is a co-occurring feature descriptor group that occurs in at least a predefined fraction of the digital images. Algorithms for frequent itemset mining in a transactional database are well-known in the art. For example, an implementation of the Eclat algorithm, described by Borgelt in the article entitled “Efficient Implementations of Apriori and Eclat” (Proc. of IEEE ICDM Workshop on Frequent Itemset Mining Implementations, 2003), can be used to mine frequent itemsets. The images corresponding to each frequent itemset generally represent a theme in the image collection. For example, a frequent itemset corresponding to the color features “orange” and “red,” and time-based features “evening” and “summer,” may produce a group of images of sunsets. The frequent itemsets detected depends on the images in the image collection and are not pre-defined.
With reference to
A storyboard representation 270 of the set of multimedia assets is defined as a hierarchical grouped and ordered set of media assets.
The behavior of storyboard metadata analyzer 140 is typically specific to the type of storyboard that is being generated. A storyboard represents a particular way of grouping, ordering, and prioritizing the media assets in multimedia collection 110. One feature of the present invention supports multiple grouping and ordering paradigms, each with its own way for prioritizing objects. For example, another feature supports a chronological paradigm, where the media assets are grouped and organized according to the media capture time. According to a different feature, the storyboard supports a thematic ordering strategy, and yet a further feature provides a combined or hybrid thematic/chronological storyboard. There may even be multiple variations on a single paradigm. For example, the present invention supports a chronological paradigm in which media assets belonging to an event or super-event are summarized, ordered, grouped, and/or prioritized according to a documentary style. An alternative variation is a time-based, or chronological, storyboard that is ordered, grouped, and prioritized so as to select a very small number of representative media assets from distinct time epochs. The latter paradigm could be used to generate storyboards that could in turn be used to generated photo calendars, with a collage for each month showing the best pictures for that month.
In some cases, respecting chronological order at a fine level, such as the level of sub-events, is not important. For example, consider a family vacation to Washington, D.C., with trips to two different Smithsonian museums on a single day, one in the morning and one in the afternoon. Each trip to the two museums constitutes its own independent sub-event. Users would expect a chronological ordering of the captured media assets to order the morning images before the evening images, and may even expect that the different exhibit areas to be ordered sequentially. However, it is often not necessary that every picture be in strict chronological order; if several pictures were taken at a given exhibit, the user may desire to have the pictures ordered in whatever arrangement results in the most aesthetically pleasing result. According to one aspect of the present invention, sub-events—such as the morning and afternoon museum visits—are mapped to parallel segments by default. This gives the system more flexibility in determining the final view-based representation because there is no constraint on the order of presenting the media assets in a parallel segment. Thus, media assets in parallel segments may be arranged in a manner that makes the best use of spatial constraints of the particular output modality. In the case where a storyboard is mapped to a physical, hardcopy output such as a photobook, images grouped in a parallel segment can be arranged in the manner that is visually most appealing because no one image is more important, or more preferred, than another. However, according to an aspect of the system, it may be preferred to specify that images within a sub-event should be ordered chronologically, in which case the corresponding segment type would be sequential. The system is such that it may support other groupings in addition to the sequential and parallel groupings defined here.
A given storyboard may have many view-based representations that respect the grouping and ordering structure. For example, in
In addition to representing, ordering, and grouping information, a storyboard may also include priority information (item 260 computed by metadata analyzer 140 in
Considering people-based metrics separately from other technical metrics, or giving people-based metrics greater weight than other technical metrics, is often appropriate given that people are often the most important part of consumer photography. In the absence of knowing the actual identity of a person in the media asset, the system may use facial similarity clustering to estimate the frequency a given person appears in a set of images. This feature operates under the assumption that a person that frequently appears in media assets of one or more given collections is more likely to be significant than a person that appears infrequently, as might occur if an extraneous stranger is photographed in the background. A feature of the present invention associates a people importance score with each image, which is computed by first computing a people importance score for each face detected within the image, and then computing an overall score for the image. The people importance score is a measure of the expected significance or importance of each individual detected and recognized using facial clustering within the image. The people importance score is used as another factor in computing the final priority score for an image. While one aspect of the invention may favor images with high people importance score, some other aspects may prefer to strive for a more normal distribution. In particular, if a collection of images contains a disproportionate number of pictures of one person, some embodiments may normalize the distribution by effectively penalizing images in which that one person appears. Because the priority may be ultimately used to determine which assets to include in a particular view-based representation, this type of normalization may be appropriate to ensure that, for example, all family members are equally represented in a particular view-based representation.
Consumer photography frequently includes images that are near duplicates of each other. For example, frequently when shooting pictures of groups of people, the photographer will take multiple pictures in rapid succession in an attempt to get a picture where each person in the group looks their best. More experienced photographers may also manually bracket a shot, which means to capture a scene at several different exposure values; more advanced cameras now include the ability to bracket automatically. Such images may all be considered semantically equivalent in that they represent essentially the same scene at the same moment in time. For a set of semantically equivalent images, an feature of the present invention provides for simply selecting a single representative image to represent the set. In the case of the group shot, the picture where everyone looks their best would be chosen; in the bracketed shots, the image having the best exposure would be chosen. Of course, determining the best or representative image may be difficult to fully automate without a deep understanding of the subject matter. A group shot may contain some people of lesser importance to the user than others, and consequently, the appearance of some people may be weighted less than others. Given sufficient information, this type of decision making may be automated, but it requires not only accurate mechanisms for assessing people's facial expression and overall appearance, but knowledge of the significance or importance of the person to the user. Such importance may be explicitly specified by the user as part of a user profile, or it may be estimated by the system by considering the frequency certain people are portrayed in a given media collection, or by considering external factors such as the user's degree of interaction with the person via social media, for example. Any two pictures captured at distinct moments in time will almost never be visually completely identical. However, by considering the subject matter of the picture—both the people and other objects portrayed—as well as the time and place the images were captured, a system can predict the likelihood the images will be viewed as functionally or semantically equivalent by the user, and therefore of a nature that at most one representative image would want to be included in any final representation.
According to one feature, for reasons of computational efficiency, the system determines semantic equivalence by considering the elapsed time between any two consecutive pictures that appeared within a given sub-event cluster, as well as the visual similarity of the images, as determined by using color block histograms. Other embodiments may incorporate the use of CBIR, as described previously, to identify near-duplicate images. In addition, other embodiments may create a synthesized representative image from a set of semantically equivalent images by taking the best elements of each picture. For example, images with high dynamic range are typically created by combining pictures of the same scene captured at different exposure values. The Microsoft® Group Shot tool provides photographers with a mechanism for creating a synthesized group picture by taking the best versions of people from different photos to create a new image. These types of techniques may be used to generate a representative image of a set of semantically equivalent images, with appropriately populated metadata values.
Referring back to
In a pure event-based or chronological storyboard, the priority associated with a given image may be computed by first ranking assets within sub-events, or some other appropriate grouping, using some combination of people-based and non-people based metrics. This allows the system to compute a normalized priority for each asset. Normalization may be based upon the average priority score for the sub-event. This normalization provides a means to compensate for situations where all pictures within a given sub-event (or other grouping) differ in some way from the overall norm. For example, pictures in a particular sub-event may all appear underexposed due to lightning conditions. Omitting this normalization step might result in all the pictures in the sub-event receiving a lower priority, which might in turn result in that sub-event not being represented in the final view-based representation.
According to an aspect of the present invention, the final priority score is computed for an event-based storyboard as a function of the normalized face-based priority for an image, with further adjustments based upon factors such as whether the user has provided a rating for the image or has manually added captions to the image. Other image usages could also be considered, such as whether the user has previously shared the image using social media, has explicitly edited that image or a property of that image, or otherwise demonstrated an interest in the image. In addition, the priority of images coming from very large sub-events may be adjusted so as to avoid images from such sub-events dominating the final view-based representation.
In addition to associating a priority with each image, a feature of the present invention also computes and associates an emphasis score with each image. Whereas the priority score provides a measure of how important an image is, the emphasis score gives a measure of how much space to allocate to an image. The meaning of “space” will depend on the final view representation and output modality. For example, for a hardcopy photobook, if an image has a higher emphasis score, then it should be allocated more space on a page of the photobook. In the instance where the output modality is a digital slideshow or other presentation designed for softcopy output, images having higher emphasis scores are allocated a greater amount of display time relative to images with lower emphasis scores. An example process for computing emphasis scores is shown in
The following XML fragment illustrates the structure of a storyboard. This XML fragment follows the structure illustrated in
A storyboard may also indicate that some images are semantically equivalent, as illustrated by the alternates (alt.) segment 440 in
Rather than use a strict priority score to determine which image from a set of semantically equivalent alternatives to include in a final view-based representation, some embodiments may leverage additional metadata, recorded or derived, as well as optionally user-provided metadata, to determine which image should be selected. For example, consider a view-based representation containing a group shot of six people, where several alternatives exist for that group shot. A viewer-sensitive view-based representation may prioritize a different representative image of the group shot depending on the viewer, so as to pick the version of the group shot that best features the viewer or people important to the viewer. Viewer information may be used at other steps in the process of generating the final view-based representation to prioritize or group assets in a manner most appropriate or tailored to a given viewer. The viewer's preferences may be known as part of a viewer profile, or they may be explicitly specified. For example, the viewer may indicate to the system that a particular person or set of people are of special importance to the viewer. More generally, a viewer's preferences or profile may be used to adjust the images selected for a particular view-based representation. To support this type of behavior, an embodiment of the present invention associates an asset manifest with the storyboard where the manifest includes in RDF format any asset metadata that might be useful in making final selection and rendering decisions.
Often, consumers prefer to arrange media assets in a manually constructed photo album according to groupings that are not strictly chronological. For example, a person may create a photo album with some pages organized in a thematic manner, such as pictures of a given location or person. Manually constructed photo albums may mix various types of groupings. For example, one will frequently find manually constructed photo albums to be predominantly organized in chronological order, but with thematic content inserted sporadically. A feature of the present invention models these types of real-world consumer behavior. The system provides for grouping within a storyboard, where the groups are determined by a variety of techniques beside chronology. For example, frequent itemset mining, face detection, face recognition, location clustering, object detection, event detection, and event recognition all could be the basis for grouping, either individually or in some combination. The system is capable of generating these and other types of groupings. Each grouping type has an appropriate algorithm for determining both the grouping and the associated priority.
A strictly thematic grouping is determined by detecting frequent itemsets in an expanded set of discretized image features. The frequent itemset mining algorithm described in U.S. Pat. No. 8,625,904 treats all features equally. However, features may vary in importance, and the relative importance of feature categories can be determined based on domain knowledge. For example, in consumer images, face-based features are considered to be very important, while color-based features may be of lower importance since there is no correlation between color and depicted object in many cases. According to an aspect of the present invention, the feature categories are assigned weights to indicate their importance in the consumer photo domain. In general, a theme may contain feature descriptors from multiple feature categories. For example, consider a multiyear set of multimedia assets containing pictures from multiple camping trips to the Adirondacks. That set of camping pictures might give rise to the theme {Location=Adirondacks, Season=Summer, Scene=NatureScene}. That same multiyear set may also contain pictures taken of the first day of school, such as a child posing in front of the school bus. A detected theme in that case might be {Color=yellow, Face=OnePerson, TimeOfDay=morning, TimeOfYear=September, Scene=outdoor}.
Table 1 shows example feature descriptors that are stored as metadata, which are divided into categories.
The asset priority scores associated with individual assets in thematic groups are computed based on the features associated with the theme. The final asset priority score reflects how representative the image is of the thematic group in which it is placed, as well as the thematic group's importance. Different asset feature priority scores are computed based on feature category, and then these scores are combined to produce a final asset priority score for the image. According to one aspect of the present invention, feature priority scores are computed for the “materials class features” and “low-level features” categories as the fraction of the image that is covered by the material or low-level color/texture. For the face-based features, an average of the facial IVI indicating the quality of the faces detected is used as the asset feature priority score. For the remaining categories, the technical IVI score that indicates the quality of the image may be used as the asset feature priority score. The feature priority scores based on the features included in the theme are combined to produce the final asset priority score for each asset in the thematic grouping. For example, an asset from the theme of {Blue, GroupOf2} would have a low-level feature priority score component based on the extent of blue areas in the asset and a face-based feature priority score component based on the quality of the two faces detected in the asset. The asset's final asset priority score is computed as the weighted sum of the individual feature priority scores, where the weights are the category-level weights. By assigning an asset priority score to each asset in the group, the system maintains a prioritized ranking of the assets in the group to facilitate future rendering of alternative view-based representations. In other words, the asset priority score enables later methods in the system to choose the top assets from the group.
Similarly, each detected frequent itemset (thematic group) is assigned a thematic group priority score, which is computed as a function of the asset priority scores included in the thematic group. Appropriate functions used to calculate thematic group priority scores include average, median, and an average computed by excluding outlier values. A threshold may be set for a minimum thematic priority score that is required for the frequent itemset to be included in the final list of thematic groups.
The method described here for computing thematic group priority scores will generally result in thematic groups that include face-based features receiving a higher priority score than those thematic groups that include only time-based and content-based features. An itemset that is a combination of multiple features will generally have a higher score than one comprised of a single feature. Given the category weights of Table 1, in the absence of face-based features, multiple other features may be needed to exceed the minimum threshold priority score for inclusion in the final list.
According to another aspect of the present invention, the feature priority score indicates the confidence that the asset is correctly assigned that feature. For example, face detectors generate a confidence score that the detected face is an actual face; a Bayes net-based scene or material detector generates a confidence score indicating the strength of the belief that the output class is true. With respect to this feature, the thematic group priority score quantifies how confident the system is that the assets assigned to a thematic group actually comply with the thematic constraints of the group.
Regarding another aspect, user profile information may be used to determine the thematic group priority score. The user's preference for particular group(s) can be saved as a part of the user's profile. For example, a user may like a theme based on the “GroupOf2Faces” feature because it mostly captures his/her two children. Therefore, according to this aspect, the system notes this user preference and assigns a higher thematic group priority score to a theme based on this feature in the products the user creates in the future. The user preference is used in lieu of or in addition to the category-level weights. Instead of explicit validation by a user of his or her preferences, the system considers a theme to have a higher thematic group priority score if the user has previously included the automatically-generated thematic group in the final version of a product (photobook, calendar, etc.). Collecting statistics on the thematic groups (i.e., groupings of features) that are considered to be important across a large number of users can be used to identify themes that are common to many users. These groups can then be automatically assigned a higher thematic group priority score in the case of unknown users (those without a profile).
Maintaining a profile can facilitate the use of a “recommendation” system. The system maintains a record of the thematic groupings that are preferred (or used) by a first set of users. The system may determine that a particular set of thematic groups is shared by numerous users. For example, the first set of users may all have indicated a preference for the following set of thematic groupings: {“Blue”, “Green”, “Foliage”}, {“Outdoor”, “Summer”, “Park”}, {“NoFaces”, “Outdoor”}, {“Red”, “Orange”, “Sunset”}}, wherein {“Blue”, “Green”, “Foliage”} represents a thematic group. If a second user has these same thematic groupings in his or her collection, along with possibly other thematic groupings, and has thus far only shown a bias towards a subset of this set of thematic groups, then the system may recommend the remaining thematic groupings to the second user. If the second user has not yet shown a preference or bias towards particular thematic groupings, then the system may recommend the entire set of thematic groups to the second user. How the system makes recommendations to the user will depend upon the embodiment. According to one aspect, a graphical user interface is used to suggest to the user thematic groups that may be of interest to the user. According to other aspects of the invention, the system may recommend subsets of thematic groups to increase the scores of those groups when they are determined to appear in the user's collection.
The system generates storyboards wherein the content is predominantly grouped in chronological order, but where thematic groupings are inserted at appropriate locations. Such storyboards are referred to herein as “hybrid storyboards.” The method for generating hybrid storyboards proceeds similarly to the method for generating storyboards from event-based groupings. However, for each event, an additional process is executed to determine if a theme-based summary segment should be generated. This is illustrated in
If the best thematic grouping satisfies some threshold as determined in step 525, then the system proceeds to generate a segment containing the assets in the selected thematic grouping (step 530) as well as a segment corresponding to the event (step 535). The two segments are then combined under a new sequential segment at step 540 and inserted into the storyboard at step 545. In alternatives of this approach, the system may generate different storyboard structures, including omitting the containing segment or making the combining segment be parallel in nature. Note that the present invention is also capable of generating a thematic grouping that corresponds to the entire set of multimedia objects.
Referring back to
At step 275 of
One aspect of the present invention for mapping storyboards to view-based representations supports a multiplicity of algorithms or methods for constructing the view-based representations. One facet of the invention allows the user to select which of the available methods should be employed. The general problem solved by all such methods is that the segment structure of the storyboard needs to be mapped to a series of distinct views. Conceptually, distinct parts of the hierarchical segment tree are mapped to different views.
The present invention may utilize two types of methods for mapping storyboards to view-based representations. One type of method uses the priority information specified in the storyboard to determine a subset of the media assets to include in the view-based representation. This method computes a priority cut-off threshold to determine which assets should be included in the mapping. In this method, only assets whose priority exceeds the threshold are included. The threshold is selected so as to satisfy the specified parameters such as the desired number of pages (views) and the desired number of media assets to include per view. The system may determine a fixed threshold, but it may also provide more flexible thresholds. For example, it may be permissible to include lower priority assets than would otherwise be included in a given view if that would make the view more aesthetically pleasing according to aesthetic measures used by the embodiment. Likewise, it may be permissible or even desirable to omit higher priority assets from a particular view if doing so helps satisfy the specified constraints of the particular output modality. As noted previously, the preferred embodiment associates with the storyboard a manifest containing sufficient additional metadata to enable such determinations.
The second method of mapping storyboards to view-based representations uses all of the media assets within the storyboard; prioritization data is not considered. The number of assets included in a given view of the view-based representation may therefore vary more widely in this approach than it would under a method that applies priority-based asset selection. For example, if the output modality is a hardcopy photo album and the media set is large, then some pages may end up with a very large number of pictures on a page. This method would be appropriate when the user wants to see all of the assets in their media collection in the final view-based representation. In either method, the emphasis score determines how much space should be allocated to a given asset.
A user may either select a specific mapping method or allow the system to automatically choose an appropriate method based upon the characteristics of the multimedia collection, the storyboard, and the desired output modality.
The problem of dividing up a series of media assets into views is not unlike the problem of determining an optimal breaking of a series of words in a paragraph into lines in a column of text. The preferred embodiment employs a dynamic programming algorithm analogous to the algorithm described in Donald E. Knuth and Michael F. Plass, “Breaking Paragraphs into Lines,” Software—Practice and Experience, Vol. 11, pp. 1119-1184. 1981.
Certain hardcopy output modalities such as duplex photobooks provide additional constraints on the nature of views. A duplex photobook consists of spreads of facing pages. Each spread may be considered to be a separate view. However the pages within a spread may also be considered as views. Typically, users expect the content on a spread of a photobook to all be related, although perhaps not as tightly related as the content within a page. Consequently, one feature provides that when generating view-based representations for duplex photobooks, some of the supported methods proceed in two phases. In the first phase, the content is split into views corresponding to each spread. In the second phase, each view corresponding to a spread is further split into two views corresponding to the facing pages. This approach could be readily extended to accommodate spreads containing more than two pages, as would be present in a photobook with centerfolds.
Another feature of the present invention provides a means for the user to select any picture in the final view-based representation and to view alternative images at different levels of the hierarchy. In particular, referring back to
The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.
This application is a continuation of U.S. application Ser. No. 14/736,917, filed Jun. 11, 2015, which is a nonprovisional of and claims priority to U.S. Application No. 62/010,781, filed Jun. 11, 2014. The aforementioned applications are hereby incorporated by reference in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| 62010781 | Jun 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14736917 | Jun 2015 | US |
| Child | 17482050 | US |
