SUMMARIZING A PHOTO ALBUM

Abstract

Embodiments generally relate to summarizing a photo album. In one embodiment, a method includes grouping photos into a plurality of groups of photos, and selecting a plurality of representative photos, where each representative photo represents a respective group from the plurality of groups, where the selecting is based on a quality score of each of the photos, and where each quality score is based on different types of attributes. The method also includes providing the plurality of representative photos to a user.

Description

BACKGROUND

Social network systems often enable users to upload photos and create photo albums that contain the uploaded photos. After a user uploads photos to a social network system, the social network system typically enables the user to create one or more photo albums. The user can then determine which photos to include in each of the photo albums. The social network system typically enables the user to share the photo album with other users of the social network system. For example, a user may allow other users to access and view photos in particular photo albums.

SUMMARY

Implementations generally relate to summarizing a photo album. In some implementations, a method includes grouping photos into a plurality of groups of photos, and selecting a plurality of representative photos, where each representative photo represents a respective group from the plurality of groups, where the selecting is based on a quality score of each of the photos, and where each quality score is based on different types of attributes. The method also includes providing the plurality of representative photos to a user.

With further regard to the method, the photos are grouped based on one or more policies. In some implementations, the photos are grouped such that the photos in a particular group are similar to each other. In some implementations, the method further includes detecting visual elements in each of the photos, and grouping photos that have similar visual elements. In some implementations, the method further includes merging a photo or group of photos with an existing group of photos if the photos in a resulting merged group are similar. In some implementations, the selecting includes determining the quality score of each of the photos, where the quality score of a particular photo is based on multiple attributes, and ranking the quality scores. In some implementations, the selecting includes determining the quality score of each of the photos, where the quality score of a particular photo is based on multiple attributes, and where the attributes include one or more of faces of people and landmarks. In some implementations, the selecting includes determining the quality score of each of the photos, where the quality score of a particular photo is based on multiple attributes, and where the attributes include cropping limitations. In some implementations, the selecting includes determining attribute scores for each of the photos, where the attribute scores may be based on one or more of social affinity, frequency of subject matter, and visual attributes, and determining the quality score of each of the photos, where each quality score is based on the attribute scores. In some implementations, the providing of the plurality of representative photos to the user includes providing a user interface that displays a first view of the photos of the groups of photos and that displays a second view of the plurality of representative photos; and enabling the user to toggle between the first view and the second view.

In another implementation, a method includes grouping photos into a plurality of groups of photos, where the photos are grouped such that the photos in a particular group are similar to each other. The grouping includes detecting visual elements in each of the photos; grouping photos that have similar visual elements; and merging a photo or group of photos with an existing group of photos if the photos in a resulting merged group are similar. The method also includes selecting a plurality of representative photos, where each representative photo represents a respective group from the plurality of groups, where the selecting includes determining a quality score of each of the photos, where each quality score is based on different types of attribute scores, where the attribute scores are based on attributes including one or more of social affinity, frequency of subject matter, visual attributes, faces of people, landmarks, and cropping limitations, and where the selecting of a particular representative photo is based on a ranking of quality scores. The method also includes providing the plurality of representative photos to a user.

In another implementation, a system includes one or more processors, and logic encoded in one or more tangible media for execution by the one or more processors. When executed, the logic is operable to perform operations including: grouping photos into a plurality of groups of photos; selecting a plurality of representative photos, where each representative photo represents a respective group from the plurality of groups, where the selecting is based on a quality score of each of the photos, and where each quality score is based on different types of attributes; and providing the plurality of representative photos to a user.

With further regard to the system, the photos are grouped based on one or more policies. In some implementations, the photos are grouped such that the photos in a particular group are similar to each other. In some implementations, the logic when executed is further operable to perform operations including detecting visual elements in each of the photos, and grouping photos that have similar visual elements. In some implementations, the logic when executed is further operable to perform operations including merging a photo or group of photos with an existing group of photos if the photos in a resulting merged group are similar. In some implementations, the logic when executed is further operable to perform operations including determining the quality score of each of the photos, and where the quality score of a particular photo is based on multiple attributes. In some implementations, the logic when executed is further operable to perform operations including determining the quality score of each of the photos, where the quality score of a particular photo is based on multiple attributes, and where the attributes include one or more of faces of people and landmarks. In some implementations, the logic when executed is further operable to perform operations including determining the quality score of each of the photos, where the quality score of a particular photo is based on multiple attributes, and where the attributes include cropping limitations. In some implementations, the logic when executed is further operable to perform operations including determining attribute scores for each of the photos, where the attribute scores are based on one or more of social affinity, frequency of subject matter, and visual attributes, and determining the quality score of each of the photos, where each quality score is based on the attribute scores.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an example network environment, which may be used to implement the implementations described herein.

FIG. 2 illustrates an example simplified flow diagram for summarizing a photo album, according to some implementations.

FIG. 3 illustrates an example simplified flow diagram for grouping photos of a photo album into multiple groups of photos, according to some implementations.

FIG. 4 illustrates an example simplified flow diagram for selecting representative photos that summarize a photo album, according to some implementations.

FIG. 5 illustrates a block diagram of an example server device, which may be used to implement the implementations described herein.

DETAILED DESCRIPTION

Implementations described herein provide a summary of a photo album. As described in more detail below, a system groups photos into groups of photos, where each group contains photos with content that is similar to each other, and where the photos contained in one group are different from the photos in other groups. For example, one group may include photos of the same person in each photo. Another group may include photos of the same group of people in each photo. Yet, another group may include landmarks. The system then selects one or more representative photos for each group, where each representative photo represents a respective group of photos. For example, the system may determine a particular photo of a person to be the best representative photo for a group of photos based on a quality score. As described in more detail below, the quality score may be based on a variety of attribute scores, which may reflect what other users are more likely to find appealing or interesting. For example, a particular photo may have a high quality score, because the photo shows a baby smiling. Another photo may have a lower quality score, because the photo is blurry and shows person who is not smiling. In various implementations, the system may rank the photos against each other as a part of a process for selecting the best photos to represent various groups of photos. The system then enables the representative photos of each group to be shared. For example, the owner of the photo album may send a summary of the photo album to a friend, where the summary includes representative photos corresponding to a variety of diverse groups of photos. Each photo sent to the friend of the owner represents its respective and distinct group of photos within the owner's photo album.

FIG. 1 illustrates a block diagram of an example network environment 100, which may be used to implement the implementations described herein. In some implementations, network environment 100 includes a system 102, which includes a server device 104 and a social network database 106. In various implementations, the term system 102 and phrase “social network system” may be used interchangeably. Network environment 100 also includes client devices 110, 120, 130, and 140, which may communicate with each other via system 102 and a network 150.

For ease of illustration, FIG. 1 shows one block for each of system 102, server device 104, and social network database 106, and shows four blocks for client devices 110, 120, 130, and 140. Blocks 102, 104, and 106 may represent multiple systems, server devices, and social network databases. Also, there may be any number of client devices. In other implementations, network environment 100 may not have all of the components shown and/or may have other elements including other types of elements instead of, or in addition to, those shown herein.

In various implementations, users U1, U2, U3, and U4 may communicate with each other using respective client devices 110, 120, 130, and 140. For example, users U1, U2, U3, and U4 may share summaries of photo albums with each other via system 102.

FIG. 2 illustrates an example simplified flow diagram for summarizing a photo album, according to some implementations. In various implementations, system 102 may summarize a photo album in a social network system, or anywhere visual media may be used and/or viewed. Referring to both FIGS. 1 and 2, the method is initiated in block 202, where system 102 groups photos into multiple groups of photos. In some implementations, system 102 groups the photos such that the photos in a particular group are similar to each other. In some implementations, system 102 may obtain the photos from an existing photo album stored in social network database 106 or stored in any other suitable database.

A group or cluster of photos may be referred to as a burst. In various implementations, a burst may be defined as a group of photos having very similar images. For example, such photos may be duplicates, near-duplicates, or simply photos of a similar subject (e.g., a person, a landmark, etc.) taken in temporal succession (e.g., within 15 minutes of each other, or any other predetermined time period).

In various implementations, system 102 may group similar photos based on one or more policies, where the policies may be associated with different algorithms that determine the degree of similarity and/or degree of difference between two or more photos. For example, in some implementations, system 102 may group photos of like subject matter together to create a number of groups, referred to as “diversity clusters,” and system 102 may place the photos into the different groups by visual appearance, time, faces, image genre, etc.

System 102 associates photos with groups having similar photos. For example, in some implementations, system 102 may determine two photos to be sufficiently similar to belong to the same group, because the two photos show the same person. In another example, system 102 may determine two photos to be sufficiently similar to belong to the same group, because the two photos show the same landmark. Example implementations directed to various grouping policies and algorithms for grouping photos are described in more detail in connection to FIG. 3.

In block 204, system 102 selects representative photos, where each representative photo represents a respective group from the groups of photos. As such, the representative photos from each group of photos give a comprehensive summary of the photo album. For example, a group of photos of a particular person may include multiple photos (e.g., 5 photos) of that person. System 102 selects one photo to represent that group of photos. Similarly, a second group of photos of a particular landmark may include multiple photos (e.g., 10 photos) of that landmark. System 102 selects one photo to represent that group of photos.

In various implementations, to select a representative photo for a given group, system 102 determines the quality of each of the photos. In various implementations, the quality of a given photo is based on multiple attributes. For example, attributes may be associated with faces of people, landmarks, events, cropping limitations, general photo quality, etc. System 102 may determine an overall quality score for each photo, where the quality score is based on different types of attribute scores. Example implementations directed to computing quality scores and attribute scores are described in more detail in connection to FIG. 4.

In block 206, system 102 provides the representative photos to a user. System 102 also enables the user to share the representative photos. For example, in some implementations, system 102 enables a user to send the representative photos to one or more other users (e.g., friends) of the social network system. In some implementations, system 102 enables a user to post the representative photos in a social network stream. In some implementations, system 102 may provide a link to a page with the photo album. If desired, users who view the representative photos may then access and view the photo album. System 102 enables the owner of the photo album to determine and select which other users may or may not view the photo album via the link. For example, system 102 may provide the owner of the photo album with a drop-down menu or field to enable the owner to select and/or enter names of users or groups of users who may access and view the photo album, or who may not access and view the photo album.

FIG. 3 illustrates an example simplified flow diagram for grouping photos of a photo album into multiple groups of photos, according to some implementations. Referring to both FIGS. 1 and 3, the method is initiated in block 302, where system 102 detects visual elements in each of the photos. Such elements may include faces of people, groups of people, landmarks, objects, etc.

In block 304, system 102 groups photos that have similar visual elements. For example, in some implementations, one group may include photos of the same person. Another group may include photos of the same group of people. Another group may include the same landmark. Another group may include various landmarks. In some implementations, system 102 separates photos that have different visual elements into separate groups. In other words, the photos contained in one group would be different from the photos in other groups. For example, in some implementations, a group would generally not contain both photos of an individual and photos of a landmark. Example implementations for grouping similar photos are described below.

In some implementations, to group photos, system 102 compares a first photo to a second photo based on one or more similarity factors. In some implementations, the similarity factors may include a visual distance between images, a time between the images, the number of faces, whether the same person is identified in photos. To clarify, visual distance refers to the degree of similarity between two images of objects in two different photos. For example, an image of a dog would be more visually distant from an image of a house than an image of a cat.

In some implementations, system 102 may assign a value for each of these similarity factors. In one example implementation, system 102 may assign a value that is proportional to the visual distance (i.e., distinctness) between images in two photos. For example, two photos showing two different types of animals (e.g., a bird and a dog) would have a higher value than two photos showing two different types of birds (e.g., a sparrow and a blue jay).

In another example implementation, system 102 may assign a value that is proportional to the difference in the number of people in the two photos. For example, two photos showing different numbers of people (e.g., a group of two people and a group of three people) would have a higher value than two photos showing the same number of people. In some implementations, the value would go down (e.g., weighted less) if the number of people in both photos goes up. For example, two photos showing different numbers of people, yet with few people (e.g., a group of two people and a group of three people), would have a higher value yet not much higher than two photos showing different numbers of people where the numbers are greater (e.g., a group of 19 people and a group of 20 people).

In another example implementation, system 102 may assign a value that is proportional to the length of the time period between the capture of the two photos. For example, two photos taken during significantly different times (e.g., 6 hours apart, a day apart, etc.) would have a higher value than two photos taken at substantially the same time (e.g., within 1 minute, within the same hour, etc.).

In some implementations, system 102 groups two given photos together in the same group if the total value for the two photos falls below a predetermined threshold. For example, if two photos are similar, their total value will be low such that system 102 will group them together. Conversely, if two photos are dissimilar, their total value will be high such that system 102 will group them separately.

In some implementations, these various similarity factors may have different weights. For example, the value associated with the visual distance between images in two photos may have a higher weight than the value associated with the number of people of the photo. This is one example, and the actual weighting will depend on the specific implementation.

In block 306, system 102 merges a photo or group of photos with an existing group of photos if the photos in a resulting merged group are similar. For example, if system 102 combines to two similar photos into a first group, system 102 would include a third photo in that same group if the third photo is similar to the first two photos (e.g., based on values described above). System 102 would not include a fourth photo in that same first group if the fourth photo is different from the first three photos. Instead, system 102 would create second group for the fourth photo. If system 102 determines that photos in a third group were similar to the photos in the first group, system 102 would merge the first and third groups. In some implementations, system 102 may merge photos or groups of photos with an existing group of photos as system 102 processes new photos or new groups of photos.

In various implementations, system 102 may apply one or more constraint policies to the grouping of photos in order to prevent two groups from being merged if their photos were sufficiently different. In some implementations, system 102 may apply a time constraint. For example, in some implementations, system 102 would not group two groups of photos together if the length of time between when the photos were taken is greater than a predetermined time period (e.g., 30 minutes, 1 hour, 2 hours, 1 day, etc.). As such, if the photos in two different groups have similar photos but the photos were taken a day apart, system 102 would not merge the two groups.

In some implementations, system 102 may apply various “person” constraints. For example, system 102 would not group photos showing people with photos that do not show people (e.g., landmarks or scenery only).

In another example, system 102 may determine if an individual shown in two different photos is the same person. For example, if the individual is the same person in the two photos, system 102 would group the photos in the same group. If the individual is not the same person in the two photos, system 102 would not group the photos in the same group. This could be a case, for example, where two brothers having similar yet different facial features are in two different photos. System 102 may include the two photos in two separate groups based on facial appearance differences.

In some implementations, system 102 may detect photos of particular types of events in a photo album. For example, system 102 may identify particular visual features (e.g., wedding cake, bridal gown, etc.) and determine that the photos are associated with a particular type of event (e.g., wedding). In another example, system 102 may identify graduation caps and determine that the photos are associated with a graduation. In another example, system 102 may identify a beach and a resort hotel and determine that the photos are associated with a vacation. As such, system 102 would group photos that are categorized as particular special events. For example, at a wedding, system 102 may ensure that a group includes photos of the bride and groom. If there are multiple groups containing random individual people, system 102 may merge the groups into one group. This may be appropriate where a series of photos of different people are taken at a wedding party.

In various implementations, system 102 may utilize a variety of recognition algorithms to recognize faces, landmarks, objects, etc. in photos. Such recognition algorithms may be integral to system 102. System 102 may also access recognition algorithms provided by software that is external to system 102 and that system 102 accesses.

In various implementations, system 102 enables users of the social network system to specify and/or consent to the use of personal information, which may include system 102 using their faces in photos or using their identity information in recognizing people identified in photos. For example, system 102 may provide users with multiple selections directed to specifying and/or consenting to the use of personal information. For example, selections with regard to specifying and/or consenting may be associated with individual photos, all photos, individual photo albums, all photo albums, etc. The selections may be implemented in a variety of ways. For example, system 102 may cause buttons or check boxes to be displayed next to various selections. In some implementations, system 102 enables users of the social network to specify and/or consent to the use of using their photos for facial recognition in general.

In situations in which the systems discussed here collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), or to control whether and/or how to receive content from the content server that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and used by a content server.

FIG. 4 illustrates an example simplified flow diagram for selecting representative photos that summarize a photo album, according to some implementations. In various implementations, system 102 may select representative photos that summarize a photo album in a social network system, or anywhere visual media may be used and/or viewed. Referring to both FIGS. 1 and 4, the method is initiated in block 402, where system 102 determines quality scores for each photo, where the quality score for a given photo is based on one or more attribute scores. In various implementations, the attribute scores of a given photo are based on multiple attributes. For example, attributes may include faces of people, landmarks, cropping limitations, etc. As described in more detail below, by basing an overall quality score of each photo on various attribute scores, system 102 is able to select representative photos that have a high probability of being interesting, appealing, and that are best photos to represent various themes, people, location, places, and times the photos were taken.

In some implementations, an attribute score may be based on facial attributes. In some implementations, system 102 may detect one or more faces in a photo, and detect associated facial features. For example, such facial features may be indicative of emotions (e.g., smiling, frowning, etc.), eye and/or head direction, face size, gender, etc. System 102 may then assign a facial attribute score based on one or more policies. For example, in various implementations, system 102 associate a higher facial attribute score with photos showing faces that have a higher probability of being appealing to a viewer. For example, system 102 may associate a higher score with photos showing faces showing more emotion, or happy emotions, etc. System 102 may associate a higher score with photos showing eyes and/or mouth that are open (e.g., where a person is smiling with eyes open). System 102 may associate a higher score with photos showing babies.

In some implementations, an attribute score may be based on landmark attributes. In some implementations, system 102 may detect one or more landmarks in a photo, and detect associated landmark features. For example, such landmark features may be indicative of particular well-known landmarks (e.g., Times Square, Eiffel Tower, etc.). System 102 may then assign a landmark attribute score based on one or more policies. For example, in various implementations, system 102 may associate a higher landmark attribute score with photos showing well-known and/or popular landmarks.

In some implementations, an attribute score may be based on cropping attributes. In some implementations, system 102 may detect one or more objects in a photo, and detect facial features, landmark features, or other features. System 102 may then compute image saliency masks for smart-cropping, where a saliency mask indicates whether or not an image can be cropped to a certain aspect ratio without a significant loss of content. For example, system 102 would not crop a particular to fit a required aspect ratio (e.g., an aspect ratio for a mobile phone screen) if the primary content (e.g., face, landmark, etc.) would be cropped. System 102 may assign a poor cropping attribute score based on one or more policies. For example, in various implementations, system 102 may associate a higher poor cropping attribute score with photos with poor cropping capabilities. In some implementations, the poor cropping attribute score is a negative number (e.g., a penalty score). When added to the over attribute scores, a higher poor cropping attribute score would lowers the overall quality of a given photo. Ultimately, system 102 would not select a particular photo to represent a group if the face or landmark would need to be cropped in order to comply with a particular aspect ratio requirement. In some implementations, system 102 may prune or crops images to ensure that a given photo may be shown well when representing each group of photos.

While some example implementations herein are described in the context of faces and landmarks, system 102 may also determine other attribute scores to be factored into an overall quality score. For example, system 102 may determine attribute scores for logos and other objects. In another example implementation, system 102 may factor in attribute scores associated with non-visual signals such as text, geo-location, and social networking aspects (e.g., friends). In some implementations, system 102 may compute attribute scores based on social affinity. For example, system 102 may give a higher attribute score to photos showing friends or family compared to photos showing people who are not friends or family. In some implementations, system 102 may give a higher attribute score to photos having tags associated with people with higher social affinity to the user (e.g., friends and/or family). In various implementations, system 102 may give a higher attribute score to photos that include known objects or places, including known landmarks.

In some implementations, system 102 may also factor in whether a photo is associated with a particular type of event. For example, system 102 may give a higher attribute score to photos associated with a significant event (e.g., wedding, graduation ceremony, vacation, etc.).

In some implementations, system 102 may also factor in repetition of photos showing a particular subject matter (e.g., person, landmark, logo, etc.). In some implementations, system 102 may compute an attribute score for frequency. For example, system 102 may compute a higher attribute score for photos with subject matter (e.g., a particular person) that appears with high frequency in other photos and may compute a lower attribute score for photos with subject matter that appears with low frequency in other photos. The high number of photos of repeating subject matter indicates high importance. For example, there may be many photos (e.g., 10 photos, 15 photos, etc.) of a newborn baby. The repetitive nature of the photos would contribute positively to a higher quality score. This is one example, and a similar implementation may apply to objects of varying subject matter (e.g., landmarks, logos, themes, etc.).

In some implementations, system 102 may also compute/factor in other attribute scores associated with other visual quality aspects such as poor quality images, motion blur, poor exposure, closed eyes, etc. Some of these attribute scores may have negative values, because they reflect attributes that lower the quality of a given photo.

In some implementations, system 102 may compute an attribute score based on other visual quality aspects such as visual/photographic attributes. The phrases “visual attributes” and “photographic attributes” may be used interchangeably. In some implementations, system 102 may compute an attribute score for various visual/photographic attributes such as sharpness. For example, system 102 may compute a higher attribute score for photos that are sharp and a lower attribute score for photos that are blurry. In some implementations, system 102 may compute an attribute score for exposure. For example, system 102 may compute a higher attribute score for photos that meet a predetermined exposure criteria, and system 102 may compute a lower attribute score for photos that do not meet a predetermined exposure criteria (e.g., photos that are overexposed or underexposed).

In some implementations, system 102 may compute an attribute score for other visual attributes such as noise quality. For example, system 102 may compute a higher attribute score for photos that have a low amount of noise and a lower attribute score for photos that have a high amount noise (e.g., grainy). In various implementations, system 102 may determine the amount of noise from a given photo in a variety of ways. For example, system 102 may determine the amount of noise from metadata associated with the photo (e.g., International Standards Organization (ISO) number provided by the camera that captured the photo). System 102 may also determined the amount of noise from detecting noise characteristic in a photo.

In some implementations, system 102 may give particular weight to photos with features associated with machine learning or other regression techniques including model training based on human-labeled preferences. For example, system 102 may extract general-purpose image content features and compute a quality score based on these content features. In some implementations, system 102 may calibrate various attribute scores based on signal vectors that contain multiple components (e.g., facial attribute score, landmark attribute score, landmark popularity, number of faces, number of duplicate photos, etc.), where the signal vectors are based on machine learning or other regression techniques.

Referring still to FIG. 4, in block 404, system 102 ranks the photos based on their respective quality scores. In some implementations, system 102 may aggregate the photos to compare them against each other. System 102 then generates a global ranking of the photos based on their respective global/overall quality scores. In various implementations, system 102 may select the highest ranking photos and then apply restrictions. For example, in some implementations, system 102 may apply a packing constraint, where system 102 would not select the same subject matter twice (e.g., the same person, the same landmark, etc.).

In block 406, system 102 selects one or more representative photos for each group based on the ranking of the photos. In some implementations, system 102 may select multiple representative photos for a given group. For example, in a given group there may be two people that warrant being in the same group, and a photo of each of the two people may both rank highly. System 102 may select both photos, one for each person, to represent the group. This is one example, and a similar implementation may apply to objects of varying subject matter (e.g., landmarks, logos, themes, etc.).

In some implementations, system 102 may generate precomputed summaries of photos. For example, after selecting a group of representative photos, system 102 may store a list of the representative photos in any suitable storage location. This eliminates or minimizes the need for system 102 to determine the representative photos in response to future requests. In various implementations, system 102 may also store the quality scores and attribute scores for photos, which minimizes computation time when updating a group of representative photos.

In some implementations, system 102 may provide the representative photos to a user via a user interface having multiple views of photos. For example, system 102 may provide a view of all photos of the groups of photos, and may provide a view of the representative photos, or “best photos.” In some implementations, system 102 may enable the user to toggle between an “all photos” view and a “best photos” view. In some implementations, in the “all photos” view, system 102 may display all photos in reverse chronological order (or other predetermined or selectable order). In the “best photos” view, system 102 may display the representative photos, or “best photos” in reverse chronological order (or other predetermined or selectable order).

In various implementations, when applying a facial recognition algorithm, system 102 may obtain reference images of users of the social network system, where each reference image includes an image of a face that is associated with a known user. The user is known, in that system 102 has the user's identity information such as the user's name and other profile information. In some implementations, a reference image may be, for example, a profile image that the user has uploaded. In some implementations, a reference image may be based on a composite of a group of reference images.

In some implementations, to recognize a face in a photo, system 102 may compare the face (i.e., image of the face) and match the face to reference images of users of the social network system. Note that the term “face” and the phrase “image of the face” are used interchangeably. For ease of illustration, the recognition of one face is described in some of the example implementations described herein. These implementations may also apply to each face of multiple faces to be recognized.

In some implementations, system 102 may search reference images in order to identify any one or more reference images that are similar to the face in the photo.

In some implementations, for a given reference image, system 102 may extract features from the image of the face in a photo for analysis, and then compare those features to those of one or more reference images. For example, system 102 may analyze the relative position, size, and/or shape of facial features such as eyes, nose, cheekbones, mouth, jaw, etc. In some implementations, system 102 may use data gathered from the analysis to match the face in the photo to one more reference images with matching or similar features. In some implementations, system 102 may normalize multiple reference images, and compress face data from those images into a composite representation having information (e.g., facial feature data), and then compare the face in the photo to the composite representation for facial recognition.

In some scenarios, the face in the photo may be similar to multiple reference images associated with the same user. As such, there would be a high probability that the person associated with the face in the photo is the same person associated with the reference images.

In some scenarios, the face in the photo may be similar to multiple reference images associated with different users. As such, there would be a moderately high yet decreased probability that the person in the photo matches any given person associated with the reference images. To handle such a situation, system 102 may use various types of facial recognition algorithms to narrow the possibilities, ideally down to one best candidate.

For example, in some implementations, to facilitate in facial recognition, system 102 may use geometric facial recognition algorithms, which are based on feature discrimination. System 102 may also use photometric algorithms, which are based on a statistical approach that distills a facial feature into values for comparison. A combination of the geometric and photometric approaches could also be used when comparing the face in the photo to one or more references.

Other facial recognition algorithms may be used. For example, system 102 may use facial recognition algorithms that use one or more of principal component analysis, linear discriminate analysis, elastic bunch graph matching, hidden Markov models, and dynamic link matching. It will be appreciated that system 102 may use other known or later developed facial recognition algorithms, techniques, and/or systems.

In some implementations, system 102 may generate an output indicating a likelihood (or probability) that the face in the photo matches a given reference image. In some implementations, the output may be represented as a metric (or numerical value) such as a percentage associated with the confidence that the face in the photo matches a given reference image. For example, a value of 1.0 may represent 100% confidence of a match. This could occur, for example when compared images are identical or nearly identical. The value could be lower, for example 0.5 when there is a 50% chance of a match. Other types of outputs are possible. For example, in some implementations, the output may be a confidence score for matching.

For ease of illustration, some example implementations described above have been described in the context of a facial recognition algorithm. Other similar recognition algorithms and/or visual search systems may be used to recognize objects such as landmarks, logos, entities, events, etc. in order to implement implementations described herein.

Implementations described herein provide various benefits. For example, the system provides representative photos for a photo album without requiring any user action. The representative photos include images of high quality and images that represent diverse groups of photos.

FIG. 5 illustrates a block diagram of an example server device 500, which may be used to implement the implementations described herein. For example, server device 500 may be used to implement server device 104 of FIG. 1, as well as to perform the method implementations described herein. In some implementations, server device 500 includes a processor 502, an operating system 504, a memory 506, and input/output (I/O) interface 508. Server device 500 also includes a social network engine 510 and a media application 512, which may be stored in memory 506 or on any other suitable storage location or computer-readable medium. Media application 512 provides instructions that enable processor 502 to perform the functions described herein and other functions.

For ease of illustration, FIG. 5 shows one block for each of processor 502, operating system 504, memory 506, social network engine 510, media application 512, and I/O interface 508. These blocks 502, 504, 506, 508, 510, and 512 may represent multiple processors, operating systems, memories, I/O interfaces, social network engines, and media applications. In other implementations, server device 500 may not have all of the components shown and/or may have other elements including other types of elements instead of, or in addition to, those shown herein.

Although the description has been described with respect to particular embodiments thereof, these particular embodiments are merely illustrative, and not restrictive. Concepts illustrated in the examples may be applied to other examples and embodiments.

Note that the functional blocks, methods, devices, and systems described in the present disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks as would be known to those skilled in the art.

Any suitable programming language and programming techniques may be used to implement the routines of particular embodiments. Different programming techniques may be employed such as procedural or object-oriented. The routines may execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, the order may be changed in different particular embodiments. In some particular embodiments, multiple steps shown as sequential in this specification may be performed at the same time.

A “processor” includes any suitable hardware and/or software system, mechanism or component that processes data, signals or other information. A processor may include a system with a general-purpose central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a geographic location, or have temporal limitations. For example, a processor may perform its functions in “real-time,” “offline,” in a “batch mode,” etc. Portions of processing may be performed at different times and at different locations, by different (or the same) processing systems. A computer may be any processor in communication with a memory. The memory may be any suitable processor-readable storage medium, such as random-access memory (RAM), read-only memory (ROM), magnetic or optical disk, or other tangible media suitable for storing instructions for execution by the processor.

Claims

1. A method comprising: grouping photos into a plurality of groups of photos, wherein the photos are grouped such that the photos in a particular group are similar to each other, wherein the grouping comprises: detecting visual elements in each of the photos;grouping photos that have similar visual elements; andmerging a photo or group of photos with an existing group of photos if the photos in a resulting merged group are similar;selecting a plurality of representative photos, wherein each representative photo represents a respective group from the plurality of groups, wherein the selecting comprises determining a quality score of each of the photos, wherein each quality score is based on different types of attribute scores, wherein the attribute scores are based on attributes including one or more of social affinity, frequency of subject matter, visual attributes, faces of people, landmarks, and cropping limitations, and wherein the selecting of a particular representative photo is based on a ranking of quality scores; andproviding the plurality of representative photos to a user.

2. A method comprising: grouping photos into a plurality of groups of photos;selecting a plurality of representative photos, wherein each representative photo represents a respective group from the plurality of groups, wherein the selecting is based on a quality score of each of the photos, and wherein each quality score is based on different types of attributes; andproviding the plurality of representative photos to a user.

3. The method of claim 2, wherein the photos are grouped based on one or more policies.

4. The method of claim 2, wherein the photos are grouped such that the photos in a particular group are similar to each other.

5. The method of claim 2, further comprising: detecting visual elements in each of the photos; andgrouping photos that have similar visual elements.

6. The method of claim 2, further comprising merging a photo or group of photos with an existing group of photos if the photos in a resulting merged group are similar.

7. The method of claim 2, wherein the selecting comprises: determining the quality score of each of the photos, wherein the quality score of a particular photo is based on multiple attributes; andranking the quality scores.

8. The method of claim 2, wherein the selecting comprises determining the quality score of each of the photos, wherein the quality score of a particular photo is based on multiple attributes, and wherein the attributes include one or more of faces of people and landmarks.

9. The method of claim 2, wherein the selecting comprises determining the quality score of each of the photos, wherein the quality score of a particular photo is based on multiple attributes, and wherein the attributes include cropping limitations.

10. The method of claim 2, wherein the selecting comprises: determining attribute scores for each of the photos, wherein the attribute scores are based on one or more of social affinity, frequency of subject matter, and visual attributes; anddetermining the quality score of each of the photos, wherein each quality score is based on the attribute scores.

11. The method of claim 2, wherein the providing of the plurality of representative photos to the user comprises: providing a user interface that displays a first view of the photos of the groups of photos and that displays a second view of the plurality of representative photos; andenabling the user to toggle between the first view and the second view.

12. A system comprising: one or more processors; andlogic encoded in one or more tangible media for execution by the one or more processors and when executed operable to perform operations comprising:grouping photos into a plurality of groups of photos;selecting a plurality of representative photos, wherein each representative photo represents a respective group from the plurality of groups, wherein the selecting is based on a quality score of each of the photos, and wherein each quality score is based on different types of attributes; andproviding the plurality of representative photos to a user.

13. The system of claim 12, wherein the photos are grouped based on one or more policies.

14. The system of claim 12, wherein the photos are grouped such that the photos in a particular group are similar to each other.

15. The system of claim 12, wherein the logic when executed is further operable to perform operations comprising: detecting visual elements in each of the photos; andgrouping photos that have similar visual elements.

16. The system of claim 12, wherein the logic when executed is further operable to perform operations comprising merging a photo or group of photos with an existing group of photos if the photos in a resulting merged group are similar.

17. The system of claim 12, wherein the logic when executed is further operable to perform operations comprising determining the quality score of each of the photos, and wherein the quality score of a particular photo is based on multiple attributes.

18. The system of claim 12, wherein the logic when executed is further operable to perform operations comprising determining the quality score of each of the photos, wherein the quality score of a particular photo is based on multiple attributes, and wherein the attributes include one or more of faces of people and landmarks.

19. The system of claim 12, wherein the logic when executed is further operable to perform operations comprising determining the quality score of each of the photos, wherein the quality score of a particular photo is based on multiple attributes, and wherein the attributes include cropping limitations.

20. The system of claim 12, wherein the logic when executed is further operable to perform operations comprising: determining attribute scores for each of the photos, wherein the attribute scores are based on one or more of social affinity, frequency of subject matter, and visual attributes; anddetermining the quality score of each of the photos, wherein each quality score is based on the attribute scores.