Patent Application
20160239519
The present invention relates to managing photos and videos in general, and in particular to reviewing photos and videos on a user device and automatically identifying unwanted photos and/or videos that can be removed.
Modern mobile devices such as, for example, smartphones and tablets are equipped with powerful cameras. As a result, it is both easy and common for users to take pictures and videos using such mobile devices, so many users often find themselves with a large quantity of photos and videos on their mobile device, computer, digital camera, or the like. This plethora of images can occupy substantial storage space on these devices. Frequently, some of the photos are not photos a user really wants to keep, such as, for example, duplicate photos, photos with bad quality, or “boring” and uninteresting photos. While it is rather easy to take many photos and videos, it can be a time consuming (and itself rather boring) task to regularly review all of one's existing photos and videos on a given device and compare them to each other in order to decide which to keep and which to delete.
What is thus needed in the art is an intelligent and automated solution that can help users to analyze their collections of images and remove unwanted photos and videos from their various devices.
It is an object of the present invention to provide a system to identify unwanted photos and videos on a device, based on, for example, at least one of: (i) reviewing the image quality of photos and videos, (ii) reviewing the user interaction with photos and videos, (iii) reviewing the properties of photos and videos, (iv) reviewing the user profile/preferences of the owner of the photos or someone depicted in the photos, or (v) reviewing information obtained from additional sources, such as, for example, social networks, user contacts, or a user calendar.
Exemplary embodiments of the present invention provide a fast way for a user to automatically identify unwanted photos that are taking up valuable storage on his or her device. Thus, with one tap, an exemplary application can identify all of a user's unwanted photos, such as, for example: photos with bad lighting, blurry shots, duplicates, similar photos, and even photos that the system predicts a user may find boring or uninteresting. Any photos that an exemplary system is unsure about may be left for the user to review and act upon with minimal effort, in a review functionality. For example, swipe right to keep, left to trash. In exemplary embodiments of the present invention, a system gets more accurate by analyzing the user's decisions; it learns which photos the user does not like and thus improves its predictions over time.
In one embodiment, the present invention includes a system for identifying unwanted photos comprising at least one processor and a memory, the memory containing instructions that cause the processor to:
These functions may be conceived of as modules within a software program or application provided on a device, such as, for example, a smartphone, tablet, computer, or the like.
In some embodiments, the image quality of a photo (a term which includes videos and other images, such as screenshots) may be evaluated by image sharpness, lighting, diversity of colors in the photo, whether the photo contains faces, and if so, how big the faces are, or whether the photo contains landscape, text, specific objects, or any combination of these criteria.
In some embodiments, the system may further (i) determine the extent of interaction of the user with each photo and (ii) assign each photo with an interaction score, and in such embodiments proposing photos for removal may be based, in part, on the interaction score.
In some embodiments, such extent of the user's interaction with each photo may be determined by: the number of times the user has viewed each photo, the extent of time the user has watched each photo, the number of times the user has sent or posted the photo, whether a user has deleted the photo or selected the photo as a favorite, or any combination of these criteria.
In some embodiments, an exemplary system may further (i) determine the properties of each photo, and (ii) assign each photo with an image properties score, and in such embodiments proposing photos for removal, may be based, at least in part, on the image properties score.
In some embodiments, the image properties of a photo may comprise: the time the photo was taken, the location the photo was taken at, the folder in which the photo is stored, identities of persons in the photo, length of a video, time of day the photo was taken at, was the photo taken at a place of interest, was the photo taken using the device's camera, is the photo a screenshot, how many photos were taken during a window of time close to the photo's date, was the photo taken at a special event, was the photo taken at a specific location, or any combination of these criteria.
In some embodiments, an exemplary system may further assign to each photo a user profile score, and in these embodiments proposing photos for removal may also be based, at least in part, on the user profile score.
In some embodiments, the user profile score may include, or take into account, information concerning where the user lives or works, statistics regarding photographic habits of the user, user preferences regarding style of photos, number of photos the user usually takes on the weekend, number of photos usually taken when on a trip or outing, whether the user likes or dislikes screenshots, whether the user prefers landscape photos or portrait photos, was the photo taken at a favorite location of the user, was the photo taken with favorite people of the user, whether the user is attending a meaningful event based on an analysis of his or her photography habits, whether the user prefers photos with large or small faces, whether the photo includes people that the user likes or frequently communicates with, whether the photo contains people who are friends or family of the user, whether the user prefers night shots or not, what the user's favorite places, or places he or she dislikes, are, whether the user prefers photos taken at work or not, or any combination of these criteria.
In some embodiments, an exemplary system may assign each photo with a 3rd party score, and where the recommendation module proposes photos for removal based, at least in part, on the 3rd party score.
In some embodiments, the 3rd party score may comprise information extracted from social media applications or networks (e.g., networks shared, number of likes received, number of comments received, etc.), user calendar, contact lists (e.g., from phone, messaging, chat or other applications), or from any other application, or any combination of these sources.
In some embodiments, user selections of whether to keep or delete a photo may be analyzed and used by the system or application to accurately predict a user's preference whether to keep or delete a photo.
The U.S. application file contains at least one drawing executed in color. Copies of the U.S. patent application publication with color drawings(s) will be provided by the U.S. Patent Office upon request and payment of the necessary fee.
In the following detailed description, reference is made to the accompanying drawings that form a part thereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. It is understood that various other embodiments may be utilized or implemented, and structural changes may be made without departing from the scope of the present invention.
It will be readily apparent that the various methods and algorithms described herein may be implemented by, for example, appropriately programmed general purpose computers and computing devices, including smartphones, tablets, PCs and the like. Typically, a processor (e.g., one or more microprocessors) on such a device will receive instructions from a memory or similar device or component, and execute those instructions, thereby performing one or more processes defined by those instructions. Further, programs that implement such methods and algorithms may be stored and transmitted using a variety of media in a number of manners. In some embodiments, hard-wired circuitry or custom hardware or chipsets may be used in place of, or in combination with, software instructions for implementation of the processes of various embodiments. Thus, embodiments according to the present invention are not limited to any specific combination of hardware and software.
As used herein, a “processor” means any one or more microprocessors, central processing units (CPUs), computing devices, microcontrollers, digital signal processors, or like devices.
Exemplary embodiments of the present invention relate to managing photos, which, as used herein, is understood to include both still photos, video files and images, such as captured screenshots and other image files.
The present invention thus relates to a system comprising a processor and memory for identifying unwanted photos. The system may include various modules, such as (i) a scanning module for identifying a plurality of photos on one or more user devices. The user device can be any computing platform such as a smartphone, a tablet, a desktop PC, a set-top box, a laptop or cloud storage. The invention can also help manage photos taken with a digital camera, after the photos were transferred to a computer.
The system may also include (ii) an image quality module for determining the quality of each photo and assigning each photo with an image quality score. Additionally, an exemplary system may include (iii) a classification or recommendation module for proposing photos to a user for removal based on an assigned image quality score.
As noted, such modules may be implemented in software, and thus a module may be a functional aspect of an overall set of instructions. Thus, as used herein, the term “system” may refer to either physical circuits and chipsets, or a software application, or any combination of the two.
In exemplary embodiments of the present invention, photos may be analyzed based on different parameters. During this process the system may gather and set different features for each photo (examples of photo features include blurriness score, or whether the photo was taken at work).
In some embodiments, based on these features, photos may, for example, be classified into two groups:
In exemplary embodiments of the present invention, such a classification may be performed via two main methods:
In exemplary embodiments of the present invention, data or features for a given photo can be gathered or analyzed in various ways, such as one or more of the following:
1. Computer Vision
A set of algorithms can be run that analyze the quality of the photo and what it contains. The system either gives a score to the photo in different quality-related parameters, or analyzes whether a photo is of a certain type or contains some object. Examples of such quality parameters include:
2. Photo Metadata
An exemplary system can analyze metadata associated with the photo and use that as features of, or for, the photo. For example:
3. User Profiling & User Preferences
In exemplary embodiments of the present invention, algorithms may be used that analyze the user data on hand (for example, a user's photograpic history) to both profile the user and understand his or her preferences. Such data can include, for example:
Based on the above, different photo features can be analyzed, such as, for example:
4. User Interactions
In exemplary embodiments of the present invention, each action a user takes with a photo may be tracked, and can be used as part of the classification process. Examples of possible user actions can include:
5. 3rd Party Data
In exemplary embodiments of the present invention, a system can connect to different social platforms, external tools or applications the user may be using, or may have used, and extract information from them that can be related directly or indirectly to the users' photos. For example, an exemplary system may determine:
As noted above, an important feature of exemplary systems according to the present invention is the ability to learn from a user's past actions and decisions to make better predictions in the future. As users take actions and make decisions, an exemplary system can learn from decisions made by users regarding various bad photos they are shown by identifying whether the user decided to keep the photo or to delete it. This learning applies both in general—i.e., analyzing the preferences of many users and deducing user preferences in general—as well as specifically—i.e., analyzing the preferences of a particular user, or group/demographic of users, and deducing that user's (or group/demographic of users') specific preferences. In other words—users are labeling the data for the system (keep vs. delete) and the system can use those actual user choices to learn whether its predictions were correct or not.
The system may then take all the labeled datasets and use machine learning algorithms, or other heuristics, to improve the outputs of the classification. This can be done in a variety ways, including:
Given the processing and analyses of photos and photo data as described above, in exemplary embodiments of the present invention, photos may be classified as “Bad” or For Review“, as noted above, and also, if applicable, as “Duplicate” or “Similar.” These categories are next described in greater detail.
Bad Photos—“Bad Photo” is the general term used herein to refer to photos that an exemplary system predicts the user would like to remove. Bad photos are photos that the system thinks are too bad to keep, and thus recommends that the user delete them. The user can clean them all with a single tap, as shown in
When users click a “Review and Clean” tab they can view a summary page of the bad photos, such as is shown in
In exemplary embodiments of the present invention, a system may also provide a user with an alternative to deletion: move all the bad photos to a different folder instead of deleting them, or, for example, even create a ZIP, or other archive file, for them.
Optionally, a scale can be shown next to a photo allowing the user to leave feedback, such as, for example, a scale running from “boring” to “very interesting.” Such information helps the system learn the user's preferences, and, as noted, this knowledge helps the system better analyze photos in the future and improve the accuracy of its predictions about user preferences as to keep or remove a photo.
Photos for Review—Photos for review are photos that the system estimates are bad but for which the accuracy of that estimate is below a predetermined value. Thus, the system presents these photos to a user one by one so that the user can decide whether to keep or delete them. An example photo review screen is shown in
Duplicates or Similar—the system may identify duplicate or similar photos based on the following criteria:
For each set of duplicates or similar photos, an exemplary system can, for example, decide which is the best photo out of the set, and can then mark that photo as “best photo” in a display screen to a user. In general, a best photo will be identified as the one with the highest ranking available. More importantly, if a system can recognize smiles it can give those photos a very high score.
In exemplary embodiments of the present invention, the following system practices and user actions can be supported in dealing with duplicate photos:
Long Videos—In exemplary embodiments of the present invention, long videos may also be detected. Because they are often not worth all the memory they occupy, long videos may thus be identified, and tagged for deletion, in a parallel fashion to “Bad Photos.” In exemplary embodiments of the present invention, long videos may be defined as videos that are either larger than a predetermined size value or longer than a predetermined temporal length value. For example, videos that are longer than 4 minutes and/or larger than 30MB. Naturally, as devices have faster processors and larger storage areas, the above values may be increased, and may be user definable in any case, in various embodiments.
Exemplary Screen Shots
In some embodiments, an exemplary system may display to the user a dashboard with pertinent information so that a user immediately appreciates how much of a problem bad photos are on his or her device. For example, an overall gallery health score can be calculated and displayed based on the overall degree to which the user's photo collection or “Gallery” is “bad.” The following categories, for example, may be used (names are exemplary):
In exemplary embodiments of the present invention, an application may be run either specifically at the user's request, or as a background task on a user device, when the user device is idle, as may be set by the user.
Exemplary Process Flow and Exemplary Algorithms and Analyses
Given the discussion of general principles and techniques provided above, next described is an exemplary process flow diagram for exemplary embodiments of the present invention.
With reference to
Scanning Module 620 is tasked with gathering and storing image properties. In this exemplary embodiments of the present invention, image properties of each photo may comprise the following:
Once a photo has been scanned, the following modules (and these divisions are logical, and may in various embodiments, be combined or subdivided further) may process the photo and photo-related information, to generate a classification (and therefore recommended action):
A. Computer Vision Module
Computer Vision Module 621 is tasked with analyzing a photo's quality and understanding the content of the photo using various computer vision algorithms. Exemplary computer vision algorithms that may be used can include:
1. Blurriness
A blurriness score may be generated for each photo using edge detection analysis. An exemplary process to calculate the score is as follows:
2. Darkness
Each photo can receive a darkness score which can, for example, be calculated in the following manner:
3. Colorfulness
A colorfulness score gives an estimate to how interesting a photo is. A photo can be clear and bright but still be “boring”, such as, for example, a photo of a whiteboard, plain blue sky, or an uninteresting object, such as a computer mouse (See
4. Face Detection
In order to detect faces in photos, algorithms such as OpenCV's face detection algorithm, or the like, may be used. Such algorithms detect faces in photos and also capture the size of each face in the photo, so one can know how dominant it is (i.e., background, foreground, primary subject of the photo, etc.).
It is noted that Google Play Services recently introduced a Faces API in the Mobile Vision library which is a significant improvement over Android's older face detection algorithm. It also includes additional data about faces such as, for example, is the person smiling? are his eyes open?, etc. In some embodiments, Google's Faces API may thus be used.
5. Detection of Objects
In a manner similar to detecting faces, in exemplary embodiments of the present invention, other types of objects may be detected in a photo, to understand its context and content. This may be done, for example, with computer vision algorithms that allow the detection of various types of objects, such as, for example:
B. User Profiling & Preferences Module
User Profiling & Preferences Module 625 is tasked with analyzing user related data and using the data, and any analysis results as another source of input when analyzing and classifying user photos. As part of this module the following may be analyzed and calculated:
1. Find the User's Home Location/Area
GPS info in the user's photos may be analyzed and used to estimate the home area of the user. This allows an exemplary system to know if a specific photo was taken within proximity of the user's home, or, on the contrary, was taken during a trip or outing far from home.
Taking a user's photos from his Camera folders only, they can be clustered into different groups based on their GPS information. For each photo, the system can check if there already exists a cluster such that the distance to this photo is below a certain threshold (for example, a few kilometers, but this is configurable). If not—a new cluster may be created.
After this processing has been completed, the cluster with the most photos is chosen as the “home area cluster”. This is based on the idea that people tend to take most of their photos in the area nearby their home.
It is noted that the home location found is not necessarily the exact home—it is a home area, based on the threshold used, which is sufficient for purposes such as deciding whether or not a photo is taken at home” or “away from home” such as on a trip.
It is noted that for frequent travelers, travel bloggers, and the like, the assumption that users take most of their photos at or near home may not apply. To detect such outliers, various tests may be used based on data such as is obtained and processed by the 3rd parties Module 633, as described below, and the clustering adapted or modified as appropriate.
2. Find the User's Work Location/Area
Finding the locations of the user's work area allows a system to know if a specific photo was taken at or near the user's work location. The algorithm to find a user's work area can be similar to the one used to look for the user's home area, with a few exceptions:
3. Categorize the User in Terms of Photo-Taking Frequency into One of Predefined Buckets
This can be calculated by dividing the total number of photos the user has by the duration in days from oldest photo to newest photo, thus getting a value for average number of photos taken per day. Based on that figure, the user may be categorized into one of three predefined photo frequency buckets: “takes many photos”, “takes average photos”, and “takes few photos”. This value may be calculated differently for each folder on the user's device, to emphasize the difference between photos stored in the “Camera” folder to photos stored, for example, in a “WhatsApp” folder.
This kind of categorization can also apply to specific sets of photos, so as to estimate the user's photo-taking habits on specific occasions. For example: how many photos does he or she usually take when they are on a trip? How many photos does he or she usually take during the weekend?, At specific life-cycle events?, etc.
4. Analyze User Preferences
Based on the user's photographic history and his or her further actions in the system (i.e., his choices of whether to keep or delete certain photos) certain user preferences can be deduced, and can then be used to adapt and personalize photo classification for that particular user.
Thus, in exemplary embodiments of the present invention, the following user preferences may be analyzed:
C. User Interactions with Photos
(https://play.google.com/store/apps/details?id=com.flayvr.flayvr).
Exemplary interaction types can include:
D. 3rd Parties Module
Returning
E. Scoring Module
Continuing with reference to
Score=(blurrinessScore>0.02?0.1:0)+0.2*blurrinessScore+0.2*darknessScore+(colorfulnessScore>0.2?0.1:0)+0.3*colorfulnessScore+0.1*facesCount
As can be seen, this exemplary score is a weighted combination of four component scores: blurrinessScore, darknessScore, colorfulnessScore and facesCount. The components and weighting can obviously be adjusted in various alternate embodiments, and other components may be added.
As described below, the exemplary photographs presented in
F. Classification Module
Classification Module 635 is in charge of classifying the photos into different buckets based on the calculation and analysis done in the previous modules. Thus, as a result of the processing of the Classification Module, some photos are classified in the “Bad Photos” bucket. As noted, bad photos are photos the system thinks are too bad to keep and thus recommends the user to delete them.
In exemplary embodiments of the present invention, a photo may be classified as a bad photo if any of a set of predefined rules apply to it. These rules may inlcude:
As noted above, other photos may be classified as “Photos for Review.” Photos for review are photos that the system estimates are bad but for which the accuracy of that estimate is below a predetermined value. Thus, the system presents these photos to the user one by one so the user can decide whether to keep or delete them.
A photo may be classified as a Photo For Review if it was not previously classified as bad, and any of a set of predefined rules apply to it. These rules can include, for example:
G. Online Learning Module/Offline Learning Module
Online Learning Module 645 in
In other words, as users label photos as keep vs. delete, the system can compare that actual user decision to the system's predicted user decision, and thereby train its predictive algorithms to be more accurate. This may be done, for example, by taking all of the labeled datasets (i.e., photos with a user “keep” vs. “delete” tag) and use Machine Learning algorithms, or other heuristics, to improve the outputs of the Classification Module 635. This can be done in a variety ways:
In exemplary embodiments of the present invention, such a learning module may run in two modes, both of which are depicted in
Thus,
As can be seen in
H. Similar Photos Module
Again with reference to
Variations in Values and Algorithmic Steps
As noted above, the various modules shown in
Exemplary Photos and Their Scores
To illustrate the above-described functionality, the overall and component scores for each photograph in
This is a very dark photo, as can be seen, the trees are really almost one dark hue, nearly indistinguishable from the building, and the building is itself difficult to see. As a result, the colorfulness is low. Thus, there is a low darknessScore, and an even lower colorfulnessScore. The photograph is blurred in the darker foreground but the edges are more distinct between the foreground and the blue sky. No faces are present. This photo has a low overall score, primarily due to the poor darknessScore and the terrible colorfulnessScore, and would be categorized as a “bad photo.”
This is a close-up of a mouse. It is very bright and mostly clear, but not a lot of color variation. Thus, a high darknessScore (=very bright) is assigned, with a very low colorfulnessScore. The photograph has some muddled edges. No faces are present. This has a higher score than
This photo has a higher score than each of
This photo has a higher score than each of
This photo has a higher score than each of
This photo has the highest score of all. It is a shot of a greenhouse structure in a park, with flowers and greenery in the foreground and trees and sky in the background. The building is white, and there is a lot of light, so a high darkness score (=light) was assigned. It is clear to see, and due to the contrasts between blue sky in the background, the white structure, and flowers in the foreground, it has a high colorfulness score, even a bit higher than that of
Although the invention has been described in detail, nevertheless changes and modifications, which do not depart from the teachings of the present invention, will be evident to those skilled in the art. Such changes and modifications are deemed to come within the purview of the present invention and the appended claims.
The present invention claims the benefit of U.S. Provisional Patent Application No. 62/114,647, entitled “SYSTEMS AND METHODS FOR IDENTIFYING UNWANTED PHOTOS STORED ON DEVICE”, and filed on Feb. 11, 2015, the entire disclosure of which is incorporated herein by this reference.
| Number | Date | Country | |
|---|---|---|---|
| 62114647 | Feb 2015 | US |
