Patent Application
20140282099
The present invention generally relates to displaying media items and, more particularly, to retrieving media items from different sources and concurrently displaying the media items.
Due to the portability and relative affordability of electronic devices, many people own or operate multiple electronic devices, such as smartphones, laptop computers, tablet computers, and desktop computers. Recent years have seen the popularity explosion of social media services, such as Facebook, Instagram, and Twitter. Each of these phenomena has contributed to the ability for a user to store different media items (such as digital images) in different locations. However, these two phenomena together have compounded the problem of a user attempting to locate certain media items, which may be stored in many different locations. If, for example, a user desires to view all images that were taken by the user within a particular month and the user is not sure if a single storage location contains all such images, then the user must manually check multiple storage locations (e.g., multiple computing devices and third-party services) in order to locate all such images.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.
In the drawings:
In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
In the common vernacular, a digital picture or digital image may either refer to an image file that may be processed to display an image or the actual image itself. As used herein, digital image data refers to digital data that may be processed to display a digital image or photo. For example, non-limiting, illustrative examples of digital image data include a .BMP file, a .JPEG file, a .TIF file, or a .GIF file. Thus, a digital image or photo is the visual display of digital image data. However, in contexts where the distinction between the visual image rendered from digital image data and the digital image data itself is not important to understand how the embodiment of the invention works, a reference to a digital image or photo may follow the common vernacular and implicitly include reference to the digital image data used to display the digital image or digital picture. For example, a description of “storing a digital image” refers to storing digital image data which, when processed, causes the display of the digital image.
Techniques are provided herein for retrieving media items from different sources and presenting the media items concurrently on a display screen. At least one of the different sources may include the same device that causes the media items to be displayed. Other sources may include other devices owned by a user and/or social network accounts registered to the user. Other sources may include accounts of “friends” or contacts of the user.
Embodiments involve not only allowing a user to view media items to which the user has access, but also allowing a user to view media items within other users' accounts in which the other users have given the user access.
Retrieved media items may be displayed in many different formats and may be grouped in many different ways. Examples of formats and groupings are provided in more detail below.
Although many of the figures and examples herein describe or depict digital images, media items include other types of items, such as audio files, video files, text files, etc.
Storage 114 may be volatile storage, non-volatile storage, or a combination of volatile and non-volatile storage. Examples of storage 114 include a hard disk, SD card, DRAM, and SDRAM.
The display of client device 110 may be integrated into client device 110, such as a touchscreen on a smartphone or tablet computer. Alternatively, the display may be separate from, but communicatively coupled to, client device 110 (e.g., a Television connected to a computing device).
Media browser 112 is configured to receive media items from one or more of remote sources 132-136 and, optionally, from (local) storage 114, and to cause the retrieved media items to be displayed. Local storage 114 may be viewed as a single local source or multiple local sources. For example, a computer may host multiple applications (e.g., iPhoto, iMovie, iTunes), each of which accesses different portions of storage 114.
Media browser 112 may be implemented in software, hardware, firmware or any combination thereof. Media browser 112 may consist of multiple modules or components, such as a retrieval component and a display component. The retrieval component is primarily responsible for retrieving media items from multiple sources while the display component is primarily responsible for determining which media items to display and how. In an embodiment, the display component may comprise multiple components, such as a stream media component that is responsible for identifying media items that satisfy one or more criteria and a layout component that is responsible for rendering a display of media items that have been identified.
In a related embodiment, media browser 112 is a web application that executes on one or more web servers (not shown) that are accessible to many client devices. A web server is typically not as resource-constrained as end-user devices, such as a tablet computer. In such a configuration, client device 110 executes a web browser (not shown) that communicates over network 120 with media browser 112 to allow a user of client device 110 to log in to media browser 112 and to retrieve media items from multiple sources and display those media items via the web browser on a display of client device 110.
Examples of media items include digital images, video files, audio files, text files, and executable files. “Displaying” a media item involves displaying all or a portion of the media item (e.g., a portion of a text document, or a frame in a video file), a smaller version of the media item (e.g., a thumbnail version of a digital image), or an icon that represents the media item, such as an icon for an executable file or for a text document.
Network 120 may be implemented by any medium or mechanism that provides for the exchange of data between client device 110 and remote sources 132-136. Examples of network 120 include, without limitation, a network such as a Local Area Network (LAN), Wide Area Network (WAN), Ethernet or the Internet, or one or more terrestrial, satellite or wireless links.
Remote sources 132-136 store or have access to media items that may be sent to (and requested by) media browser 112. Examples of remote sources 132-136 include other computing devices owned or operated by a user of client device 110, social network providers (such as Facebook, Twitter, Instagram, Google+), and third party storage services (such as Picasa and DropBox). Thus, remote sources 132-136 may be provided by different parties.
One or more of remote sources 132-136 may require user authentication before granting access to media items by media browser 112. For example, user A of client device 110 has established user accounts at Facebook and Twitter. The user accounts are said to be “registered” to user A. User authentication of user A may be required for media browser 112 to access media items from those two sources. Additionally or alternatively, one or more of remote sources 132-136 allows unrestricted access to media items associated with (or owned by) user A. In that case, an identifier that uniquely identifies the user may be the only piece of data that is required for media browser 112 to access media items from that remote source.
As noted above, an example of one of remote sources 132-136 is a second computing device (e.g., laptop computer) (not shown) owned by the user of client device 110. In order to communicate with the second computing device, client device 110 and the second computing device may be registered with a network service that allows each device to request or access data from the other device. Thus, in order to access data from the second computing device, client device 110 may send a request that is directed to the network service and that identifies the second computing device. The network service may then forward the request (or modify the request and send the modified request) to the second computing device.
Remote sources 132-136 may send the entirety of a requested media item or a portion of a requested media item to client device 110. For example, remote source 132 may send a smaller resolution version of a digital image to which remote source 132 has access.
Many social network providers allow a user to access media items and other content that are available in accounts other than the user's accounts. Such other accounts may be accounts of “friends” of the user. A “friend” or contact of a user, with respect to a service provider, is an individual that has provided input that allows the user to contact the individual through the service provider. Both the user and the individual have registered with the service provider. Thus, if user A has established a relationship with user B through service provider S, then user A is, thereafter, able to communicate with user B through service provider S. Examples of service provider S include an instant messaging service and a social network service, such as Facebook and LinkedIn.
Thus, for example, user A may view photos uploaded by user B to user B's account. In an embodiment, media items that are retrieved from a social network provider include media items that were uploaded by another user to their own account. Thus, media browser 112 is able to not only retrieve and display media items that a user has created, uploaded, or manipulated in some manner, but also retrieve (potentially-relevant) media items that the user has never seen before, but to which the user has access through social or other network providers. Such media items may include digital images and video uploaded by the user's friends and, optionally, any associated metadata of the media items, such as comments, tags, likes, etc.
In an embodiment, media items that are retrieved from multiple sources are stored persistently on client device 110. In this way, media browser 112 is not required to retrieve the same media items from a particular source multiple times. The media items stored on client device 110 may be stored in association with the sources from which the media items were retrieved. For example, first storage data that identifies remote source 132 is stored in association with a first media item that was retrieved from remote source 132 while second storage data that identifies remote source 134 is stored in association with a second media item that was retrieved from remote source 134.
In a related embodiment, storage available on client device 110 is divided or partitioned based on the sources from which media items are retrieved. For example, storage partition A stores media items that were retrieved from remote source 132 while storage partition B stores media items that were retrieved from remote source 134. The size of a storage partition may be increased or decreased depending on the number of media items in the storage partition and/or other storage partitions.
In an embodiment, media browser 112 stores data that indicates the most recently-retrieved media items from a particular source. Such data is referred to herein as “last retrieval data.” For example, media browser 112 stores a timestamp that is associated with remote source 132. Media browser 112 uses the timestamp to retrieve only media items that were stored at remote source 132 after the date and/or time indicated by the timestamp. As another example, media browser 112 stores data that identifies the most recent media item that was retrieved from remote source 132. Media browser 112 uses that particular media item to retrieve only media items that were uploaded to remote source 132 after that particular media item was uploaded to remote source 132.
In a related embodiment, different sources may be associated with different types of last retrieval data. For example, a timestamp may be stored in association with remote source 132 while an identifier that identifies a media item may be stored in association with local storage 114.
In an embodiment, media items may be deleted or modified based on one or more criteria. The one or more criteria may be different from one computing device to another, depending, for example, on the memory capacity of the computing device upon which media browser 112 executes. For example, if a storage limit for media items is approached, then media items that have been least recently accessed or displayed may be deleted from persistent storage. As another example, high resolution representations of media items may be removed or deleted over time (e.g., based on access/display frequency or based on time) while low resolution representations of the media items are retained. In this way, media browser 112 is able to display the media item to a user when the user desires to view the media item. Additionally, while a low resolution representation of a media item is displayed, media browser 112 may send, to the appropriate source, a request for a high resolution version of the media item and then update the display when the high resolution version is retrieved.
In an embodiment, media browser 112 retrieves media items from a particular source and performs duplicate media item detection on those media items relative to media items that have already been retrieved from the particular source. Such an embodiment is useful if media browser 112 is not configured to request only media items that have not already been retrieved. A duplicate media item may be deleted from persistent storage or, instead, may be “suppressed” at display time.
Duplicate detection may be performed by comparing metadata information of various media items. For example, comparing names, sizes, and dates associated with two media items may indicate that the two media items are the same and, thus, one of the two media items may be deleted. As another example, if the sizes or names of two media items are different, then the two media items may be considered different, despite all other metadata information being the same.
Additionally or alternatively, duplicate detection may involve comparing the content of two media items. For example, a “footprint” of each media item is calculated based on the color schemes of the media items and then compared to each other. As another example, a root mean square (RMS) is calculated for each media item and then compared to each other.
At block 210, a first set of media items is retrieved from a first source. The first source may be local or remote relative to the process or entity that implements process 200. For example, if a client device implements process 200, then the first source may be storage that is included in (or connected directly to) the client device, such as a hard disk or an SD card. As another example, the first source may be a computing device that is different than client device 110, which is to display at least a portion of the first set of media items.
At block 220, a request for media items is sent over a network to a second source that is different than the first source. For example, media browser 112 sends a request, over network 120, to remote source 132. In an embodiment, block 220 involves sending multiple requests simultaneously, each request being sent to a different source.
At block 230, a second set of media items is received from the second source. For example, media browser 112 receives at least one digital image from remote source 132.
At block 240, the first and second sets of media items are concurrently displayed. Block 240 may involve media browser 112 determining, based on one or more criteria, how to arrange the first and second sets of media items on a display of client device 110.
In an embodiment, the content of media items is not requested from one or more sources until media browser 112 receives a request to display the content. For example, for each source, media browser issues, to the source, a batch query to retrieve a list of media identifiers. In response, the source sends the list to media browser 112. For each source, media browser 112 compares the received list of media identifiers with media identifiers in a metadata cache that is associated with the source. In other words, media browser 112 maintains a metadata cache for each source. A metadata cache includes for each media item, a media item identifier and one or more other metadata, such as size of content of the media item, a date associated with the media item, a location associated with the media item, one or more names of one or more people associated with (e.g., depicted in) the media item, aspect ratio (for digital images), likes, and comments. In this way, media browser 112 can determine which media items have been added (if any) to the source and which media items have been removed from the source.
If no new media items are detected based on the identifier comparison (with respect to a particular source), then no further query is issued to the particular source. Otherwise, media browser 112 issues another batch query to the particular source to retrieve metadata (i.e., more than just an identifier, such as date, time, people, comments, likes, etc.) for each newly discovered media item. For example, the batch query may only include identifiers for the newly discovered media item. If a source's metadata cache is empty (i.e., a “cold cache”), then this subsequent batch query is for metadata of all the media items identified in the list. In response to the subsequent batch query, the source sends the requested metadata to media browser 112.
At this point, media browser 112 still does not have any of the media content (including thumbnails) associated with the requested metadata (or associated with existing metadata in the metadata cache). In some embodiments, media browser 112 does not send, to one or more sources, a request (or requests) for such content until media browser 112 receives a request (e.g., initiated by a user) to display a media stream that requires the content.
In an embodiment, media browser 112 is configured to allow a user to register one or more remote sources with media browser 112. “Registering” a remote source may involve the user providing a user name and password (and/or other user credentials) that the user provides the remote source when accessing the user's account maintained by the remote source. Once a remote source is registered with media browser 112, media browser 112 will not have to prompt the user again for user credentials before retrieving media items from the remote source.
In this example, the requested data includes digital images and, optionally, metadata and other data associated with the digital images. Examples of metadata and other data associated with a digital image include a quality metric (e.g., resolution) of the digital image, an author of the digital image, a date and/or time of the digital image, an indication of a geographical location of the digital image, comments from “friends” of the user regarding the digital image (including an identification of those friends), a number of “Likes” of the digital image (including an identification of those who “liked” the digital image), an identification of each (or at least one) person (e.g., “George Smith”) or other object (e.g., “Eiffel Tower”) reflected in the digital image, and a number of people or “friends” reflected in the digital image.
In an embodiment, one or more remote sources have “system support” while one or more other remote sources do not have “system support.” For example, Facebook and Twitter may have system support and Instagram may not have system support. “System support” means that an operating system (OS) of client device 110 has built-in support for a remote source. The OS is aware of the user account at the remote source and can give that account information to an application if the application is given access. In this way, a developer of media browser 112 is not required to write code to support authentication, etc.
Media wall 400 may be a default wall, such as when a user “opens” media browser 112, such as by selecting an icon that represents media browser 112. A default wall may include all media items to which the user has access. Alternatively, a default wall may include all media items that satisfy one or more criteria, such as being associated with a certain date range or being associated with certain people. Any criteria associated with the default wall may be modified by a user.
The media items that qualify for inclusion in a media wall, regardless of whether all the media items may be displayed simultaneously, are referred to herein as a “media stream” that is associated with the media wall. The one or more criteria that are used to determine whether a media item qualifies for inclusion in a media wall are referred to herein as “stream criteria.” Thus, a media stream includes one or more media items that satisfy one or more stream criteria that are associated with the media stream. Media items that satisfy one or more stream criteria are said to “belong” to the media stream and the media wall that is associated with the one or more stream criteria. Thus, different media streams are associated with different stream criteria.
While embodiments are described in the context of displaying and searching media items that are retrieved from different sources, embodiments are not limited to that context. In some embodiments, the context involves media items that are retrieved from or found in a single source.
The media items in a media stream may be ordered based on one or more order criteria, such as time, location, popularity, and/or person identity. Thus, after media browser 112 determines a set of media items that satisfy one or more stream criteria, media browser 112 may determine how to order the set of media items on a media wall based on, for example, a time of day or date associated with each media item in the set. Alternatively, media browser 112 may randomly select, for display, media items from the set of media items that belong to a media stream.
In the depicted example, the digital images in media wall 400 are of different sizes and aspect ratios. In a different embodiment, all digital images depicted in media wall 400 have the same aspect ratio (e.g., 4:3) and/or resolution. Different layout options for media items in a media wall are described in more detail below.
The size of a media item in media wall 400 may be determined by media browser 112 based on one or more size criteria. Examples of the size criteria include popularity and a number of people reflected in the media item. Popularity may be based on one or more factors, such as a number of people who “liked” a media item, a number of people who “commented” on a media item, a number of “friends” of the user who are reflected in a media item, and whether the user and a “close” friend are reflected in a media item.
In an embodiment, a media item that is displayed in a media wall includes an item source indicator. An item source indicator is data that indicates a source of the media item with which the item source indicator is associated. An item source indicator may be text, an icon, or other graphic. In one embodiment, an item source indicator is overlaid on a media item, such that the media item and the item source indicator are visible simultaneously. In another embodiment, an item source indicator is displayed only in response to input that selects the corresponding media item or that selects an option (not depicted) to view sources of multiple media items that are displayed in a media wall.
In the depicted example, each digital image in media wall 400 includes an item source indicator. For example, item source indicator 412 on media item 410 may indicate that media item 410 was retrieved from local storage 114 while item source indicator 422 on media item 420 indicates that media item 420 was retrieved from Facebook.
In an embodiment, if media browser 112 detects that if a media item in a media wall is from multiple sources, then the media wall includes information that indicates that the media item was found from multiple sources. The information may involve the display of two or more item source indicators on the media item. For example, the two or more item source indicators may be stacked on top of each other.
In an embodiment, if a media wall is not large enough to contain all media items that belong to a media stream, then the media wall becomes “scrollable.” In other words, in response to input from a user, the display of the media wall changes to include additional media items that were not previously displayed and exclude media items that were previously displayed.
A user is able to scroll through media wall (such as media wall 400) through one or more input mechanisms, such as a mouse, voice commands, keyboard, or touchscreen. The scrolling may be vertical (such as top to bottom) or horizontal (such as left to right). For example, in order to scroll through media wall 400, a user swipes a finger on a touchscreen from right to left in order to view media items that are to the “right” of the currently-displayed media items.
In some cases, the number of media items that belong to a media stream may greatly exceed the number of media items that may fit in a media wall, thus requiring a user to perform a significant amount of scrolling in order to view all the media items in the media stream. In an embodiment, to address this issue, media browser 112 allows a user to initiate accelerated scrolling of a media wall (such as media wall 400). “Accelerated scrolling” is scrolling that is faster than scrolling that can be initiated by normal scroll initiation input, such as holding down an arrow key, swiping a finger across a touchscreen, or selecting a scroll button on a user interface with a cursor. With touchscreen devices, “normal” scrolling speed may be the scrolling speed that corresponds to the speed of a finger swipe. Thus, a fast swipe of a finger across a touchscreen will cause a media wall to scroll faster than a slow swipe of the finger across the touchscreen.
In an embodiment, accelerated scrolling is initiated by a user dragging or swiping multiple fingers across a media wall. For example, if a user drags two fingers on a touchscreen in a particular direction (e.g., horizontal), then scrolling is accelerated by four times (4×) the normal scrolling speed. As another example, if a user drags three fingers on a touchscreen in a particular direction, then scrolling is accelerated by eight times (8×) the normal scrolling speed.
In a related embodiment, while in accelerated mode, media browser 112 causes a heads-up display (HUD) that indicates context of what images in a media wall are currently being (or about to be) displayed. For example, if a media wall is organized by date, then the HUD may indicate a specific date or a month and year. As another example, if a media wall is organized by location, then the HUD may indicate a city and/or country. As another example, if a media wall is organized by person, then the HUD may indicate a name of a person and, optionally, a date.
In an embodiment, a media wall includes a scroll bar that, when moved based on input from a user, media items in the media wall do not change until the user “let's go” of the scroll bar. While the scroll bar is engaged (e.g., a cursor button is depressed while the scroll bar is selected), a HUD is displayed that includes an example set of media items at the position in the media wall that corresponds to the position of the scroll bar and/or a description of the context. Similar to the HUD described previously, if, for example, the media wall is organized by date, then the description may indicate a specific data, month, or year. If the media wall is organized by location, then the description may indicate a city, country, or geographical location.
In an embodiment, user selection of a media item in a media wall causes media browser 112 to enter a mode that focuses on the media item. For example, if the selected media item is a digital image, then media browser 112 enters a full image mode where the selected digital image occupies a majority of the screen space occupied by media browser 112. Thus, for example, media item 410 occupies a relatively large portion of media wall 400.
Once in full image mode, the user may provide input relative to the selected media item. For example, a user may provide input (e.g., a “pinch close” on a touchscreen) that causes the display of the media item to shrink in size (or “zoom out”) or input (e.g., a “pinch open”) that causes the display of the media item to expand in size (or “zoom in”). As another example, a user may provide input to scroll through media items one at a time, such as by swiping the displayed media item from right to left. As another example, if the selected media item corresponds to a video file, then media browser 112 enters a playback mode where the selected video file is played or at least controls for navigating the video file are displayed, such as play, stop, pause, rewind, and forward controls.
In an embodiment, media item duplicate detection is performed to avoid displaying media items that are identical. In some situations, multiple sources may store a copy of the same media item. For example, a user transfers a digital image from her camera to her laptop computer, after which the user uploads a copy of the digital image from the laptop computer to her Facebook account. Later, when media browser 112 executing on the laptop computer retrieves media items from multiple sources, those sources might include the laptop computer and the user's Facebook account. Thus, media browser 112 might display both copies of the digital image.
Any technique may be used to determine that two media items are copies of each other. For example, the content of the media items may be compared. In the case of digital images, an image footprint algorithm may be used to generate “footprints” of two digital images and then the footprints are compared to determine whether the two digital images are similar enough. If so, then the two digital images are deemed the same. A “footprint” of an image is data that describes one or more features detected in the image. An image footprint is typically much smaller than the image from which the image footprint was generated. Alternatively to an image footprint algorithm, color RMS may be used to determine whether two digital images are the same.
As another example, each media item may include an identifier that may be used as a roughly “unique” identifier. For example, the identifier may be an identifier generated by a camera that generated the initial copy of a media item. The identifier may be transferred along with the media item to one or more other devices. Another proxy UID may be a name of the media item. If the names of two media items are the same, then further analysis may be performed on both media items to determine whether they are the same media item, such as comparing content of the two media items.
As noted previously, media items from multiple sources may be grouped or clustered according to one or more stream criteria, regardless of the sources of the media items. A media item may belong to different media streams, depending on the current one or more stream criteria that are applied at a given moment.
For many users, it is not intuitive to search for media items based on what source the media item is coming from. Rather, it is intuitive to search for media items based on people who are associated with (or depicted in) those media items. In other words, many users do not want to worry about whether certain media items are coming from Photo Stream, Facebook, or Flickr. In an embodiment, media browser 112 allows a user to create (virtual) media streams, one for each person in which the user is interested, regardless of where the media items in those media streams originate. A person may be associated with a media item if the person, for example, caused the media item to be created, owns a device to which the media item was sent, is detected in the media item (such as the person's face or voice), or is identified in metadata associated with the media item.
With the explosion of social media, many users have hundreds of “friends” or contacts. Thus, the list of people whose pictures a typical user has access to is relatively large. To address this issue, in an embodiment, media browser 112 supports the designation of “VIPs” or “Favorites” in order to allow a user to specify which people the user is most interested in. For example, in response to user input associated with a given name, an indicator (e.g., a “star”) is displayed next to the given name. The user input may involve (1) the user right-clicking the given name, which causes a menu to appear, and (2) the user selecting a “VIP” menu option in k link 4520 and through communication interface 4518, whelection input and is being displayed may be displayed with the (e.g., star) indicator. Then, media browser 112 may aggregate all the media items associated with any name in the VIP list into a single media wall, thus, creating a “VIP” media stream.
In the depicted example, each name in the list of names in media menu 610 is associated with a “VIP” star. When a VIP star is filled in, then the associated person is said to have the VIP (i.e., very important person) status. In this example, Ralf Weber has VIP status while Renny G. does not.
In an embodiment, if a media item is not associated with any name that is associated with a VIP status, then that media item is not considered as a candidate when determining which media items to display in a media wall. For example, if the only name that a media item is associated with is Renny G., then that media item will not be displayed.
In a related embodiment, whether a media item is associated with a person with VIP status is a factor in only certain display modes. For example, location and time-based queries for media items may not consider VIP status while a default (or initial) query at startup may utilize VIP status.
In an embodiment, media browser 112 (or another associated component) analyzes names that are associated with media items from multiple sources (whether local or remote) to determine which names match. The names are reflected in metadata associated with the media items. If there are any differences between names from different sources, then the names are treated as being names of different people. For example, “Rachel M.” may have been identified in data from remote source 132 and “Rachel R. M.” may have been identified in data from remote source 134. Alternatively, even if there are differences between names from different sources, the names are analyzed to determine whether they should be treated as identifying the same individual. This latter analysis may involve determining how close the names are to each other or whether there are common misspellings of one or more of the names.
Additionally or alternatively, the analysis may involve analyzing metadata from multiple sources. For example, birthday information indicated in metadata of multiple media items may be used to determine that two names that are similar (but not the same) are of the same person (if, for example, the birthdates are the same) or are of different people (if, for example, the birthdates are different).
Additionally or alternatively, the analysis may involve analyzing content of media items associated with the different names to determine if the names are of the same person. For example, (1) a first digital image associated with one name from a first source and (2) a second digital image associated with another name from a second source may be analyzed to determine whether a face detected in the first digital image is of the same person as a face detected in the second digital image.
If media browser 112 determines that two different names are (or should be) associated with the same person, then media browser 112 stores name association data that associates both names with each other. Later, when the list of names is displayed, both names may be displayed as being associated with each other. Thus, when displaying media items associated with at least one of the names, media browser 112 analyzes the name association data to determine whether the name is associated with one or more other names that have been identified as identifying the same person.
A user may be allowed (through one or more user interface controls (not shown)) to “break” a name association, after which media browser 112 stores name disassociation data that identifies both names and that indicates that both names should not be considered to be of the same person.
Similarly, a user may be allowed to select and “combine” two or more names that are included in the list of names. In response, media browser 112 stores name association data that identifies the two or more names. For example, “Ralf Weber” that is identified in remote source 132 may be known as “TRex83” in remote source 134. If media browser 112 first compares names to determine commonality, then it is unlikely that media browser 112 would ever consider the names “Ralf Weber” and “TRex83” as possibly identifying the same person.
In
Media wall 1000 includes “bubble” icons on some of the media items. A bubble icon on a media item indicates that one or more comments are associated with the media item. The one or more comments include text from different users and/or indications that other users “liked” the media item. For example, media item 1020 may have been retrieved from remote source 132 and, at remote source 132, the one or more comments were stored in association with a copy of media item 1020. Previously, media item 1020 (or a copy thereof) may have been originally uploaded to remote source 132, which allowed “friends” (or contacts) of the user to comment on media item 1020. In an embodiment, if a user selects a bubble icon, such as bubble icon 1022, then the comments associated with the associated media item are displayed.
Media browser 112 allows a user to restrict the scope of media items that are displayed in a media wall by allowing the user to select a group or album. However, there are many cases where such a selection is not what users desire or how users naturally look for media items. Many times, if users want to tell a story or find a picture, they will look for media items that were created (or taken) during a particular time period, in a particular place, or with particular people.
In an embodiment, media browser 112 allows a user to view different “smart groups.” A “smart group” is a group of media items that media browser 112 establishes based on metadata of the media items. In other words, the source from which the media items were retrieved does not create or establish the group. “Smart groups” is one way in which media browser 112 leverages metadata of multiple media items to form new groups. Smart groups allow a user to show all the media items (that satisfy certain criteria) across all (or multiple) of the user's associated sources.
Each smart group is associated with a different type of grouping criteria. For example, a location-based smart group is based on location-related grouping criteria while a temporal-based smart group is based on temporal grouping criteria. Once a smart group is selected, media items from a set of media items are analyzed based on grouping criteria associated with the smart group.
In order to generate the map, media browser 112 analyzes location data associated with each media item retrieved from a particular source (or from multiple sources) and determines a scale of a map based on the location data. For example, the scale is determined such that a resulting map depicts each location identified in the location data, each pin (that identifies a location) is no closer than a first distance from an edge of the map, and the closest pin to the edge of the map is no further than a second distance from an edge of the map.
In an alternative embodiment, instead of displaying a map, media menu 1510 lists names of locations (e.g., country and/or city names). Media menu 1510 may also indicate, for each location name, a number of media items that are associated with the location.
The media items in media wall 1700 may be ordered based on one or more criteria, such as time or popularity. For example, the media items that media browser 112 determines are most popular are depicted in media wall 1700 while less popular media items are displayed later if a user scrolls through media wall 1700.
In an embodiment, media items that are displayed concurrently are organized in a particular manner or layout. In a related embodiment, a user is allowed to change the manner in which media items are organized on the display.
Because media wall 400 is displayed according to an “organic” layout, the “Organic” option is visually distinguished relative to the other layout options. In an “Organic” layout, aspect ratio is not preserved, although aspect ratio preservation may be preferred. In the “Organic” layout option, cropping may be performed, for example, for panoramic images.
In an embodiment, once a set of media items is determined for a media wall based on one or more stream criteria, the set of media items may be further grouped according to one or more grouping criteria. For example, a stream criteria associated with a media wall may be all media items associated with “Ralf Weber.” Then, the media items associated with “Ralf Weber” are grouped by location, such that media items associated with Paris, France are grouped (and displayed) separately from media items associated with London, England. The one or more criteria that is used to group media items that belong to a media stream are referred to herein as “group by criteria.” The group by criteria is compared to metadata or properties associated with media items in a media stream to determine how to group the media items.
Alternatively, a user may have selected “Location” and, in response, media browser 112 groups images based on location. Media browser 112 may select the type of location based on one or more criteria. The groups may be based on city, state or province, country, continent or any other geographical boundaries, whether legal-based or not (such as a forest or desert that spans multiple countries).
Further, different groups based on location may correspond to different types of locations. For example, multiple groups may be based on different cities, another group may be based on particular country, and another group may be based on a forest that spans multiple states or provinces. Such a scenario may be useful if a user is associated with only a few media items in a foreign country (relative to the user) but is associated with many media items in the user's country.
Returning to
The screen shot of
In an embodiment, when a group by option (e.g., “Country”) has already been selected and a group by option button (e.g., button 2610) is selected, then, in response, a “drill down” indicator data is displayed to allow a user to select a subset of the groups that are created based on the group by option. Arrow icon 2630 is an example of “drill down” indicator data.
Six of the seven options identify a different country. The last of the listed options is “Unspecified”, which indicates that media items that are assigned to the “Unspecified” group are not associated with any country or at least that media browser 112 is unable to determine with which countries those media items are associated. For example, a media item might not be associated with geographical coordinates or any other location information. Thus, the media item cannot be assigned to any particular country. As another example, a media item might be associated with geographical coordinates or any other location information but media browser 112 might not be configured to derive the country based on such information.
However, in an embodiment, media browser 112 is configured to determine a country (or other location type, such as city or state) associated with a media item even though metadata associated with the media item does not specify the country (or other locale). For example, if a media item is associated with geographical coordinates, then media browser 112 uses map data and the geographical coordinates to determine the country that corresponds to the geographical coordinates. As another example, if a media item is associated with a name of a geographical area or a city, then media browser 112 uses mapping data and the name to determine the country associated with the area name or city name. As another example, media browser 112 may analyze a media item to detect and identify an object depicted in the media item (such as the Eiffel Tower or the Great Sphinx of Giza). Media browser 112 then uses mapping data that associates objects with locations, which may include a country, to determine the country associated with the media item.
In an embodiment, media browser 112 allows a user to select multiple unselected media items with a combination of touchscreen gestures, referred to herein as “painting selection” gestures.
For example, a long touch on an unselected media item causes “painting selection” mode to be entered. Entering the painting selection mode causes scrolling to be locked, such that as the user moves his finger, additional media items will be displayed and currently-displayed media items will not “move off” the display.
A selected media item may be distinguished from an unselected media item in one of many ways, examples of which include a border of the selected media item being highlighted and a translucent color overlay on the selected media item.
Without releasing the finger that engaged the painting selection mode, the user drags over media items that the user desires to select. If the user drags his finger over media items that are already selected, those media items remain selected. Once the user's selection is complete, the user releases his finger from the touchscreen. At this point, the media wall becomes “scrollable” again.
In an embodiment, the painting selection gestures are used to deselect selected media items, regardless of how the media items were selected. For example, while seven media items on a media wall are selected, a user holds a finger on a selected media item until “painting deselection” mode is entered. Entering the painting deselection mode causes scrolling to be locked. Without releasing the finger that engaged the painting deselection mode, the user drags over media items that the user desires to deselect. If the user drags his finger over media items that are already unselected, those media items remain unselected. Once the user's deselection is complete, the user releases his finger from the touchscreen. At this point, the media wall becomes “scrollable” again.
In an embodiment, while a media wall is in a painting selection (or deselection) mode, a user is able to scroll through the media wall without selecting (or deselecting) media items. Thus, for example, while in painting selection mode and while the user's finger is over a selected media item, a second finger of the user is detected, which causes painting selection to be paused and scrolling to be re-enabled. The user's second finger may drag across the touchscreen, causing the media wall to be scrolled and new media items to be displayed. After the second finger is released, the painting selection mode is reengaged and scrolling is disabled. Selection of media items begins once the first finger moves. Thus, a user may pause scrolling, lift the second finger, and then scroll again with the second finger without selecting the media item that stopped underneath the first finger.
In some instances, a user may desire to see more media items that are similar in some way to a particular media item in a media wall. Examples of similarity include time (such as date or month), location, and people. For example, a user may see a person reflected or depicted in a digital image in a media wall that is organized based on time and then desire to see other media items associated with that person.
Thus, in an embodiment, media items that belong to a media wall are “filtered” based on input relative to a particular media item in the media wall. The user selects the particular media item and selects an appropriate filter. For example, a user selects a digital image that depicts a particular person. The selection may involve (a) double-clicking the digital image or (b) a two-finger long tap gesture. The selection causes a filter UI to be displayed. The filter UI may indicate multiple filter options, such as “Filter same Day,” “Filter same Month,” “Filter same City,” or “Filter these People.”
Once a filter is selected, media items that are associated with the selected media item and are associated with the same filter criteria as the selected media item are highlighted. For example, a user selects “Filter same Day” after selecting a particular media item in a media wall. All other media items (in the media wall) that are associated with the same day as the particular media item are highlighted. “Highlighting” media items may instead involve causing other media items (i.e., that do not “match” a particular media item based on a selected filter) to be dimmed or darkened.
Once a subset of the currently-displayed media items are highlighted (at least relative to one or more other media items in a media wall), a user may scroll through the media wall and see other media items that are highlighted, which indicates that those media items share a criterion (corresponding to the selected filter) in common with the user-selected media item.
In a related embodiment, instead of highlighting media items relative to other media items that are not associated with the same filter criteria as the selected media item, a new media wall is created where the only media items that are included in the media wall are media items that are associated with the same filter criteria as the selected media item. Thus, the new media wall may include a strict subset of the media items that are included in the “unfiltered” media wall.
In an embodiment, the capabilities provided by media browser 112 are leveraged by one or more other applications. Thus, while some examples above describe media browser 112 as a standalone application, another application may integrate media browser 112 into its own functionality or employ an API to leverage the media browsing capabilities of media browser 112. Any application where a user desires to browse media items, such as a photo library, may be able to use media browser 112 as a service, which may be provided by an operating system.
The application may be any software application that allows a user to include or “attach” media items, such as an email application, a word processing application, a media presentation application, a photo management application, etc. For example, if the application is an email application, then a user might want to include a media item as an attachment to an email. Thus, the user may leverage media browser 112 by having the email application communicate with media browser 112 in order to receive and display media items provided by media browser 112. Thus, media browser 112 may be used as a tool by a developer of a software application so that the developer does not have to worry about developing the media item retrieval or the display of media items themselves.
In an embodiment, a user is allowed to submit a query against media items in a media wall or media board. In this way, the media items may be identified without individually selecting media items and without traversing through a media menu hierarchy.
In an embodiment, a user provides voice input that is translated into a search query that is used to search for media items that satisfy the search query. A speech recognition component (which may be part of media browser 112) receives the voice input, translates the voice input into text data, and provides the text data to media browser 112 for searching. For each criterion in the search query, the text data may indicate a type or category of the criterion. For example, a user may speak, “Show me all pictures of my girlfriend and I in New York last year.” The speech recognition component identifies people (“my girlfriend” and “I”), a location (“New York”), and a date range (“last year”). Media browser 112 searches for media items that satisfy the search criteria.
In the examples above, media browser 112 provides a media menu (such as media menu 510) that allows a user to select (or create) media streams and group media items within media streams. Such a media menu is text-based. In an embodiment, media browser 112 provides an icon-based menu, where each icon either corresponds to (a) a container that corresponds to (or includes) multiple containers or (b) a container that includes only media items. The former container is referred to as a “navigational group” while the latter container is referred to as a “leaf group.”
According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
For example,
Computer system 4500 also includes a main memory 4506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 4502 for storing information and instructions to be executed by processor 4504. Main memory 4506 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 4504. Such instructions, when stored in non-transitory storage media accessible to processor 4504, render computer system 4500 into a special-purpose machine that is customized to perform the operations specified in the instructions.
Computer system 4500 further includes a read only memory (ROM) 4508 or other static storage device coupled to bus 4502 for storing static information and instructions for processor 4504. A storage device 4510, such as a magnetic disk or optical disk, is provided and coupled to bus 4502 for storing information and instructions.
Computer system 4500 may be coupled via bus 4502 to a display 4512, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 4514, including alphanumeric and other keys, is coupled to bus 4502 for communicating information and command selections to processor 4504. Another type of user input device is cursor control 4516, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 4504 and for controlling cursor movement on display 4512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
Computer system 4500 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 4500 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 4500 in response to processor 4504 executing one or more sequences of one or more instructions contained in main memory 4506. Such instructions may be read into main memory 4506 from another storage medium, such as storage device 4510. Execution of the sequences of instructions contained in main memory 4506 causes processor 4504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 4510. Volatile media includes dynamic memory, such as main memory 4506. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 4502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 4504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 4500 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 4502. Bus 4502 carries the data to main memory 4506, from which processor 4504 retrieves and executes the instructions. The instructions received by main memory 4506 may optionally be stored on storage device 4510 either before or after execution by processor 4504.
Computer system 4500 also includes a communication interface 4518 coupled to bus 4502. Communication interface 4518 provides a two-way data communication coupling to a network link 4520 that is connected to a local network 4522. For example, communication interface 4518 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 4518 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 4518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
Network link 4520 typically provides data communication through one or more networks to other data devices. For example, network link 4520 may provide a connection through local network 4522 to a host computer 4524 or to data equipment operated by an Internet Service Provider (ISP) 4526. ISP 4526 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 4528. Local network 4522 and Internet 4528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 4520 and through communication interface 4518, which carry the digital data to and from computer system 4500, are example forms of transmission media.
Computer system 4500 can send messages and receive data, including program code, through the network(s), network link 4520 and communication interface 4518. In the Internet example, a server 4530 might transmit a requested code for an application program through Internet 4528, ISP 4526, local network 4522 and communication interface 4518.
The received code may be executed by processor 4504 as it is received, and/or stored in storage device 4510, or other non-volatile storage for later execution.
In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.
This application claims the benefit of U.S. Provisional Application No. 61/783,377, filed Mar. 14, 2013, the entire contents of which is hereby incorporated by reference as if fully set forth herein, under 35 U.S.C. §119(e).
| Number | Date | Country | |
|---|---|---|---|
| 61783377 | Mar 2013 | US |
