Patent Application
20180367838
The present invention related to the field of multimedia presentation on television. More specifically, it's for browsing, finding and viewing personal photos and videos from huge numbers of photos and videos in media libraries across set-top box, TV-connected device, PC, media server, storage device or even cloud photo services with a unified interface.
People have lots of photos and videos stored everywhere such as smartphone, PC, storage device at home and cloud services or social networks on the Internet. While enjoying those photos with family at home together, people typically search the photos or videos from smartphones and cast to TV for everybody to see. The problem with casting is that you only cast one photo at a time from one person's phone and everybody else cannot see the complete collections or whole stories. People usually use voice or keyword search to find interested media from a huge library, but are only able to have limited files returned from search engine. The search results are not persistent or not able to browse further.
In some aspects, a unified TV user interface for personal photo and video browsing in huge or deep media libraries, such as a storage in a TV-connected device, set-top box, PC or media storage in local network, or remote cloud services or social networks is disclosed. In some embodiments, the overall media libraries contain thousands of photos or videos for the whole family from many years, and users are still able to find media easily on TV. The user interface is part of the software that is able to be implemented inside TV, a dedicated TV set-top box, a storage device with TV connection, or a TV app in an existing TV connected device. In an aspect, the user interface is presented as a two layer structure having channels and programs, which are both virtual categories. The channel can be the first layer and the program can be the second layer. In some embodiments, the channel and program structure is related to where media are physically stored. In some other embodiments, the channel and program structure are not related to where media are physically stored.
Channel
In some embodiments, the channel is the top-level category that a user manually creates. In some embodiments, the channel is generated intellectually based on common scenarios or analysis from intelligence algorithms by a software. For example, users are able to manually create a “Super Car” channel, which contains all the super car photos and videos that the user collected. In some embodiments, the system creates channels based on common scenarios such as “Travel”, for media taken away from home. In some embodiments, the channels for “Event” are created by a user or by the system for photos taken in the same place within a few hours timespan (e.g., same day or within 24 hrs). In some embodiments, the channels for “Family” are created by a user or by the system for photos and videos containing faces of the family members. By using a software intelligent analysis of the media collections by face, the system is able to automatically generate a channel like “Kids” by finding all photos and videos containing children. With a similar mechanism, the software is able to identify various sorting criteria for generating various channels.
In some embodiments, the media contents that are selected are “of” the person/user or “related to” to the person/user. The factors that are used of searching, categorizing, or sorting include a weight ratio that is related to the person or user. For example, even if the user only enters “Kids” as search criteria, the software automatically uses dual factors as searching criteria including the name “John” of the user (e.g., by comparing/matching voice fingerprint to identify that it is John providing the search input) and “Kids” for searching the media collections. As such, the software will have a search code as (INPUT=SEARCH “JOHN” and “KIDS.”) In another example, the relationship is also included in the search command, such as (INPUT=SEARCH “JOHN'S KIDS.”) The weight ratio can be based on the importance or closeness of the relationships. For example, for a search of the term “Kids,” the search weight ratio can be 100% when only one kid that is John's kid is in a photo. The search weight ratio can be 70% when there are John's kid and another kid that is not John's kid in the photo or video. The media content (e.g., the photo, sounds, or video) can be categorized and/or sorted (e.g., category, types, or presenting priority) based on the scores or numbers of the weight ratio.
When the software (e.g., smart computing system or device that installed the software or smart software) learns that John Hack has two kids (e.g., from FACEBOOK®'s face and relationship tagging) as Peter Hack and Kevin Hack. The software can use various social media or online search engines to automatically identify various personal relationships. For example, the software can automatically generates two Programs as “Peter Hack” and “Kevin Hack,” which are the subset/secondary tier of the structure within the channel (e.g., first or primary tier) of “Kids” or “John's Kids”. In some embodiments, the software obtains the user's permission to create the new programs of “Peter Hack ” and “Kevin Hack.”
In some embodiments, the software uses its intelligent analysis of the media collections by location. The system automatically generates a channel like “Asian Travel”, which contains photos taken from Asian cities. By intelligent analysis of the media collection by theme, the system can automatically generate a channel like “Water Sports” for those photos containing swimming pool or beaches. Above are some examples of channel characterization, and other types of channels are within the scope of the present disclosure.
In some aspects, the channels are obtained from the cloud services. For example, if a user grants access to their FACEBOOK® page, the system is able to create a “Facebook” channel containing all photos the user posted on FACEBOOK®. Similar mechanisms are applicable to other social media, websites, or smart phone APPs, such as YOUTUBE®, INSTAGRAM®, or FLICKR®.
In some aspects, the system generates a personalized and unique channel list based on the photos and videos from their own media libraries. It is also unified since users can see all their media collections in one user interface (e.g., on TV). Unified media browsing interface for all cloud accounts (YOUTUBE®, INSTAGRAM®, or FLICKR®, GOOGLE® Photos) and local storage in the TV, set-top box, PC or storage device in the same local network.
In some embodiments after the channels are created, automatically or manually, the contents in the channels are not static and are able to be dynamically updated. The contents in the channel are able to grow over time when new photos or videos meet the criteria of the channels. In some aspects, the channels are deleted, renamed or shared to other people.
Program
In some embodiments, the program is set as a second-level category, in which the system is able to generate the program automatically based on the characteristic of the channel. For example, the photos in the “Event' channel will be arranged by date/time of the event. The photos in the “Asian Travel” channel will be arranged by location of the photos taken.
In some aspects, unique channel and program arrangement are based on media metadata, such as date/time, location, theme (outdoor, indoor, nature, party, water/ocean, etc), device type (SLR, smartphone, action camera, security camera) and are able to be also based on the storage places, such as some cloud storage or services.
In some embodiments, the channel and program structure are mainly designed to be presented on a TV screen, since most of the TV has resolution of 1280×720 (720p) or more with horizontal layout. It is an advantageous feature to display channels on the side of the screen, but still have plenty of space in the center for displaying multiple program thumbnails, which are still able to be seen clear with size of 300×200 pixels or more. In some embodiments, the channel and program structure are used in smaller screens, such as smartphones or tablets, with a different layout. In some embodiments, the software disclosed herein is able to take/sample only a portion (e.g., 1/10-1/1000 of the original file) of the media content (e.g., photo and video) and display as thumbnail or smaller (e.g., less pixels or less resolution) of a photo on the TV, so that that the software does not need to download the entire media content to be displayed.
In some embodiments, the system includes a touch-based remote control (e.g., a device) to easily browse and find specific photos from various channels and programs and even enjoy it as a full-screen mode, because the TV is typically big and far away from where the users sit. With two layers of structure (e.g., channel and program), a user can easily find any photos from thousands of photo and video collections. In some aspects, the touch-based remote control is able to be implemented as an app in a smartphone or dedicated hardware with touch screen display.
In some embodiments, the screen in the remote control is changed along with the TV screen, concurrently or with some time shift. When the TV screen is showing the “channel view”, which contains a list of channels and programs from the current selected channel, the remote control displays the similar layout so when the user scrolls, selects or presses the UI on remote and the TV reflects such change accordingly. When the TV enters the “program view”, which shows the photos within a program. The remote screen shows the similar layout and allows users to go to the previous or next program, play slideshow or show particular photo on full-screen.
Other features and advantages of the present invention will become apparent after reviewing the detailed description of the embodiments set forth below.
Embodiments will now be described by way of examples, with reference to the accompanying drawings which are meant to be exemplary and not limiting. For all figures mentioned herein, like numbered elements refer to like elements throughout.
Reference is made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings. While the invention is described in conjunction with the embodiments below, it is understood that they are not intended to limit the invention to these embodiments and examples. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which can be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, numerous specific details are set forth in order to more fully illustrate the present invention. However, it is apparent to one of ordinary skill in the prior art having the benefit of this disclosure that the present invention can be practiced without these specific details. In other instances, well-known methods and procedures, components and processes have not been described in detail so as not to unnecessarily obscure aspects of the present invention. It is, of course, appreciated that in the development of any such actual implementation, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, such as compliance with application and business related constraints, and that these specific goals vary from one implementation to another and from one developer to another. Moreover, it is appreciated that such a development effort can be complex and time-consuming, but is nevertheless a routine undertaking of engineering for those of ordinary skill in the art having the benefit of this disclosure.
There are so many typical Smart TVs that are Internet connected or TV connected streaming devices that allow a user to watch Internet contents on TV. Better improvement of the image and video presentation and control are needed. It is still inconvenient to use the typical media presentation devices and method today. For example, when people want to show photos or videos on TV, usually they need to use different devices, software, or mobile APP in order to find the photos stored in different places. For example, when a user wants to show a picture from a PC in the home network, they need to use a DLNA compatible TV or streaming device to find a picture from a PC with compatible software running as media server. When a user wants to show a picture from Google Photos, the user needs to open the APP on the mobile device and cast it to the TV, one photo at a time.
In the following, some exemplary embodiments are disclosed. A Unified User Interface (UUI) with the video and photo presentation methods in accordance with some embodiments are able to be implemented inside a TV, a TV streaming device or any TV connected hardware that can also connect to user's home network and Internet. In some embodiments, a unified user interface is provided that not only allows users to browse and find photos and videos stored in any devices, but also in an organized way in a channel and program structure, which a user can manually arrange as well as taking advantage of state-of-the-art computer vision algorithms and software to automatically arrange for the user.
In some embodiments, a UUI 202 (e.g., a Unified User Interface) comprises channels 204 and programs 206, which pull and gather photos, video, and other media from the various sources described above to the UUI 202. In some embodiments, the UUI 202 comprises stored credentials or passwords for automatically accessing or maintaining the accessibility to various devices and services, such as FACEBOOK®. In some embodiments, the UUI 202 automatically negotiates or exchanges credentials or accesses with various media storage 218, so that the UUI 202 can access the media and requests the media to be downloaded, displayed, or streamed to the UUI 202. Using the UUI 202, the users do not need to use many different APPS and different configurations and do not need to manually connect cables from a particular device to TV and display when people want to share photos from these devices to TV in front of the whole family.
At a step 256, the “channel creator” uses a built-in intelligent media indexing service to analyze the photo or video contents and its corresponding metadata, along with other media files coming along with this media file. For example, all photos taken from a party event contains similar people and took place at the same location within a few hours timespan on the same date. Then the system can decide to include all these photos or videos in the “Event” channels. In another case, when the user imports the casual family photos at home, the system can recognize the family member with face recognition technologies and determine that these photos should go to a “Family photo” channel. In another example, if the imported photos are taken in some tourist locations, the system will find all related photos and videos taken in a few days span in the nearby areas, and decide to put them into the “Travel” channel.
At the step 254, the “Program Creator” starts to sort all the media in a channel with a few meaningful sorting mechanisms, such as date/time, location, people, theme, etc. The reason for several different sorting mechanisms is that each sorting mechanism is able to make the sorting more useful and easier to use/view for the users and audience when the media are collected in one channel rather than sorting them when all the uncategorized media are gathered, which is able to save computing time or speed up the computing efficiency by grouping first and then sorting in that group.
In some embodiments, the system determines the appropriate sorting methods, but a user can change it or override it from the UUI when they want to sort in the other ways. When the user requests the sorting method changes, the UUI is able to respond immediately since all the necessary metadata generated by the “Program Creator” and “Channel Creator” are stored locally in the “Index database” so the system does not need to sort all the media in the storage on the fly since the media may not be able to access if they are stored in the Internet. In certain scenarios, the full resolution of the media is not needed, since the display can have much less resolution than the original media. While retrieving necessary channel and program indexes from the local database and the smaller resolution media from the original storage location, the user can easily browse and find the favorite media from huge media libraries across multiple storages easily.
At a step 258, the “Dynamic Channel/Program Presenter” (the “presenter”) is the module to compose the channel and program on TV screen based on user's input from “UUI handler” (the “handler”). At a step 260, the “UUI handler” takes the user input from a remote App or hardware remote device to inform the presenter which channel and program to show, and the presenter pulls the actual media information out of the database and also retrieves the media itself from the original storage and composes them on the TV screen. The term UUI or UI used in the present disclosure is able to also include the backend operating systems, devices, processes, and methods. For example, although the term UUI or UI is used, it can also include the hardware and software that are needed to perform such action, including a hard drive, a CPU, a memory, and software/intelligent software programs.
The UUI handler takes the user input for channel selector, channel scrolling, program selector, sorting changes, and media playback. The system includes the remote control design to all faster scrolling on channels and programs from huge media data libraries across multiple places.
For example as shown in the step 364, if there is an existing “Travel” channel, when user imports media files taken from a recent vocation, the system can automatically import them to the existing “Travel” channel because it met the criteria of the “Travel” channel. On the other hard, if the “Travel” channel does not exist before the import, the system will automatically recommend a new “Travel” channel to keep these media files. The UUI is able to inquire or receive channel and program information from other connected or related device or services for its own use.
At a step 366, the user can also manually create their own media channel for their own media collections for their specific purpose. The user can also specify the criteria for the channels so all future media meeting the criteria, will be included in the media channels as well. The UUI is able to inquire or receive channel and program information from other connected or related devices or services, for how others are manually creating their channels and systems, so that the UUI can use that information to automatically create the same or similar structure therewithin.
At a step 368, the “Program Creator” analyzes all media in each of the channels and creates many sorting mechanisms. For example, the “Travel” channel is able to have the default sorting method as “Location,” since usually there are many photos taken in one place and move on to the other place. A user is able to change the sorting methods to “Date/Time” if they remember the time when they visited a particular place. In another case, the default sorting method is able to be “People” for the “Family Time” channel. Since most of the photos in this channel can be taken at home so the user may want to see the picture of the particular family member as a group. When the photos of a particular person are grouped together, the user may easily see how the person grows if they are a child.
Sorting Criteria
In some embodiments, the sorting criteria comprise dual factors for sorting and/or categorizing. In some embodiments, the sorting criteria contains “of the person”, such as “personal,” “his/her personal effect,” or “related to the user.” Various degrees or weight factors of the person can be used. In one example, “of the person” is set as a default factor. When a user inputs “travel” as a search or sorting criteria, the device and system disclosed herein search both “related to the user” and “travel” dual factors. In use, the user input “travel this year” as a sorting command, the system and device disclosed herein search photos and/or videos that are taken while travel or related to travel in this year and also only photos that are related to the user himself/herself. In other words, photos and videos that are in the news report not related to the user are not selected or sorted. In some embodiments, the factor “relating to the user” is shown to the user or viewer. In other embodiments, the factor “relating to the user” is concealed or not revealed to the user or viewer. In such an example, since the system actively does not select, delete/remove, skip the media files that are not related or have a low relevance (e.g., weight factor less than 30), the computing device is able to categorize or sort in a more efficient way, which saves computing resources.
In some embodiments, the media is structured to be selected and stored in channels 402, as a method of sorting different media of different characteristics into various channels. In the channels, the media are further sorted into programs 404, which are media (e.g., photos, sound, and video) of the same category and/or characteristic.
In some embodiments, the channels 402 are able to be sorted by categories, collections, and live streams. In some embodiments, the channels are categorized and created by user-specific topics, including “Travel,” “Family,” and “Events” as family activity related. In some embodiments, the media are categorized into one or more specific channels when one or more characteristics are detected or found. For example, in a “Kids” channel, photos containing young kids and/or toys are characterized as photos in the “Kids” channel, which can be selected, sorted, and categorized along with the personal factors.
In some embodiments, the characteristics are identified automatically, such as using a face identification software, an object identification software, a graphic analysis software, a content analysis and image captioning software, and/or the metadata. The metadata can include technical metadata, descriptive metadata, and administrative metadata. In some embodiments, the descriptive metadata comprises captions, headlines, titles, keywords, and location of capture. In some embodiments, the types of the metadata include IPTC-IIM, IPTC Core & Extension, PLUS, XMP, EXIF, and Dublin Core. Using the metadata, the media is able to be attributed to or characterized with one or more specific characteristics. For example, when the location of a photo stored in the metadata is not the place of a usual visit or with personal effects, such as home, office, or frequently visited restaurants, the photo is able to be tagged or characterized as a traveling photo or a candidate for a traveling photo.
In some embodiments, the identification of the characteristics is performed manually. A user is able to enter the characteristics of the photo by verbally or typing. In some embodiments, a user is able to enter the characteristics for the first few photos, and machine learning is used to automatically identify the persons, objects, or locations in the photo through the similarity of the objects (e.g., >65% of image similarity) or recognizable patterns.
In some embodiments, if the location of the photo taken is beyond a predefined distance from work or home, the photo is tagged or characterized as a travel photo or a candidate of the travel photo. In some embodiments, the GPS location of where the photo is taken is determined as one of popular sites for visitors or the image is determined as a famous object or landmark (e.g., the Golden Gate Bridge), the photo is tagged or characterized as a travel photo or a candidate for a travel photo.
In some embodiments, the media for the “family” channel is able to be determined by a user's definition via a relationship input. In some embodiments, the family relationship is learned by the device or the cloud server via other connections, such as the photo labeling functions on Facebook®. Any methods and mechanisms that are able to be used to identify the relationships between the device user/owner and the person in the media are within the scope of the system herein.
In some embodiments, the media for the “events” channel is able to be determined by location, time, person or content of the image, or a combination thereof of the photo/media is taken. For example, the GPS location recorded as Liberty Island of New York is able to be determined as a vacation or sightseeing event for the event channel. In another example, when the recording time of a photo is taken during the day that the user's calendar is marked as son's birthday or when the photo is analyzed with a birthday cake, the photos are tagged or attributed as birthday photos.
In some embodiments, the channels function as dynamic or living storage. Users are able to continuously update photos that have similar characteristics for the specific channel. In some embodiments, the photos or media are automatically assigned one or more characteristics based on the above detectable characteristics (e.g., the GPS location), so that the photos and media are automatically grouped/stored with the photos and media with the same or similar characteristics. In some embodiments, a user is able to browse the photos and media as the channels in a TV program guide. In some embodiments, the system is configured to automatically organize into “programs” or “albums” based on the sorting methods, including dates, locations, contributors, and scenes. In some embodiments, the users manually create “programs,” which are used as channels. In some embodiments, each of the programs is a collection of photos that contain information matches one or more of the sorting criteria (e.g., the characteristics). In some embodiments, each of the programs is dynamically generated, such that the users do not need to manually update the programs.
Using the system 500, the numbers of photos and media that are needed to be loaded into the computing memory are reduced, since pre-qualified photos and media (e.g., matching the sorting/attributing criteria) are reduced. Such advanced features enhance the computing efficiency and computing speed by narrowing down to less qualified items to be read and/or loaded.
In one example, when “Family Trip 410” in
In some embodiments, the sorting methods of the system 500 use various sorting factors (e.g., priority), processes, criteria, and/or characteristics. For example, the sorting of the photos and media is able to be done based on “contributors,” “devices,” “scenes,” and any other sorting criteria. In some embodiments, the system 500 sorts the photos and media based on “contributors,” which is a sorting method based on the owner of the photos. For example, the system sorts the photos according to the alphabetical order of the last name of the photo owners. In some embodiments, the sorting method of the system 500 sorts the photos and media using “devices” as the sorting criteria. The photos in the system 500 are able to be a collection of photos and media from various devices (e.g., smart phone, notebook computer, and tablet). The system 500 is able to sort the photos and media based on the source of the devices.
In some embodiments, the sorting method of the system 500 is able to automatically detect the scenes in the photo/media and arrange the photos and media based on the scenes detected. For example, the scenes detected can be beach, sky, building, people, animal, and sunset. In some embodiments, each of the photos or media is only attributed to one of the characteristics, so each of the photos or media will only appear in one channel or be attributed to one characteristic. In the above case, only the most highest priority or highest relevance tag or characteristic is used as the sorting criteria. In some embodiments, multiple characteristics are used as factors in deciding/factoring which channel or channels that the photo or media should be attributed to.
In a typical user interface, if channel/program arrangement is browsed through with the typical TV remote control, it would still be very difficult to use since it usually has only 4-way arrow key and numeric keys. Even with some touch based remotes from Apple TV, Roku or Amazon Fire TV, they are still not good enough.
In some embodiments, a remote control is provided to deal with a multiple focus group, left channel group and middle program group, and top menu selection in accordance with some embodiments.
In some embodiments, a touch screen based remote control contains a rectangle touch pad or remote control app on a smartphone. The user interface 1002 contains multiple regions 1004 (e.g., touch screen buttons) associated with TV screens. On the channel screen 1006, scrolling on the left pane is scrolling through the channel list up and down. Scrolling on the middle pane 1008 is scrolling through the programs up and down since this can be multiple pages. The faster the user scrolls on the remote, the faster the program moves as well. On the UUI, a user can also see the cover photo 1010 of the programs in the remote control to select the program to view. The icons on the channel screen 1006 can be thumbnail photos, illustration figures, texts, or a combination thereof.
In some embodiments, the screen layout on the touch remote control 1000 matches on TV and includes the same text and picture thumbnails that are displayed on TV. In some embodiments, the remote control 1000 contains a mobile App on smartphones or a dedicated hardware with display, touch and gesture detection. A few boxes on the top 1004 are associated with the top menu with the same name shown on the TV. The left panel is the channel 1006 lists, which contains a scroll bar for sliding up and down, faster or slower and stop when the targeted channel is selected on TV. The program thumbnails 1008 on the right bottom should be changed based on the channel selected.
To utilize, the media (e.g., photos and videos) are presented in channels and programs as a two layer vertical structure.
In operation, the media is acquired, one or more characteristics are associated or attributed to the media, and the media is sorted based on its characteristics. The sorting of the media can be performed before a user makes a selection, such that the media has been pre-sorted and/or reallocated in the storage location, which enhances the loading and computing efficiency by shortening the sorting time and calculation time when a selection command is made.
The present invention has been described in terms of specific embodiments incorporating details to facilitate the understanding of principles of construction and operation of the invention. Such reference herein to specific embodiments and details thereof is not intended to limit the scope of the claims appended hereto. It is readily apparent to one skilled in the art that other various modifications can be made in the embodiment chosen for illustration without departing from the spirit and scope of the invention as defined by the claims. Features in various examples or embodiments are applicable throughout the Present Specification.
This application is a continuation-in-part application (CIP) of a co-pending U.S. patent application Ser. No. 15/406,956, filed Jan. 16, 2017 and titled, “DEVICE FOR AND METHOD OF TV STREAMING AND DOWNLOADING FOR PERSONAL PHOTOS AND VIDEOS PRESENTATION ON TV THAT SEAMLESSLY INTEGRATES WITH MOBILE APPLICATION AND CLOUD MEDIA SERVER,” which claims priority under 35 U.S.C. § 119(e) of the U.S. Provisional Patent Application Ser. No. 62/280,983, filed Jan. 20, 2016 and titled, “DEVICE FOR AND METHOD OF TV STREAMING FOR PERSONAL PHOTOS AND VIDEOS PRESENTATION ON TV THAT SEAMLESSLY INTEGRATES WITH MOBILE APP AND CLOUD MEDIA SERVER,” which are also hereby incorporated by reference in their entirety for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 62280983 | Jan 2016 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15406956 | Jan 2017 | US |
| Child | 16113915 | US |
