Patent Grant
10834454
The present disclosure relates generally to an augmented reality interactive system and more generally to an augmented reality interactive system able to provide additional information related to content being viewed.
This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
As users gain access to larger amounts of content and begin viewing content from a point after a series has begun or miss episodes of a series, then additional information may be necessary to provide background in order for the user to understand story lines or unfamiliar characters in scenes of the content. If a user does not understand a story, has missed an episode and cannot follow the current storyline, or is unaware of/unfamiliar with characters introduced during a missed episode, the user may lose interest and stop watching the content.
In order to retain viewer interest, broadcasters need to find ways to provide additional information to users as they are engaged in the content to assist in their understanding of the story and characters to prevent the user from becoming confused or lost by the story or characters.
Consequently, it is important to develop technology that is able to supplement the viewing of content with additional information to maintain a user's interest in the content so they continue to view future episodes of the content.
Current augmented reality (AR) systems cannot replace certain items in a background of the content with items which the user may find more pleasing or replace certain items with items more appropriate for users of a certain age. Current AR systems cannot replace or overlay certain specific items appearing within viewed content in an individualized manner allowing each user to view content which is personalized based on their viewer profile.
Current Augmented Reality (AR) interactive systems cannot create an enhanced viewing environment enabling an Augmented Reality (AR) user to feel as if they are part of the scene being displayed. Current AR interactive systems cannot supplement the viewing of content by providing an enhanced environment surrounding the screen of the content display device. Current AR interactive systems cannot provide a background image surrounding content being displayed in an individualized manner allowing each AR user to view enhanced content which is personalized based on their user profile.
An augmented reality (AR) interactive system and method is provided. In one embodiment additional information related to content is provided by the augmented reality system. A content viewing history for each specific user within a profile of the augmented reality system is gathered. The content is provided for display using the augmented reality system. An event occurring in content provided for display that is related to other content not contained within the viewing history is detected. Additional information related to the detected event is correlated. The additional information related to the detected event is presented. The presentation is effected by the augmented reality system. In another embodiment gathering the content viewing history comprises gathering information of a viewing history of a user via access to a database. In a further embodiment providing the content using the augmented reality system comprises providing the content for display on a lens of augmented reality glasses. In a still further embodiment detecting an event occurring in the content that is related to other content that is not contained within the viewing history comprises detecting metadata in a scene within the content provided for display that is absent from the content viewing history. In one embodiment the detected metadata includes additional information identifying at least one of a previous episode, a precursor series, a plot background and a character introduction that is absent from the viewing history of the user of the augmented reality system. In a still further embodiment correlating additional information related to the detected event comprises searching a database for additional information related to the event. In an even further embodiment presenting the additional information related to the detected event comprises presenting the additional information as simultaneously with the displayed content in the augmented reality system. In one embodiment, the additional data may be presented as subtitles. Presenting the additional information related to the other detected event comprises pausing the displayed content such that the additional information can be presented to the user before resuming content presentation in a further embodiment. Presenting the additional information related to the detected event during a pause of the displayed content comprises presenting the additional information as subtitles in a still further embodiment. In another embodiment the additional information includes at least one of subtitles for conversations according to the users' preferred language or hearing abilities, information about the plot, information about previous episodes, character backgrounds and language/cultural connotations based on the users' viewing history and profile. In a further embodiment, providing the additional information includes receiving information regarding a position of the user interface relative to the content viewing device, determining a position for projection of the additional information and providing data identifying the position for projection of the additional information to the user interface. In one embodiment the information regarding a position of the user interface relative to the content viewing device includes an anchoring home position between the user interface and the content viewing device; and a current position of the user interface. In a still further embodiment projection of the additional information is not moved relative to the content viewing device when a center point of a field of view of the user interface is moved.
In another embodiment, an augmented reality (AR) interactive system is provided that comprises a controller device for providing additional information related to audio/video content being displayed. The controller device includes an interface for receiving audio/video content including metadata identifying characteristics of the audio/video content for display. A transmitter/receiver communicates with a user interface. A storage device stores user profiles and viewing history. A processor extracts the metadata from the audio/video content and compares the metadata to the user profile and viewing history to determine if data in the user profile and viewing history match the metadata. A router requests additional information from a remote server related to the audio/video content upon determining at least one of the user profile and viewing history do not match the metadata. The additional information received is provided to the user interface via the transmitter/receiver for display. In another embodiment the storage device comprises a user profile database for storing the user profile, the user profile including user preference information including language and user presentation preferences. In a further embodiment the storage comprises a viewing history database for storing the viewing history, the viewing history including information related to a viewing history of a user. The processor provides the additional information for display on a lens of augmented reality glasses in another embodiment. In a further embodiment the processor detects an event occurring in the content that is related to other content that is not contained within the viewing history by detecting metadata in a scene within the content provided for display that is absent from the content viewing history. In a still further embodiment the metadata detected by the processor includes additional information identifying at least one of a previous episode, a precursor series, a plot background and a character introduction that is absent from the viewing history of the user of the augmented reality system. In an even further embodiment the processor accesses additional information related to the detected event by searching a remote database for additional information related to the event. The processor presents the additional information related to the detected event simultaneously with the displayed content in the augmented reality system in one embodiment. The additional information can be presented as subtitles. The processor presents the additional information related to the other detected event by pausing the displayed content such that the additional information can be presented to the user before resuming content presentation in a still further embodiment. In an even further embodiment the processor presents the additional information related to the detected event during a pause of the displayed content by presenting the additional information as subtitles. In a yet further embodiment the additional information requested by the router includes at least one of subtitles for conversations according to the users' preferred language or hearing abilities, information about the plot, information about previous episodes, character backgrounds and language/cultural connotations based on the users' viewing history and profile. In an even further embodiment the processor receives information regarding a position of the user interface relative to the content display device, determines a position for projection of the additional information, and provides data identifying the position for projection of the additional information to the user interface with the replacement image. In a still further embodiment the information regarding a position of the user interface relative to the content display device includes an anchoring home position between the user interface and the content viewing device; and a current position of the user interface. In a yet further embodiment projection of the additional information is not moved relative to the content viewing device when a center point of a field of view of the user interface is moved.
In one embodiment, a method of presenting personalized additional information along with displayed content includes accessing a content viewing history and a user profile. Detecting an event by extracting metadata related to content provided for display, the content provided for display related to other content not contained within the content viewing history. Accessing additional information related to the detected event, and presenting the additional information related to the detected event for display on a content display device. The additional information may be presented as subtitles on the content display device. In an embodiment, presenting the additional information related to the detected event for display includes presenting the additional information to a user interface of an augmented reality system. The method may further include comparing the user profile and the metadata to determine if an object within the content is identified to be overlaid on the display and then select a replacement image for overlay of the object. The replacement image is then provided for display on the content display device. A provision of the replacement image includes receiving information regarding a position of a user interface of an augmented reality system relative to the content display device, determining a position for projection of the replacement image, and providing data identifying the position for projection of the replacement image to the user interface.
In an embodiment, the method may include accessing a background image from the extracted metadata associated with the content. The background content is provided to a user interface of an augmented reality system for projection at a position surrounding the content displayed on the content display device.
Providing the background content includes receiving information regarding a position of the user interface relative to the content viewing device and determining a position for projection of the background image. Data identifying the position for projection of the background image to the user interface is provided.
In one aspect of the embodiments, accessing a content viewing history and a user profile included accessing information of a viewing history of a user, and language and user presentation preferences via access of one or more databases. Detecting an event by extracting metadata related to content provided for display, where the content provided for display is related to other content not contained within the content viewing history includes detecting metadata in a scene within the content provided for display that is absent from the content viewing history. The detected metadata includes information identifying at least one of a previous episode, a precursor series, a plot background and a character introduction that is absent from the viewing history of the user.
In one embodiment, a controller device for providing additional information related to audio/video content being displayed includes a display interface driving a display of the audio/video content, a transmitter/receiver for communicating with a network providing the audio/video content including metadata identifying characteristics of the audio/video content, a storage device for storing a user profile and viewing history, and a processor for extracting the metadata from the audio/video content and comparing the metadata to the user profile and viewing history to determine if data in the user profile and viewing history match the metadata. A router requests additional information from a remote server related to the audio/video content upon determining that at least one of the user profile and viewing history do not match the metadata, wherein the additional information received is provided to the display interface for display on a content display device. The additional content may be displayed as subtitles.
In one embodiment, the controller device further includes a transmitter/receiver for a user interface of an augmented reality system. The storage device includes a user profile database for storing the user profile, the user profile including user preference information including language and user presentation preferences, and a viewing history database for storing the viewing history, the viewing history including information related to a viewing history of a user. The processor of the controller device detects an event occurring in the content that is related to other content that is not contained within the viewing history by detecting metadata in a scene within the content provided for display that is absent from the content viewing history.
In an embodiment of the controller device, the processor is configured to receive information regarding a position of the user interface relative to the content display device, determine a position for projection of a replacement image within the content for display, and provide data identifying the position for projection of the replacement image to the user interface with the replacement image. In an embodiment, the processor of the controller device is configured to access a background image from extracted metadata associated with the content, and provide the background content to the user interface for projection at a position surrounding the content displayed on the content display device.
In another embodiment, the inventive concepts herein may be embodied as instructions on a computer-readable media or data carrier, which, when executed by a computer, cause the computer to carry out the method described herein. Additional features and advantages are realized through the techniques of the present disclosure. Other embodiments and aspects of the configuration are described in detail herein. For a better understanding of the configuration with advantages and features, refer to the description and to the drawings.
The invention will be better understood and illustrated by means of the following embodiment and execution examples, in no way limitative, with reference to the appended figures on which:
Wherever possible, the same reference numerals will be used throughout the figures to refer to the same or like parts.
It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for purposes of clarity, many other elements found in typical digital multimedia content delivery methods and systems. However, because such elements are well known in the art, a detailed discussion of all such elements is not provided herein. In
Augmented reality (AR) systems such as the one depicted in
The viewing history may also be manually or automatically updated when the user views new content. Analysis of the viewing history and user profile may enable a determination of background images that may be pleasing to the AR user. The viewing history may also be manually or automatically updated when the AR user views new content.
When content is received, metadata embedded in the content may be extracted and compared with the viewing history and sensitivities to determine if content being viewed includes information not within the viewing history or that the user may benefit from additional information related to a scene of or characters within the content being reproduced. Such information may be related to a character in the scene or a story line introduced during an episode of the series, which the viewing history indicates the user did not see. The additional information may include background information that would assist a user in understanding of the story such as providing identifying information on a character or aspects of the plot line with which the viewer may not be familiar. The additional information may also include additional context to help in the understanding of actions or phrases used by characters appearing in the content and unfamiliar to the user. The metadata may include information identifying at least one of a previous episode, a precursor series, a plot background and a character introduction. Metadata embedded in the content may be extracted and compared with sensitivities indicated by the viewer profile to determine if items within the content being viewed may be objectionable to the AR user, may be inappropriate for the AR user to see or may be replaced by an item more pleasing to the AR user. Additionally, the replacement/overlay of items may be personalized for each AR user using and logged in to a user interface, i.e. the replacement items seen by each AR user through a personal user interface may be different based on each AR user's particular user profile and/or viewing history. Additionally, the replacement items for replacement of displayed content items for each AR user may be coordinated with each other and the scene in which they are placed to maintain conformity and not distract from viewing of the content. The replacement images may include but not limited to any of static images, video or 3D rendered objects. The replacement images are preferably 3D rendered objects when the content being overlaid is a 3D video or image.
In one aspect of the configuration, the AR system can also display images on a background of the room where the display device is displaying content. Thus, the room environment becomes an enhanced background environment surrounding the displayed content. In this context, background images are those images that are displayed to the user of an AR system that appear overlaid upon the room that surrounds the content display device. When content is received, metadata embedded in the content may be extracted and compared with or used in conjunction with the user profile to determine if a particular background image would be appropriate based on the content and preferences of the AR user. Additionally, the metadata may be used to search for background images related to the content or that include background images that are appropriate based on the content being displayed. The background images are generally related to the setting of the scene within the content being displayed. The background images may also be based on the user profile and sensitivities identified by the user profile indicating certain preferences of the AR user and used to enhance the viewing experience associated with the displayed content. Additionally, the background images may be personalized for each AR user using and logged in to a user interface, i.e. the background images seen by each AR user through a personal user interface may be different based on each AR user's particular user profile and/or viewing history. Additionally, the background images for each AR user may be coordinated with the scene it surrounds to maintain conformity and not distract from viewing of the content. The background image may include but is not limited to any of static images, video or 3D rendered objects. The background image is preferably 3D rendered objects when the content being surrounded and enhanced is a 3D video or image.
It is noted that additional information for display as subtitles in the displayed content does not mandate the use of an AR system. Such additional information to be displayed may be placed as subtitles to be displayed superimposed on displayable content. The content may be displayed on any suitable display device regardless of the use of an AR system. However, added enjoyment to the user of an AR system may be achieved using the principles of the disclosure.
A detected event occurring in the content that is related to other content not contained within the viewing history of a user identifies that additional information may be of assistance to the user. Additionally, the user profile may be compared with the metadata to determine if any cultural or language connotations present in the content may be unfamiliar to the viewer based on their background information. If it is determined a user may benefit from additional information related to the scene or explanation of cultural or language connotations, additional information explaining these connotations may be obtained. Upon determining that additional information should be provided to a user, the STB/router 110 may contact the remote server 140 to request this information. The additional information may be preferably stored in a remote database on the remote server 140 and accessed via the STB/router 110 when the comparison indicates additional information should be provided to the viewer. In one embodiment, the additional information is displayed to the viewer as subtitles in either a system that uses an AR user interface or in a system that displays contents without an AR interface.
The remote server 140 may be a general server that receives and stores information from broadcasters, a server associated with a content provider such as a cable or satellite company or a server controlled by a broadcaster of content. Alternatively, the server may be included at the STB/router 110 whereby the server receives and stores the additional information upon extraction of metadata from content to which the STB/router 110 is tuned. Preferably, the metadata will include information on how and where to access any additional information that is determined should be provided to a viewer. The STB/router 110 will then provide the additional information to the user interface 120. The user interface 120 may project the additional information onto a lens in the line of sight but in a position that does not obstruct a view of the content being displayed so it can be seen by the viewer. Alternatively, the additional information related to the detected event may be presented as subtitles simultaneously with the displayed content.
Presenting the additional information related to the other detected event may include pausing the displayed content such that the additional information can be presented to the user before resuming content presentation. Presenting the additional information related to the detected event during a pause of the displayed content may present the additional information as subtitles. Further, the additional information may include at least one of subtitles for conversations according to the users' preferred language or hearing abilities, information about the plot, information about previous episodes, character backgrounds and language/cultural connotations based on the users' viewing history and profile.
The additional information received may include information on the position at which the information is projected onto the lens or display of the user interface 120. The comparison of the metadata and viewing history may be performed in any of the user interface 120, STB/router 110, or remote server 140. Additionally, the viewing history may be stored in a database of the user interface 120, STB/router 110, or remote server 140.
The user interface 120 in the embodiment of
In the example depicted in
In one embodiment as shown in
As indicated,
Furthermore, the head mounted user interface 120 can be one of many alternatives that embed or allow the user to see a private screen through specialty lenses and may be a part of a head-mounted display (HMD), a headset, a harness, a helmet for augmented reality displays, or other wearable and non-wearable arrangements as can be appreciated by those skilled in the art. In the alternative, none of the components may be connected physically or a subset of them may be physically connected selectively as can be appreciated by those skilled in the art.
Referring back to the embodiment of
Once logged into the system, the user interface is anchored to the content display device 130 on which the content will be viewed as indicated in S15. Anchoring of the user interface to the display allows the user interface to determine where the display is positioned with respect to the AR device. Anchoring establishes a reference location of the display screen within the field of view of the AR user wearing a user interface 120. The initial position of the user interface with respect to the display upon anchoring will be termed the “anchoring home position” for purposes of this description. When determining the position of the display and where on the lenses of the user interface the additional information needs to be projected so as not to obscure the displayed content, movement of the user wearing the user interface from the anchoring home position may also be taken into account. Thus, when the user interface is being worn by an AR user and the user moves in some fashion such as by walking to another seat, turning their head or placing their head at an angle such as when laying down to view the content as opposed to sitting upright, the user interface is able to account for this movement and adjust the position at which additional information is projected on the lenses. This ensures the additional information will continue to be projected in a proper position so as not to obscure the displayed content. By anchoring the user interface to the display screen of the content display device, the user interface is able to calculate a proper position for projection of the additional information on the lenses thereof to not obscure the content being displayed.
Any of numerous methods may be used to anchor the user interface to the display. Some of these methods require a camera on the user interface. An exemplary method involves pattern recognition by the user interface. As the display screen is normally of a standard shape and generally affixed to a surface and thus stationary, the user interface may be moved in different directions and positioned at different angles while a camera connected thereto scans the area for the screen on the content display device. Upon recognizing the display screen by the camera, the user interface will be able to anchor to the position of the display screen. Another method of anchoring the user interface to the display screen includes the transmission of watermarks with the content being displayed. The watermarks may be positioned at corners of the content when being displayed to identify the edges of the displayable area of the content. A sensor on the user interface is able to detect the watermarks to determine the boundaries of the displayed content and thus anchor the user interface to the display screen. A further method of anchoring the user interface to the display is through an Infrared (IR) communication between the user interface and display similar to a remote control operation. Through the IR communication, the content display device can relay positioning information to the user interface or allow the user interface to detect the position of the display. Anchoring of the user interface to the display screen allows the user interface to determine the position of the content display device with respect to the position and orientation of user interface 120, such as the AR glasses 150, at any time while compensating for any movement of the user interface 120. The user interface 120 may also be able to sense any movement thereof and automatically adjust the projection of additional information upon sensing of the movement. This is possible due to the knowledge of the location of the content display device 130 or content being displayed with respect to the user interface 120.
Once logged into the system, the user's profile and viewing history are accessed as stated in S20. Alternatively, accessing of the user profile and viewing history may occur prior to anchoring the user interface to the content display device. The user profile includes information such as user preferences, user background such as country of origin, language, and other information. In S30, it is determined whether the feature for obtaining additional information during content viewing is enabled. If this feature is not enabled, the method proceeds to S40 and terminates. If the feature is enabled, the system checks for (listens for metadata, detects metadata) and extracts metadata embedded within the content being viewed and compares the metadata to data within the user profile and viewing history in S50. In this manner, the metadata may be considered a trigger to display additional information to the viewer.
The feature that is tested at step S30 may include one or more of (a) displaying additional information as subtitles that is not in the user's viewing history, (b) overlaying an object onto the content being displayed on the content display device, and (c) displaying background image information the surrounds the content display device in the view of the AR user. Any or all of these features may be present simultaneously. Each is discussed below.
From step S50, the system checks if display of the content is finished in S60. If it is determined that display of the content has finished, the method proceeds to S40 and terminates. If display of the content has not finished, in S70, the system continues to check for metadata that can trigger additional information to be displayed. If no metadata is detected, the system returns to S50 and continues to check for metadata. If metadata is detected, the metadata is compared with the user profile and viewing history to determine if the metadata indicates information relevant to the viewer is available in S80. Metadata that applies to the user can trigger additional information to be displayed. Such trigger metadata may apply if the metadata is related to other contents (metadata includes a list of content identifiers) and at least one of the related contents is not in the viewing history of the user. Other metadata may be determined to be a trigger if it specifies a demographic or cultural background that is different from the user's profile information. The metadata embedded in the content may include content ID's related to the content, e.g. information related to the content (season and episode number, introduction of characters, etc.) and information related to demographic and/or cultural background of the content. In order to determine if additional information should be provided to the viewer, it is determined if content ID's included within the metadata are absent from the user's viewing history or user profile indicating the viewer did not view content relevant to or necessary for understanding of the content being viewed.
Additionally, if the demographic and/or cultural background of the content is different from that in the user profile, additional information may be required to understand certain actions or language being used in the content being viewed. If it is determined additional information is not necessary based on the comparison of the metadata and viewing history or user profiles, the system returns to S50 and continues to check for additional metadata. If it is determined additional information is necessary based on the comparison of the metadata and viewing history or user profile, the system looks for additional information within the metadata or contacts a database in the remote server for additional information in S90.
The additional information is then provided to the user interface for projection thereon and viewing by the user in S100. Additional information related to content may include information on a story line that began or in which a significant plot twist occurred in an episode not within the viewing history and therefore determined to be missed by the viewer. Additional information related to demographic and/or cultural differences may include explanations of actions such as a handshake, which may be an unfamiliar gesture to someone of the viewer's background or a phrase or language that may be determined to be unfamiliar to someone from the geographic location of the viewer.
Prior to projection of the additional information, the user interface detects its current position and the current viewing tilt/angle with respect to the position of the display in S110. Determination of the current position and tilt/angle of the user interface with respect to the display is performed to provide a proper perspective mapping for projection of the additional information on the lenses of the user interface so as not to obscure the displayed content. While the display is stationary, the user interface is mobile and can be moved when, for example, a user changes their seat, moves their head to a different position or angles their head with respect to the display. Thus, in order for information to be properly mapped with respect to the content, the displacement of the user interface from the anchoring home position at which the display was anchored to the user interface may be calculated. The position of projection for the additional information is adjusted in S120. In S130 the additional information is projected onto the lenses of the user interface so as not to obscure the content being displayed. For example, if the user interface was anchored to the display with a first positional relationship, upon determining additional information should be projected on the lenses of the user interface, the user interface may accommodate for any change in the position of the user interface to project the information at the proper positon on the lenses of the user interface. If the user turns their head 20° to the right, projection of the additional information on the lenses of the user interface should be projected at 20° to the left (or in an opposite direction from the direction of movement of the user interface) from the anchoring home positon.
An additional feature may be detected at Step S30. At step S30, if it is determined that the feature for providing replacement images within the content displayed on a content display device is enabled, then step S50 is entered. At S50, the system checks for and extracts metadata embedded within the content being viewed and compares the metadata to data within the user profile and viewing history in S50. Based on the sensitivities of the user profile and viewing history, it is determined if certain items within the content may be inappropriate or objectionable to a user. In this manner, the metadata may be considered a trigger to obtain and project a replacement image to overlay the item within the content determined to be inappropriate or objectionable. This feature may be implemented in either an AR system, or one without AR capabilities. Steps S60-S130 can then apply as needed using the feature of providing replacement images within the content displayed on a content display device.
In an additional feature detected at step S30, the AR system can display images on a background of the room where the content display device is displaying the content. Thus, the room environment becomes an enhanced background environment surrounding the displayed content. In this context, background images are those images that are displayed to the user of an AR system that appear overlaid upon the room that surrounds the content display device. When content is received, metadata embedded in the content may be extracted and compared with or used in conjunction with the user profile to determine if a particular background image would be appropriate based on the content and preferences of the AR user. Additionally, the metadata may be used to search for background images related to the content or that include background images that are appropriate based on the content being displayed. The background images are generally related to the setting of the scene within the content being displayed. The background images may also be based on the user profile and sensitivities identified by the user profile indicating certain preferences of the AR user and used to enhance the viewing experience associated with the displayed content. Additionally, the background images may be personalized for each AR user using and logged in to a user interface, i.e. the background images seen by each AR user through a personal user interface may be different based on each AR user's particular user profile and/or viewing history. Additionally, the background images for each AR user may be coordinated with the scene it surrounds to maintain conformity and not distract from viewing of the content. The background image may include but is not limited to any of static images, video or 3D rendered objects. The background image is preferably 3D rendered objects when the content being surrounded and enhanced is a 3D video or image.
Thus, if at step S30, the feature of adding an immersive background around the content display device in an AR view is enabled, the system checks for and extracts metadata embedded within the content being viewed in S50. The metadata within the content includes at least one of a background image to be projected by the user interface or instructions for obtaining a background image. Metadata identifying the background image may be compared with the user profile and/or viewing history, to determine if the background image may be inappropriate or objectionable to an AR user. In this manner, the metadata may be considered a trigger to obtain and project a suitable background image to be projected by the user interface in a line of sight to surround the content being displayed. Steps S60-S130 can then apply as needed using the feature of providing an immersive background in an AR environment that surrounds the content display device.
Concerning the user profile information, the profile and viewing history for the user may be stored within the user interface 120 and accessible by the STB/router 110. The content is provided to the STB/router 110 from a content provider via network 115 and then to the display 130 for viewing. To check for feature (a) above, the STB/router 110 extracts the metadata embedded in the content and compares content identifiers (IDs) within the metadata with data in the viewing history database. If the content IDs are not contained within the viewing history, the STB/router 110 determines the viewer may have missed certain information related to the story or characters within the story and contacts the remote database for additional information related to the content being displayed. Upon receipt of the additional information, the STB/router 110 retrieves information related to the anchoring home position of the user interface with respect to the content display device and a current position of the user interface and renders the additional information image to the user interface at a position compensating for movement of the user interface from the anchoring home position as shown in 450.
To check for feature (b) above, the STB/router 110 accesses the user profile database 170 and viewing history from a database 180 contained therein. Alternatively, the user profile and viewing history for the user may be stored within the user interface 120 and accessible by the STB/router 110. The content is provided to the STB/router 110 from a content provider via network 115 and then to the content display device 130 for display as discussed in 430. The STB/router 110 extracts metadata embedded in the content and compares content identifiers (IDs) or other information within the metadata with data in the user profile and/or viewing history database as in 450. Based on this comparison, the STB/router 110 determines whether certain items or information within the content being viewed may be objectionable to or inappropriate for the AR user and contacts the remote database within server 140 for replacement images or information to be projected on the user interface so as to overlay or obstruct the objectionable or inappropriate images within the content being displayed. Alternatively, the user profile may include content ID's indicative of images, items or information that the user finds objectionable. If the metadata received with the content includes the content ID's included in the user profile, the STB/router 110 may determine certain items or information within the content being viewed may be objectionable to or inappropriate for the AR user and contacts the remote database for replacement images or information to be projected on the user interface. Upon receipt of the replacement images or information, the STB/router 110 provides the information to the user interface 120 for display.
To check for feature (c) above, the STB/router 110 accesses the user profile database 170 and viewing history from a database 180 contained therein. Alternatively, the user profile and viewing history for the AR user may be stored within the user interface 120 and accessible by the STB/router 110. The content is provided to the STB/router 110 from a content provider via network 115 and then to the content display device 130 for display as in 430. The STB/router 110 extracts metadata embedded in the content and compares content identifiers (IDs) or other information within the metadata with data in the user profile and/or viewing history databases as in 440. Based on this comparison, the STB/router 110 selects an appropriate background image that is not objectionable to or inappropriate for the AR user based on a comparison with user sensitivities identified in the user profile or viewing history and contacts the remote database within server 140 for the background image to be projected on the user interface so as to surround the content being displayed as in 450. Alternatively, the user profile may include content ID's indicative of images, items or information that the user finds objectionable. If the metadata received with the content includes the content ID's included in the user profile, the STB/router 110 may determine certain background images may be objectionable to or inappropriate for the AR user and contacts the remote database for an alternate background image to be projected on the user interface. Upon receipt of the background image, the STB/router 110 retrieves information related to the anchoring home position of the user interface with respect to the content display device and a current position of the user interface and renders the background image to the user interface at a position compensating for movement of the user interface from the anchoring home position as in 450.
Upon activation, the user interface 120 will anchor itself to the content display device 130 on which the content will be viewed. Anchoring of the user interface 120 to the content display device 130 allows the user interface 120 to determine where the content display device 130 is positioned with respect to the user interface 120. As the user interface 120 is mobile and can be moved to different places or rotated and positioned at different angles, its relationship to the stationary content display device 130 may change. When determining the position of the user interface 120 with respect to the content display device 130 and where on the user interface 120 additional information need to be projected to not obscure the content being displayed, movement of the AR user wearing the user interface 120 and the position of the content display device 130 with respect to the user interface 120 must also be taken into account. Thus, when the user interface 120 is being worn by an AR user and the AR user moves the user interface 120 from the anchoring home position in some fashion such as by walking to another seat, turning their head or placing their head at an angle such as when laying down to view the content as opposed to sitting upright, the user interface 120 is able to account for this movement and adjust the position at which additional information is projected. Alternatively, the user interface 120 can provide the STB/router 110 with information related to the anchoring home position and the current position of the user interface and the STB/router 110 can calculate the correct position for projection of the additional information by the user interface 120. This information can be provided to the user interface 120 with the additional information to be projected thereby. This ensures the additional information will continue to be projected in a proper position on a lens of the user interface to not obscure the content being viewed. Any of numerous methods, as previously discussed, may be used to anchor the user interface 120 to the content display device 130.
The database 190 can also store data representative of the additional information and replacement images to be provided to a user related to the content upon determining that items within the content may be inappropriate or objectionable to the AR user. The additional information or replacement images may provide a more enjoyable experience for the user or prevent the user from viewing content that is inappropriate based on, for example, the user's age. The main control processor 160 extracts the embedded metadata within the content and compares it with data from the user profile and viewing history to determine if items within the content may be objectionable to or inappropriate for the user. If it is determined that information identified in the metadata may be objectionable to or inappropriate for the user, the main control processor 160 accesses additional information and replacement images within the overlay content database 190 that can replace or overlay the identified items to provide a more enjoyable experience for the AR user.
The database 190 can also store data representative of background images to be provided to the user interface related to the content upon determining a background image should be provided for viewing by the AR user. The background images may provide a more enjoyable experience for the user by creating an environment in which the AR user may feel immersed in the scene of the displayed content. The main control processor 160 extracts the embedded metadata within the content and may compare it with data from the user profile and viewing history to determine if items within a background image may be objectionable to or inappropriate for the AR user. If it is determined that information identified in the metadata may be objectionable to or inappropriate for the AR user, the main control processor 160 accesses background images within the background image database 190 that are provided to the user interface for projection on the lenses thereof. If it is determined that information identified in the metadata may be objectionable to or inappropriate for the AR user, the main control processor 160 accesses background images within the background image database 190 that are provided to the user interface for projection on the lenses thereof.
The user interface 120 is serviced via a user interface transmitter/receiver 612 in
The augmented reality system 100 of
The implementations described herein may be implemented in, for example, a method or process, an apparatus, or a combination of hardware and software. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms. For example, implementation can be accomplished via a hardware apparatus, hardware and software apparatus. An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to any processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device.
Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions may be stored on a processor or computer-readable media such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette (“CD” or “DVD”), a random access memory (“RAM”), a read-only memory (“ROM”) or any other magnetic, optical, or solid state media. The instructions may form an application program tangibly embodied on a computer-readable medium such as any of the media listed above or known to those of skill in the art. The instructions thus stored are useful to execute elements of hardware and software to perform the steps of the method described herein.
This application claims the benefit, under 35 U.S.C. § 365 of International Application PCT/US2016/066079 filed Dec. 12, 2016, which was published in accordance with PCT Article 21(3) on Jun. 22, 2017 in English and which claims the benefit of U.S. provisional patent application Nos. 62/268,577, 62/268,582 and 62/268,587 all filed Dec. 17, 2015. The provisional and PCT applications are expressly incorporated by reference herein in their entirety for all purposes.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2016/066079 | 12/12/2016 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2017/106807 | 6/22/2017 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 8606645 | Applefeld | Dec 2013 | B1 |
| 8726172 | Vignocchi et al. | May 2014 | B1 |
| 8884984 | Flaks et al. | Nov 2014 | B2 |
| 8913085 | Anderson et al. | Dec 2014 | B2 |
| 20090013052 | Robarts et al. | Jan 2009 | A1 |
| 20090106671 | Olson et al. | Apr 2009 | A1 |
| 20100328460 | Merkel et al. | Dec 2010 | A1 |
| 20110161999 | Klappert et al. | Jun 2011 | A1 |
| 20120218297 | Ur | Aug 2012 | A1 |
| 20140111687 | Difrancesco et al. | Apr 2014 | A1 |
| 20140282192 | Grossman et al. | Sep 2014 | A1 |
| 20140344661 | Sipe et al. | Nov 2014 | A1 |
| 20150014417 | Finlow-Bates et al. | Jan 2015 | A1 |
| 20150187136 | Grimaud | Jul 2015 | A1 |
| 20150301596 | Qian | Oct 2015 | A1 |
| 20160109940 | Lyren | Apr 2016 | A1 |
| 20160155231 | Raghoebardajal et al. | Jun 2016 | A1 |
| 20160189268 | Gandhi | Jun 2016 | A1 |
| 20160335645 | Holz | Nov 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 201336146 | Oct 2009 | CN |
| 101933027 | Dec 2010 | CN |
| 102981616 | Mar 2013 | CN |
| 105164727 | Dec 2015 | CN |
| WO 2013157898 | Oct 2013 | WO |
| Entry |
|---|
| Scherp, Ansgar, “Semantics Support for Personalized Multimedia Content”, Proceedings of the IASTED International Conference, Internet & Multimedia Systems & Applications and Visual Communications, Mar. 17-19, 2008, Innsbruck, Austria. |
| Herling, Jan et al., “Advanced Self-contained Object Removal for Realizing Real-time Diminished Reality in Unconstrained Environments”, Ilmenau University of Technology. |
| Cascio Jamais, “Filtering Reality”, Dec. 18, 2012, pp. 1-4, www.theatlantic.com/magazine/archive/2009/11/filtering-reality/3077131, retrieved on Aug. 4, 2016. |
| Communication pursuant to Rule 164(2)(b) and Article 94(3) EPC for EP 16 836 134.3-1213 dated Jun. 9, 2020 (EP Counterpart Application). |
| CN201336146Y TR EN machine full Aug. 31, 2020. |
| Number | Date | Country | |
|---|---|---|---|
| 20180367835 A1 | Dec 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62268577 | Dec 2015 | US | |
| 62268582 | Dec 2015 | US | |
| 62268587 | Dec 2015 | US |
