Patent Application
20120324492
Obtaining real-time feedback for video programming may pose various challenges. For example, some past approaches utilize sample groups to provide feedback to broadcast television content. Such feedback may then be used to guide future programming decisions. However, the demographics of such sample groups may rely upon the goals of the entity that is gathering the feedback, and thus may not be helpful in making programming decisions regarding many potential viewers outside of the target demographic profile. Further, such feedback is generally used after presentation of the program for future programming development, and thus does not affect the programming currently being watched as the feedback is gathered.
Various embodiments are disclosed herein that relate to selecting video content items based upon data from video viewing environment sensors. For example, one embodiment provides a method comprising determining identities for each viewer in a video viewing environment from data received from video viewing environment sensors, obtaining a video item based on the determined identity or identities, and sending the video item to a display device for display.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
Broadcast television has long been a one-way channel, pushing out programming and advertisement without providing a real-time feedback loop for viewer feedback, making content personalization difficult. Thus, the disclosed embodiments relate to entertainment systems including viewing environment sensors, such as image sensors, depth sensors, acoustic sensors, and potentially other sensors such as biometric sensors, to assist in determining viewer preferences for use in helping viewers to discover content. Such sensors may allow systems to identify individuals, detect and understand human emotional expressions, and provide real-time feedback while a viewer is watching video. Based on such feedback, an entertainment system may determine a measure of a viewer's enjoyment of the video, and provide real-time responses to the perceived viewer emotional responses, for example, to recommend similar content, record similar content playing concurrently on other channels, and/or change the content being displayed.
Detection of human emotional expressions may further be useful for learning viewer preferences and personalizing content when an entertainment system is shared by several viewers. For example, one viewer may receive sports recommendations while another may receive drama recommendations. Further, content may be selected and/or customized to match the combined interests of viewers using the display. For example, content may be customized to meet the interest of family members in a room by finding content at the intersection of viewing interests for each of those members.
Further, detecting viewer emotional feedback as the viewer views content may also allow content to be updated in real-time, for example, by condensing long movies into shorter time periods, by cutting out uninteresting scenes, by providing a different edited version of the content item, and/or by targeting advertisements to viewers more effectively.
Media computing device 104 may process data received from sensor system 106 to generate temporal relationships between video items viewed by a viewer and each viewer's emotional response to the video item. As explained in more detail below, such relationships may be recorded as a viewer's emotional response profile for a particular video item and included in a viewing interest profile cataloging the viewer's video interests. This may allow the viewing interest profiles for a plurality of viewers in a viewing party to be retrieved and used to select items of potentially greater interest for viewing by the current audience.
As a more specific example, image data received from viewing environment sensor system 106 may capture conscious displays of human emotional behavior of a viewer, such as an image of a viewer 160 cringing or covering his face. In response, the viewer's emotional response profile for that video item may indicate that the viewer was scared at that time during the item. The image data may also include subconscious displays of human emotional states. In such a scenario, image data may show that a user was looking away from the display at a particular time during a video item. In response, the viewer's emotional response profile for that video item may indicate that she was bored or distracted at that time. Eye-tracking, facial posture characterization and other suitable techniques may also be employed to gauge a viewer's degree of emotional stimulation and engagement with video item 150.
In some embodiments, an image sensor may collect light within a spectral region that is diagnostic of human physiological conditions. For example, infrared light may be used to approximate blood oxygen levels and/or heart rate levels within the body. In turn, such levels may be used to estimate the person's emotional stimulation.
Further, in some embodiments, sensors that reside in other devices than viewing environment sensor system 106 may be used to provide input to media computing device 104. For example, in some embodiments, an accelerometer included in a mobile computing device (e.g., mobile phones and laptop and tablet computers) held by a viewer 160 within video viewing environment 100 may detect gesture-based emotional expressions for that viewer.
Next, in
Next, in
The brief scenario described above relates to the selection of video items 150 based on the respective identities and emotional profiles of viewers 160. Further, in some embodiments, real-time emotional response data may be used to update a video content item currently being viewed. For example, based upon real-time emotional responses to a video item, a version of the item being displayed (e.g., content-edited vs. unedited) may be changed. As a more specific example, if media computer 104 detects that a viewer 160 is embarrassed by strong language in video item 150, media computing device 104 may obtain an updated version having strong language edited out. In another example, if video viewing environment sensor system 106 detects viewer 160 asking viewer 162 what a character in video item 150 just said, media computing device 104 may interpret the question as a request that a related portion of video item 150 be replayed, and replay that portion in response to that request.
As mentioned above, in some embodiments, sensor data from sensors on a viewer's mobile device may be provided to the media computing device. Further, supplemental content related to a video item being watched on a primary viewing environment display may be provided to the viewer's mobile device. Suitable mobile computing devices include, but are not limited to, mobile phones and portable personal computing devices (e.g., laptops, tablet, and other such computing devices). Thus, in some embodiments, method 400 may include, at 402, sending a request from a mobile computing device belonging to a viewer in the video viewing environment to the media computing device to register the mobile computing device with the media computing device, and at 404, registering the mobile computing device. In some of such embodiments, the mobile computing device may be registered with a viewer's personal profile.
At 406, method 400 includes collecting sensor data from video viewing environment sensor system 106 and potentially from mobile device 140, and at 408, sending the sensor data to the media computing device, which receives the input of sensor data. Any suitable sensor data may be collected, including but not limited to image data, depth data, acoustic data, and/or biometric data.
At 410, method 400 includes determining an identity of each of the plurality of viewers in the video viewing environment from the input of sensor data. In some embodiments, a viewer's identity may be established from a comparison of image data collected by the sensor data with image data stored in a viewer's personal profile. For example, a facial similarity comparison between a face included in image data collected from the video viewing environment and an image stored in a viewer's profile may be used to establish the identity of that viewer. In this example, the viewer may not use a password to log in. Instead, the media computing device may detect the viewer, check for the existence of a profile for the viewer, and, if a profile exists, confirm the identity of the viewer. A viewers' identity also may be determined from acoustic data, and/or any other suitable data.
At 412, method 400 includes obtaining a video item for display based upon the identities of the plurality of viewers in the video viewing environment. It will be appreciated that aspects of 412 may occur at the media computing device and/or at a server computing device at various embodiments. Thus, aspects that may occur on either device are shown in
Obtaining the video item may comprise, at 414, correlating viewing interest profiles stored for each of the plurality of viewers with one another and with information about available video items, and then, at 416, selecting the video item based on the correlation. For example, in some embodiments, the video item may be selected based on an intersection of the viewing interest profiles for the viewers in the video viewing environment, as described in more detail below.
A viewing interest profile catalogs a viewer's likes and dislikes for video media, as judged from the viewer's emotional responses to past media experiences. Viewing interest profiles are generated from a plurality of emotional response profiles, each emotional response profile temporally correlating the viewer's emotional response to a video item previously viewed by the viewer. Put another way, the viewer's emotional response profile for a particular video item organizes that viewer's emotional expressions and behavioral displays as a function of a time position within that video item. As the viewer watches more video items, the viewer's viewing interest profile may be altered to reflect changing tastes and interests of the viewer as expressed in the viewer's emotional responses to recently viewed video items.
In the example shown in
In some embodiments, semantic mining module 502 may be configured to distinguish between the viewer's emotional response to a video item and the viewer's general temper. For example, in some embodiments, semantic mining module 502 may ignore, or may report that the viewer is distracted during, those human affective displays detected when the viewer's attention is not focused on the display device. Thus, as an example scenario, if the viewer is visibly annoyed because of a loud noise originating external to the video viewing environment, semantic mining module 502 may be configured not to ascribe the detected annoyance with the video item, and may not record the annoyance at that temporal position within the viewer's emotional response profile for the video item. In embodiments in which an image sensor is included as a video viewing environment sensor, suitable eye tracking and/or face position tracking techniques may be employed (potentially in combination with a depth map of the video viewing environment) to determine a degree to which the viewer's attention is focused on the display device and/or the video item.
A viewer's emotional response profile 504 for a video item may be analyzed to determine the types of scenes/objects/occurrences that evoked positive and negative responses in the viewer. For example, in the example shown in
In some embodiments, additional filters may be applied (e.g., age-based filters that take into account the ages of members of the present viewers, etc.) to further filter content for presentation. For example, in one scenario, a video program may switch from a version that may include content not suitable for viewers of all ages to an all-ages version in response to a child (or another person with a viewing interest profile so-configured) entering the video viewing environment. In this scenario, the transition may be managed in an apparently seamless transition, so that a gap in programming does not result. In another scenario, a suitable display (for example, a 3D display paired with 3D glasses, or an optical wedge-based directional video display in which collimated light is sequentially directed at different viewers in synchronization with the production of different images via a spatial light modulator) may be used to deliver viewer-specific versions of a video item according to individual viewing preferences. Thus, a child may view an all-ages version of the video item and be presented with advertisements suitable for child audiences while an adult concurrently views a more mature version of the video item, along with advertisements geared toward an adult demographic group.
Turning back to
As a more specific example, in some embodiments, the correlation may be related to a video item genre that the viewers would like to watch. For example, if the viewers would like to watch a scary movie, the viewing interest profiles may be correlated based on past video item scenes that the viewers have experienced as being scary. Additionally or alternatively, in some embodiments, the correlation may be based on other suitable factors such as video item type (e.g., cartoon vs. live action, full-length movie vs. video clip, etc.). Once the video item has been selected, method 400 includes, at 418, sending the video item for display.
As explained above, in some embodiments, similar methods of selecting video content may be used to update a video item being viewed by a viewing party when a viewer leaves or joins the viewing party. Turning to
At 424, method 400 includes determining from the additional sensor data a change in constituency of the plurality of viewers in the viewing environment. As a more specific example, the media computing device determines whether a new viewer has entered the viewing party or whether an existing viewer has left the viewing party, so that the video item being displayed may be updated to be comparatively more desirable to the changed viewing party relative to the original viewing party.
In some embodiments, a viewer may be determined to have exited the viewing party without physically leaving the video viewing environment. For example, if it is determined that a particular viewer is not paying attention to the video item, then the viewer may be considered to have constructively left the viewing party. Thus, in one scenario, a viewer who intermittently pays attention to the video item (e.g., directs her attention to the display for less than a preselected time before diverting her gaze again) may be present in the video viewing environment without having her viewing interest profile correlated. However, the media computing device and/or the semantic mining module may note those portions of the video item that grabbed her attention, and may update her viewing interest profile accordingly.
At 426, method 400 includes obtaining updated video item based on the identities of the plurality of viewers after the change in constituency is determined. As explained above, aspects of 426 may be performed at the media computing device and/or at the server computing device. Thus, in embodiments where aspects of 426 are performed at a server computing device, 426 includes, at 427, sending determined identities for the plurality of viewers to a server, the identities reflecting the change in constituency, and, at 433, receiving the updated video item from the server. In embodiments in which aspects of 426 are performed at a media computing device, processes 427 and 433 may be omitted.
In some embodiments, 426 may include, at 428, re-correlating the viewing interest profiles for the plurality of viewers, and, at 430, selecting the updated video item based on the re-correlation of the viewing interest profiles after the change in constituency. In such embodiments, the re-correlated viewing interest profiles may be used to select items that may appeal to the combined viewing interests of the new viewing party, as explained above. Once the video item has been selected, method 400 includes, at 434, sending the video item for display.
In some embodiments, the selected updated video item maybe a different version of the video item than that was being presented when the viewing party constituency changed. For example, the updated video item may be a version edited to display appropriate subtitles according to a language suitability of a viewer joining the viewing party. In another example, the updated video item may be a version edited to omit strong language and/or violent scenes according to a content suitability (for example, if a younger viewer has joined the viewing party). Thus, in some embodiments, 426 may include, at 432, updating the video item according to an audience suitability rating associated with the video item and the identities of the plurality of viewers. Such suitability ratings may be configured by individual viewers and/or by content creators, which may provide a way of tuning content selection to the viewer.
In some embodiments, the selected updated video item may be a different video item from the video item being presented when the viewing party constituency changed. In such embodiments, the viewers may be presented with an option of approving the updated video item for viewing and/or may be presented with a plurality of updated video items from which to choose, the plurality of updated video items being selected based on a re-correlation of viewing interest profiles and/or audience suitability ratings.
It will be appreciated that changes and updates to the video item being obtained for display may be triggered by other suitable events and are not limited to being triggered by changes in viewing party constituency. In some embodiments, updated video items may be selected based a change in the emotional status of a viewer in response to the video item being viewed. For example, if a video item is perceived by the viewers as being unengaging, a different video item may be selected. Thus, turning to
At 440, method 400 includes determining a change in a particular viewer's emotional response to the video item using the sensor data. For example, in some embodiments where the video viewing environment sensor includes an image sensor, determining a change in a particular viewer's emotional response to the video item may be based on image data of the particular viewer's emotional response. Likewise, changes in emotional response also may be detected via sound data, biometric data, etc. Additionally or alternatively, in some embodiments, a change in the particular viewer's emotional response may include receiving emotional response data from a sensor included in the viewer's mobile computing device.
At 442, method 400 includes obtaining an updated video item for display based on a real-time emotional response of the particular viewer. As explained above, aspects of 442 may be performed at the media computing device and/or at the server computing device. Thus, in embodiments where aspects of 442 are performed at a server computing device, 442 includes, at 443, sending determined identities for the plurality of viewers to a server, the identities reflecting the change in constituency, and, at 452, receiving the updated video item from the server. In embodiments in which aspects of 442 are performed at a media computing device, processes 443 and 452 may be omitted.
In some embodiments, 442 may include, at 444, updating the particular viewer's viewing interest profile with the particular viewer's emotional response to the video item. Updating the viewer's viewing interest profile may keep that viewer's viewing interest profile current, reflecting changes in the viewer's viewing interests over time and in different viewing situations. In turn, the updated viewing interest profile may be used to select potentially more desirable video items for that viewer in the future.
In some embodiments, 442 may include, at 446, re-correlating the viewing interest profiles for the plurality of the viewers after updating the particular viewer's viewing interest profile and/or after detecting the change in the particular viewer's emotional response. Thus, if the viewer had an adverse emotional reaction toward the video item, re-correlation of the viewing interest profiles may lead to an update of the video item being display. For example, a different video item or a different version of the present video item may be selected and obtained for display.
In some embodiments, 442 may include, at 448, detecting an input of an implicit request for a replay of a portion of the video item, and, in response, selecting that portion of the video item to be replayed. For example, it may be determined that the viewer's emotional response included affect displays corresponding to confusion. Such responses may be deemed an implicit request to replay a portion of the video item (such as a portion being presented when the response was detected), and the user may be presented with the option of viewing the scene again. Additionally or alternatively, detection of such implicit requests may be contextually-based. For example, a detected emotional response may vary from a predicted emotional response by more than a preselected tolerance (as predicted by aggregated emotional response profiles for the video item from a sample audience, for example), suggesting that the viewer did not understand the content of the video item. In such cases, a related portion of the video item may be selected to be replayed.
It will be understood that explicit requests for replay may be handled similarly. Explicit requests may include viewer-issued commands for replay (e.g., “play that back!”) as well as viewer-issued comments expressing a desire that a portion be replayed (e.g., “what did she say?”). Thus, in some embodiments, 450 may include, at 444, detecting an input of an explicit request for a replay of a portion of the video item, and, in response, selecting that portion of the video item to be replayed.
Turning to
In some embodiments, as indicated at 458, updated video items selected based on a particular viewer's viewing interest profile may be presented to that viewer on the mobile computing device for the particular viewer. This may provide personalized delivery of finely-tuned content for a viewer without disrupting the viewing party's entertainment experience. It may also provide an approach for keeping viewers with marginal interest levels engaged with the video item. For example, a viewer may watch a movie with a viewing party on the primary display device while viewing subtitles for the movie on the viewer's personal mobile computing device and/or while listening to a different audio track for the movie via headphones connected to the mobile computing device. In another example, one viewer may be presented with supplemental content related to a favorite actor appearing in the video item via his mobile computing device as selected based on his emotional response to the actor. Concurrently, a different viewer may be presented with supplemental content related to a filming location for the video item on her mobile display device, the content being selected based on her emotional response to the a particular scene in the video item. In this way, the viewing party may continue to enjoy, as a group, a video item selected based on correlation of their viewing interest profiles, but may also receive supplemental content selected to help them, as individuals, get more enjoyment out of the experience.
As introduced above, in some embodiments, the methods and processes described in this disclosure may be tied to a computing system including one or more computers. In particular, the methods and processes described herein may be implemented as a computer application, computer service, computer API, computer library, and/or other computer program product.
The computing system includes a logic subsystem (for example, logic subsystem 116 of mobile computing device 104 of
The logic subsystem may include one or more physical devices configured to execute one or more instructions. For example, the logic subsystem may be configured to execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result.
The logic subsystem may include one or more processors that are configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic subsystem may be single core or multicore, and the programs executed thereon may be configured for parallel or distributed processing. The logic subsystem may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and/or configured for coordinated processing. One or more aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.
The data-holding subsystem may include one or more physical, non-transitory, devices configured to hold data and/or instructions executable by the logic subsystem to implement the herein described methods and processes. When such methods and processes are implemented, the state of the data-holding subsystem may be transformed (e.g., to hold different data).
The data-holding subsystem may include removable media and/or built-in devices. The data-holding subsystem may include optical memory devices (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory devices (e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), among others. The data-holding subsystem may include devices with one or more of the following characteristics: volatile, nonvolatile, dynamic, static, read/write, read-only, random access, sequential access, location addressable, file addressable, and content addressable. In some embodiments, the logic subsystem and the data-holding subsystem may be integrated into one or more common devices, such as an application specific integrated circuit or a system on a chip.
It is to be appreciated that the data-holding subsystem includes one or more physical, non-transitory devices. In contrast, in some embodiments aspects of the instructions described herein may be propagated in a transitory fashion by a pure signal (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for at least a finite duration. Furthermore, data and/or other forms of information pertaining to the present disclosure may be propagated by a pure signal.
The terms “module,” “program,” and “engine” may be used to describe an aspect of the computing system that is implemented to perform one or more particular functions. In some cases, such a module, program, or engine may be instantiated via the logic subsystem executing instructions held by the data-holding subsystem. It is to be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” are meant to encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
It is to be appreciated that a “service”, as used herein, may be an application program executable across multiple user sessions and available to one or more system components, programs, and/or other services. In some implementations, a service may run on a server responsive to a request from a client.
When included, a display subsystem may be used to present a visual representation of data held by the data-holding subsystem. As the herein described methods and processes change the data held by the data-holding subsystem, and thus transform the state of the data-holding subsystem, the state of display subsystem may likewise be transformed to visually represent changes in the underlying data. The display subsystem may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logic subsystem and/or the data-holding subsystem in a shared enclosure, or such display devices may be peripheral display devices.
It is to be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated may be performed in the sequence illustrated, in other sequences, in parallel, or in some cases omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
