Patent Application
20180093174
Video gaming has become a popular spectator activity. Every month, millions of minutes of gameplay video are uploaded to streaming video services, which store the gameplay video and make it available for playback. Some services even offer “live” broadcasting of gameplay video in real time or near real time as the gameplay happens.
In a typical configuration, gameplay is recorded at a gaming console, smartphone or personal computer. A video encoder encodes frames of the gameplay, depicting animation rendered to the display and/or other graphical details. An audio encoder encodes sound effects, voice effects and/or other audio that accompanies the gameplay video. The encoded data is uploaded to a streaming video service, which makes the gameplay video (including associated audio) available for streaming.
In some configurations, the person playing a video game at a gaming console can provide audio commentary to narrate the gameplay as it happens. In some existing approaches, the player can also share a video of himself/herself playing the video game. This allows the player to add gestures, facial expressions, etc. to the gameplay video that is recorded at the gaming console. In these existing approaches, however, the camera video of the player appears at a fixed location in the gameplay video that is recorded at the gaming console. Depending on the game or stage of the game, the camera video overlay of the player may obscure an important part of the gameplay area in the gameplay video that is streamed to viewers. Also, in some existing approaches, the camera video overlay of the player may obscure part of the gameplay area that the player sees, which can be highly disruptive to the player.
In summary, the detailed description presents ways for a gaming console or companion device associated with the gaming console to allow a user to specify placement (that is, position) of a camera video overlay on gameplay video. This can permit the user to choose the portion of the gameplay video that is obscured by the camera video overlay, which can improve the quality of the gameplay video that is broadcast. The detailed description also presents approaches for a gaming console to composite a camera video overlay on gameplay video. With such approaches, the gameplay area that is rendered by the gaming console locally for a player is not obscured even when a camera video overlay is composited on gameplay video for broadcasting, which improves playability by avoiding disruptions to the player in the gameplay area.
According to one aspect of the innovations described herein, a gaming console or companion device renders for display a configuration menu that shows options for placement of a camera video overlay on gameplay video. For example, the options for the placement of the camera video overlay include four corner locations on the gameplay video. Alternatively, the options for the placement of the camera video overlay include other and/or additional locations on the gameplay video (e.g., middles of sides, free positioning at arbitrary x, y coordinates on the gameplay video). The configuration menu can also show (a) a preview of the camera video overlay as composited on the gameplay video at the specified placement, (b) an option for enabling or disabling input from a video camera for the camera video overlay, (c) an option for enabling or disabling input from a microphone, (d) options for re-sizing the camera video overlay, (e) options for including a chat stream and/or (f) other controls or options.
The gaming console or companion device receives user input specifying the placement of the camera video overlay on the gameplay video. The gaming console or companion device stores an indication of the specified placement (e.g., locally and/or in cloud-based storage). The gaming console can use the indication of the specified placement when later compositing the camera video overlay on the gameplay video for broadcasting.
The specified placement of the camera video overlay on the gameplay video can be used throughout an entire gameplay broadcast. Alternatively, the placement of the camera video overlay on the gameplay video can be changed during broadcasting. For example, during broadcasting, the gaming console or companion device renders the configuration menu again, receives new user input specifying the placement of the camera video overlay on the gameplay video, and stores an indication of the newly specified placement, which is used when compositing the camera video overlay on the gameplay video for broadcasting. In this way, during broadcasting, the user can change which portion of the gameplay video is obscured by the camera video overlay, depending on the stage of the gameplay or some other factor.
According to a second aspect of the innovations described herein, a gaming console receives a frame of gameplay video (e.g., from a screen capture module that captures gameplay rendered for display) and also receives a frame of camera video overlay (e.g., from a video camera that captures natural video). The gaming console can resize (e.g., down-sample) the frame of camera video overlay before compositing, or the frame of camera video overlay can be provided (e.g., by the video camera) at the appropriate resolution.
The gaming console composites the frame of the camera video overlay on the frame of the gameplay video to produce a composited frame. For example, the gaming console composites the frame of camera video overlay at a placement specified by a user, which can be selected as described above. The gaming console provides the composited frame to a video encoder, which produces encoded video data. The gaming console provides the encoded video data to a buffer for transmission. For example, the encoded video data is streamed over a network to a server for storage and/or broadcasting to one or more viewers. Or, the encoded video data is streamed over a network to another gaming console or a viewing device (such as a companion device for the other gaming console or the originating gaming console) for storage and/or playback. The gaming console can repeat this process on a frame-by-frame basis.
When the gameplay is rendered for display at the gaming console, the gameplay can be rendered without compositing the camera video overlay. Thus, the gameplay is not obstructed by the camera video overlay at the gaming console, even when the camera video overlay is composited on the gameplay video that is broadcast.
The foregoing and other objects, features, and advantages of the invention will become more apparent from the following detailed description, which proceeds with reference to the accompanying figures.
The detailed description presents ways for a gaming console or companion device associated with the gaming console to allow a user to specify the placement (that is, position) of a camera video overlay on gameplay video. For example, a configuration menu can present options for overlaying the camera video at different corners of the gameplay video for broadcasting, allowing the user to specify which corner of a gameplay video is obscured by the camera video overlay in the gameplay broadcast. The detailed description also presents approaches for a gaming console to composite a camera video overlay on gameplay video. With such approaches, the gameplay area that is rendered by the gaming console locally for a player is not obscured even when a camera video overlay is composited on gameplay video for broadcasting.
Although operations described herein are in places described as being performed by a gaming console, in many cases the operations can be performed by another type of computing device (e.g., smartphone, tablet computer, personal computer).
More generally, various alternatives to the examples described herein are possible. For example, certain techniques described with reference to flowchart diagrams can be altered by changing the ordering of stages shown in the flowcharts, by splitting, repeating or omitting certain stages, etc. The various aspects of the disclosed technology can be used in combination or separately. For example, the approaches to compositing a camera video overlay on gameplay video for broadcasting without obscuring local rendering of gameplay can be used without adjustable positioning of the camera video overlay on the gameplay video for broadcasting. Different embodiments use one or more of the described innovations. Some of the innovations described herein address one or more of the problems noted in the background. Typically, a given technique/tool does not solve all such problems.
With reference to
The gaming console (100) optionally includes a media player (140) and video input (142). The media player (140) can play DVDs, Blu-ray disks, other disk media and/or other formats of media. The video input (142) can accept input video in analog or digital form (e.g., from a cable input, HDMI input or other input).
The gaming console (100) includes one or more network interface devices (150). The network interface device(s) (150) enable communication over a network to another computing entity (e.g., server, other gaming console). The network interface device(s) (150) can support wired connections and/or wireless connections, for a wide-area network, local-area network, personal-area network or other network. For example, the network interface device(s) can include one or more Wi-Fi transceivers, an Ethernet port, a cellular transceiver and/or another type of network interface device, along with associated drivers, software, etc. The network interface device(s) (150) convey information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal over network connection(s). A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the network connections can use an electrical, optical, RF, or other carrier.
The storage (160) may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, optical disk media and/or any other media which can be used to store information and which can be accessed within the gaming console (100). The storage (160) stores instructions for the software (130) implementing one or more innovations for positioning and/or compositing a camera video overlay on gameplay video.
A camera input (170) accepts video input in analog or digital form from a video camera, which captures natural video. In particular, the camera input (170) can provide frames of a camera video overlay to be composited on gameplay video. The frames of the camera video overlay can be received in the appropriate size for compositing. Or, the frames of the camera video overlay can be down-sampled after they are received by the gaming console (100). An audio input (not shown) accepts audio input in analog or digital form from a microphone, which captures audio.
The gaming console optionally includes a motion sensor/tracker input (172) for a motion sensor/tracker, which can track the movements of a user and objects around the user. For example, the motion sensor/tracker allows a user (e.g., player of the game) to interact with the gaming console (100) through a natural user interface using gestures and spoken commands. The motion sensor/tracker can incorporate gesture recognition, facial recognition and voice recognition.
A game controller input (180) accepts control signals from one or more game controllers, over a wired connection or wireless connection. The control signals can indicate user inputs from one or more directional pads, buttons, triggers and/or one or more joysticks of a game controller. The control signals can also indicate user inputs from a touchpad or touchscreen, gyroscope, accelerometer, angular rate sensor, magnetometer and/or other control or meter of a game controller.
A video output (190) provides video output to a display device. The video output (190) can be an HDMI output or other type of output. When the video output (190) conveys gameplay to the display device, a screen capture module (not shown) can capture frames of the gameplay as gameplay video. Alternatively, the frames of gameplay video are captured in some other way. An audio output (not shown) provides audio output to one or more speakers.
The gaming console (100) includes a video encoder (112) that can encode gameplay video and/or camera video. For example, the video encoder (112) is provided with special-purpose video encoding hardware as part of a SoC with the processing unit(s) (110). Alternatively, the video encoder is implemented as a software encoder that runs on one of the processing unit(s) (110). Or, the video encoder can be implemented using hardware acceleration with graphics hardware of the gaming console (100). The video encoder (112) can produce video formatted according to the H.264/AVC standard or formatted according to another standard or format (e.g., H.265/HEVC, VPx, SMPTE 421M or another format).
The gaming console (100) can also include an audio encoder (not shown) that encodes audio associated with the gameplay video and/or camera video. The audio encoder can produce encoded audio data in AAC format, HE-AAC format, a Windows Media Audio format, MP3 format, or another format. The gaming console (100) can also include a video decoder (not shown) and audio decoder (not shown).
The gaming console (100) may have additional features. For example, the gaming console (100) includes one or more other input devices and/or one or more other output devices. The other input device(s) may be a touch input device such as a keyboard, mouse, pen, or trackball, a scanning device, or another device that provides input to the gaming console (100). The other output device(s) may be a printer, CD-writer, or another device that provides output from the gaming console (100).
An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the gaming console (100). Typically, operating system software (not shown) provides an operating environment for other software executing in the gaming console (100), and coordinates activities of the components of the gaming console (100).
One or more “companion” devices (not shown) can be associated with the gaming console (100). A companion device runs a companion application that facilitates interaction with the gaming console (100). A companion device is typically a smartphone or tablet, but can instead be a computing device with some other form factor. The companion application can allow the companion device to serve as an additional screen and/or controller for media content or gameplay. Or, the companion application can allow the companion device to serve as a media source or messaging tool for the gaming console (100).
The innovations can be described in the general context of computer-readable media. Computer-readable media are any available tangible media that can be accessed within a computing environment. By way of example, and not limitation, with the gaming console (100), computer-readable media include memory (120), storage (160), and combinations thereof.
The innovations can be described in the general context of computer-executable instructions, such as those included in program modules, being executed in a computing system on a target real or virtual processor. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Computer-executable instructions for program modules may be executed within a local or distributed computing system.
The terms “system” and “device” are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on a type of computing system or computing device. In general, a computing system or computing device can be local or distributed, and can include any combination of special-purpose hardware and/or hardware with software implementing the functionality described herein.
For the sake of presentation, the detailed description uses terms like “determine,” “receive” and “provide” to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer, and should not be confused with acts performed by a human being. The actual computer operations corresponding to these terms vary depending on implementation.
The broadcasting application (210) is adapted to control gameplay broadcasting. In particular, the broadcasting application (210) requests or controls certain operations performed by the screen capture module (230), the video camera (240), the compositing module (250) and the video encoder (260). The broadcasting application (210) can request/control such operations on a frame-by-frame basis. In some example implementations, the gameplay broadcasting can be performed in real time or near real time with smooth frame rates such as 30 frames per second.
The screen capture module (230) is adapted to capture frames of gameplay video that represents the gameplay rendered for display. The broadcasting application (210) requests and receives one of the gameplay video frames (232) recorded by the screen capture module (230). The screen capture module (230) can be part of the platform for a gaming console. The screen capture module (230) can record frames, for example, by reading pixel values from a display buffer after gameplay animations, graphics, etc. have been rendered to the display buffer by a rendering engine. Or, the screen capture module (230) can record frames of the gameplay video in some other way.
The video camera (240) captures frames of natural video. The broadcasting application (210) requests and receives one of the video overlay frames (242) recorded by the video camera (240).
If the video overlay frames (242) do not have the appropriate resolution for compositing, a scaler (not shown) resizes the video overlay frames (242). For example, the scaler scales a 1080p camera video frame to a smaller resolution such as 360×240 or 240×180. In some example implementations, video overlay frames (242) always have the same spatial resolution before they are composited onto gameplay video frames (232). Alternatively, the broadcasting application (210) can adjust (or request that the video camera (240) adjust) the spatial resolution of the video overlay frames (242).
The broadcasting application (210) instructs the compositing module (250) to composite the received frame of camera video overlay on the received frame of gameplay video. The compositing module (250) can be part of the platform for a gaming console, e.g., in a rendering engine. Alternatively, the broadcasting application (210) can include the compositing module. For the compositing operation, the compositing module (250) replaces certain pixel values of the frame of the gameplay video with pixel values of the frame of the camera video overlay. The compositing module (250) can overwrite the to-be-replaced pixel values of the frame of the gameplay video. In this way, the compositing module (250) produces composited frames (252).
The composited frames (252) are provided to the video encoder (260), which is instructed by the broadcasting application (210) to encode the composited frames (252). The video encoder (260) encodes the composited frames (252) and produces a bitstream of encoded video data (262). In some example implementations, the video encoder (260) is a hardware encoder provided as part of the platform for a gaming console. Alternatively, the video encoder (260) is a software video encoder or hardware-accelerated video encoder. The video encoder (260) can produce encoded video data (262) in H.264/AVC format or some other format (e.g., H.265/HEVC, VPx, SMPTE 421M).
As part of the encoding, the video encoder (260) can use intra-frame compression to remove redundancy within a given one of the composited frames (252). For example, the video encoder (260) uses spatial prediction, frequency transforms and quantization for spatial prediction residuals, and entropy coding of quantized transform coefficients and side information such as spatial prediction directions, mode decisions, etc. As part of the encoding, the video encoder (260) can also use inter-frame compression to exploit redundancy between frames of the composited frames (252). For example, the video encoder (260) uses motion compensation, frequency transforms and quantization for motion-compensated prediction residuals, and entropy coding of quantized transform coefficients and side information such as motion vector values, mode decisions, etc. In some example implementations, the video encoder (260) can partition a composited frame into regions (e.g., associated with different slices) in order to separate the encoding for the camera video overlay from the encoding for the gameplay video. In addition to improving coding performance (e.g., by disabling prediction across slice boundaries), this can simplify the process of allocating different levels of quality to the camera video overlay relative to the gameplay video. For example, the encoder can assign higher quality to the camera video overlay relative to the gameplay video. Or, the encoder can assign lower quality to the camera video overlay relative to the gameplay video.
The encoded video data (262) is provided to a buffer, where it is stored for transmission over a network connection across the network (290). For example, the encoded video data (262), after transport-level packetization, forward error correction, etc. is transmitted to a server for storage and/or broadcasting to one or more viewers. The server can be associated with a Web site for general video streaming, or the server can be associated with a specialized Web site for streaming of gameplay video.
Locally, the camera video overlay is not rendered to the display device (except perhaps as part of a preview, as described below). Thus, the camera video overlay does not obstruct the gameplay area seen by the player of the game. The camera video overlay is only added to the gameplay video that is broadcast (and, in some cases for some implementations, shown in the preview, as described below). Alternatively, the camera video overlay can be composited over the gameplay that is locally rendered.
The broadcasting application (210) can also control configuration of various aspects of gameplay broadcasting. For example, a configuration module (212) of the broadcasting application (210) causes the gaming console to render for display a configuration menu. The configuration menu shows options for placement of the camera video overlay on the gameplay video. The configuration module (212) of the broadcasting application (210) receives user input that specifies the placement of the camera video overlay on the gameplay video. The user input can be received from the gaming console, which receives it, for example, from the user through a game controller input or other input. The broadcasting application (210) stores an indication of the specified placement in settings (216) for gameplay broadcasting. The settings (216) can be stored locally (e.g., in the gaming console). Or, the settings can be stored in cloud-based storage for the user. The settings (216) can be specific to a user profile for the user, specific to a game under the user profile, or generally applicable for the gaming console.
In the example of
The configuration menu can also show options and controls for other settings for gameplay broadcasting. For example, the configuration menu shows (a) a preview of the camera video overlay composited on the gameplay video at the specified placement, (b) an option for enabling or disabling input from a video camera for the camera video overlay, (c) an option for enabling or disabling input from a microphone, (d) options for re-sizing the camera video overlay, (e) options for including a chat stream, and/or (f) another option or control. The configuration menu can include one or more text entry areas for the user to specify a title, player, author, date, comments, etc. for the gameplay broadcasting. The configuration menu can even include encoding options for the gameplay broadcasting, such as spatial resolution for composited frames, temporal resolution (that is, frame rate) and/or quality level (expressed in terms of bitrate or classification such as low/medium/high).
Different video games have different graphical features such as score, map, inventory, timer, message stream, health indicator, roster of players or characters, and so on. Depending on the game, or even depending on the stage of the game, the parts of a screen that show the most important information can be different. When a camera video overlay is always positioned at the same location, it may cover or obscure important information for the game. In some example implementations, a gaming console (or companion device) permits a user (e.g., player of a game) to specify the position at which a camera video overlay is composited on gameplay video. Thus, the user can select the position that the user considers best for the camera video overlay that is composited on gameplay video for broadcasting. This can improve the quality of the gameplay video that is broadcast. The gaming console (or companion device) can also permit the user to control other aspects of the configuration of gameplay broadcasting.
The configuration menu shows settings for gameplay broadcasting. The settings include a control (340) for on/off of the camera video overlay. When the camera is enabled (camera video overlay is on), the settings include a control (350) for position of the camera video overlay on gameplay video for broadcasting. For example, in
The configuration menu also includes a preview area (330). In the preview area (330), a still frame of the gameplay video (or a series of frames of the gameplay video) is shown with the camera video overlay composited at the specified location. This can give the user a sense of how the gameplay broadcast will be viewed by others, without obscuring the gameplay area (320) itself or obscuring the gameplay area in full-screen mode.
The configuration menu can include other and/or additional settings. For example, in
The configuration menu can remain open during gameplay. In this case, the configuration menu can include a chat stream with comments, suggestions, requests, etc. from viewers of the gameplay broadcast. Viewers can also verbally respond to the gameplay broadcast, with the audio responses conveyed back to the gaming console of the player and rendered. Also, whether the configuration menu remains open during gameplay or is closed then reopened during gameplay, the user can change the placement of the camera video overlay on the gameplay video during broadcasting.
The gaming console or companion device renders (410) for display a configuration menu. The configuration menu shows options for placement of a camera video overlay on gameplay video, which represents gameplay rendered for display. For example, the configuration menu is part of a graphical bar that is snapped to the side of a gameplay area, as described with reference to
The options for the placement of the camera video overlay can include four corner locations on the gameplay video. Or, the options for the placement of the camera video overlay can include other and/or additional locations on the gameplay video (e.g., middles of sides, free positioning at arbitrary x, y coordinates on the gameplay video). The configuration menu can present other and/or additional information for options, controls, settings, etc., as described above with reference to
The gaming console or companion device receives (420) user input specifying the placement of the camera video overlay on the gameplay video. For example, the user input can specify one of the four corner locations on the gameplay video, or otherwise specify an option for a location of the camera video overlay on the gameplay video for broadcasting.
The gaming console or companion device stores (430) an indication of the specified placement. For example, the gaming console stores the indication of the specified placement in settings for a broadcasting application. The settings for the broadcasting application can be stored locally (e.g., in the gaming console) and/or in cloud-based storage for the user. The settings can be specific to a user profile for the user, specific to a game under the user profile, or generally applicable for the gaming console. Subsequently, during gameplay, the gaming console can render gameplay for display. For gameplay broadcasting, the gaming console captures the gameplay as frames of the gameplay video. The gaming console composites frames of the camera video overlay on respective frames of the gameplay video at the specified placement. The frames of the camera video overlay are not composited on the gameplay that is rendered for display, however. Thus, local gameplay is not obscured by the camera video overlay.
A specified placement of camera video overlay on gameplay video can be used throughout an entire gameplay broadcast. Alternatively, the placement of the camera video overlay on the gameplay video can be changed during broadcasting. For example, during broadcasting, the gaming console or companion device renders (410) the configuration menu again, receives (420) new user input specifying the placement of the camera video overlay on the gameplay video, and stores (430) an indication of the newly specified placement, which can be used when compositing the camera video overlay on the gameplay video for broadcasting. In this way, during broadcasting, the user can change which portion of the gameplay video is obscured by the camera video overlay, depending on the stage of the gameplay or some other factor.
The gaming console receives (510) a frame of gameplay video, which represents gameplay rendered for display. For example, the frame of the gameplay video is received from a screen capture module that captures the gameplay rendered for display.
The gaming console also receives (520) a frame of a camera video overlay. For example, the frame of camera video overlay is received from a video camera that captures natural video. The gaming console can re-size the camera video overlay, if appropriate. For example, the gaming console can down-sample or otherwise scale down frames of the camera video overlay to a size appropriate for a picture-in-picture window in the gameplay video.
The gaming console composites (530) the frame of the camera video overlay on the frame of the gameplay video to produce a composited frame. For example, the gaming console replaces certain pixel values of the gameplay video frame with pixel values of the frame of the camera video overlay.
The gaming console provides (540) the composited frame to a video encoder that produces encoded video data. Finally, the gaming console provides (550) the encoded video data to a buffer for transmission. The encoded video data can be streamed over a network to a server for storage and/or broadcasting to one or more viewers. Or, the encoded video data can be streamed over a network to another gaming console or a viewing device (such as a companion device for the other gaming console or the originating gaming console) for storage and/or playback.
The gaming console checks (560) whether to continue with a next frame. If so, the gaming console receives (510) another gameplay video frame and receives (520) another frame of the camera video overlay. Thus, the gaming console can repeat some operations or all operations of the technique (500) on a frame-by-frame basis. At the same time, locally, the gaming console renders gameplay for display without compositing the camera video overlay on the gameplay. Thus, the gameplay seen by the player is not obstructed by the camera video overlay.
In many of the preceding examples, frames of camera overlay video are composited on frames of gameplay video to produce composited frames, and the composited frames are encoded and streamed. Alternatively, a gaming console encodes frames of camera overlay video for a first bitstream of encoded video data, encodes frames of gameplay video for a second bitstream of encoded video data, and streams the two bitstreams to a server or other gaming console (or viewing device). In this case, a server-side application or viewer-side application can composite the frames of camera overlay video on the gameplay video. Or, the frames of camera video can be displayed on a tablet computer or smartphone (e.g., as a companion device) while the gameplay video is rendered on a display device through a gaming console.
In view of the many possible embodiments to which the principles of the disclosed invention may be applied, it should be recognized that the illustrated embodiments are only preferred examples of the invention and should not be taken as limiting the scope of the invention. Rather, the scope of the invention is defined by the following claims. We therefore claim as our invention all that comes within the scope and spirit of these claims.
The present application claims the benefit of U.S. Provisional Patent Application No. 61/979,888, filed Apr. 15, 2014, the disclosure of which is hereby incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61979888 | Apr 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14465701 | Aug 2014 | US |
| Child | 15712959 | US |
