Patent Grant
10498794
The present invention is related to the field of streaming media processing and messaging technologies. More specifically, the present invention relates to a social entertainment platform that enables live social video streaming and messaging.
Traditional platforms that allow a streamer to stream a live video stream and viewers to watch the live video stream, introduce significant delays in communications between the streamer and the viewers on the average of 45 seconds. Additionally, communications from the viewers to the streamer appear on a separate screen or in a separate window of the streamer's computing device. Furthermore, these communications are not controlled and, thus, can include irrelevant messages and can become voluminous, making the communications difficult to read. These factors detract from user experience. There exists a need for improvements in streaming video and messaging technologies to better the user experience.
Embodiments of the present invention are directed to a social entertainment platform that supports live social video streaming and messaging. One common use for this social entertainment platform is to allow audience members to watch and interact with a person playing an electronic game. The social entertainment platform allows a streamer to stream live (in substantially real-time) on a network(s), such as the Internet, for a live audience to view and to respond to the live entertainment. The social entertainment platform includes several unique areas, including a real-time broadcasting component, a real-time messaging component, and an integrated user interface (UI) component, that make the social entertainment platform advantageous to use over traditional platforms.
In one aspect, a method of using social entertainment platform for real-time distribution of content from a streaming device to a plurality viewing devices is provided. The method includes injecting, by a first software executing on the streaming device, a DLL into a second software that is also executing on the streaming device such that video generated by the second software is not sent to a processing unit on the streaming device but is rather captured, frame buffer by frame buffer, by the first software executing on the streaming device. The method also includes sending by the first software the captured video to a remote location for real-time broadcasting while simultaneously generating by the first software a modified video from the captured video for rendering by the processing unit on a display unit of the streaming device.
In some embodiments, the first software is executed on the streaming device prior to executing the second software on the streaming device.
In some embodiments, generating by the first software a modified video includes writing comments to the frame buffers such that the comments are displayed as overlays when the modified video is rendered on the streaming device.
In some embodiments, the comments are generated on web browsers of the plurality of viewing devices.
In some embodiments, the comments are filtered comments that are more relevant to a user associated with the streaming computing device than not.
In some embodiments, the method further includes, after capturing, frame buffer by frame buffer, and before sending by the first software the captured video, encoding by the first software raw frames of the captured video into a compressed video stream to be transmitted in real-time by the first software to the remote location.
In some embodiments, the remote location includes a cloud infrastructure of a plurality of remote servers configured in a plurality of repeater levels such that the captured video simultaneously reaches the plurality of viewing devices.
In some embodiments, the compressed video stream also includes a live camera feed captured by the streaming device.
In some embodiments, the method further also includes receiving the captured video simultaneously at each of the plurality of viewing devices, rendering the captured video by a web browser on each of the plurality of viewing devices, and displaying a filtered view of comments on each of the plurality of viewing devices.
In another aspect, a system for real-time distribution of content from a streaming device to a plurality of viewing devices is provided. The system includes a streaming device redirecting video generated on the streaming device to be captured rather than to be rendered, transmitting therefrom an original version of the captured video while simultaneously generating a modified version of the captured video to be rendered. The system also includes a cloud infrastructure including a fan-out architecture of remote servers arranged in a plurality of repeater levels. Each of the remote servers receives and forwards a compressed video stream. The system further includes a plurality of viewing devices simultaneously receiving the compressed video stream from the cloud infrastructure. Each of the viewing devices decodes and renders the video stream in a web browser.
In some embodiments, the streaming device captures a live camera feed and transmits the live camera feed along with the original version as the compressed video stream.
In some embodiments, the streaming device and the remote servers each implements WebRTC protocols to enable web broadcasting of the compressed video stream.
In some embodiments, the plurality of viewing devices generates a plurality of comments from the web browsers.
In some embodiments, the plurality of comments are filtered for each viewer associated with one of the viewing devices and for a streamer associated with the streaming device.
In some embodiments, the modified version of the captured video is the original version of the captured video modified with filtered comments for the streamer.
In some embodiments, the filtered comments are displayed as overlays when the modified version of the captured video is rendered on the streaming device.
In some embodiments, a filtered view of comments for each viewer is also displayed in the web browser on a respective viewing device.
In some embodiments, the system also includes a real-time messaging component comprising a plurality of annotator elements and a sieve element. The plurality of annotator elements is configured in an arrangement with a first and a last of the plurality of annotator elements both in direct communication with the sieve element.
In some embodiments, each of the annotator elements includes logic to create a specific annotation regarding a comment for each requesting device, and the sieve elements includes logic to make a determination as to whether to pass the message to each requesting device based on the annotations from the plurality of annotator elements.
The foregoing will be apparent from the following more particular description of example embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present invention.
In the following description, numerous details are set forth for purposes of explanation. However, one of ordinary skill in the art will realize that the invention can be practiced without the use of these specific details. Thus, the present invention is not intended to be limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and features described herein.
Embodiments of the present invention are directed to a social entertainment platform that supports live social video streaming and messaging. One common use for this social entertainment platform is to allow audience members to watch and interact with a person playing an electronic game. The social entertainment platform allows a streamer to stream live (in substantially real-time) on a network(s), such as the Internet, for a live audience to view and to respond to the live entertainment. The social entertainment platform includes several unique areas, including a real-time broadcasting component, a real-time messaging component, and an integrated user interface (UI) component, that make the social entertainment platform advantageous to use over traditional platforms. The social entertainment platform and associated service(s) described herein will be provided by Caffeine Inc., currently located in Palo Alto, Calif. as of the time of this application.
Overview.
The cloud infrastructure includes remote server(s) 120 that implement Caffeine's service described herein, particularly the real-time broadcasting of video. For the sake of simplicity, only one streamer 105 and only one viewer 115 are shown, although the social entertainment platform is configured to support numerous streamers 105 and numerous viewers 115 for each streamer 105 who are simultaneously utilizing the Caffeine service. Streamers 105 and viewers 115 are all users/members of the Caffeine service. The term “user” and the term “member” are used interchangeably herein. In some embodiments, each user who uses the Caffeine service has created a user account at a server, which can be a remote server 120 or an account/web server that is separate and distinct from the remote server 120.
In
For purposes of discussion, assume the streamer 105 is a video gamer and the viewer 115 accesses the streamer's feed. However, it should be noted that although the discussion herein relates to the social entertainment platform being used by video gamers and their followers/fans, it is contemplated that the social entertainment platform can be used by any and all creators in different media sectors and their followers/fans. Followers/fans are subscribers who subscribe to the streamer's feeds.
Referring back to
Comments 310, which are herein also referred to as reactions or messages, input by the viewers 115 for the streamer 105 are displayed in real-time as overlays on top of the game 305, as illustrated in an exemplary user interface 300 of
Referring back to
After a successful log in, the viewer 115 sees the viewer's own personalized lobby of feeds associated with the viewer's friends. The viewer has a mutually-initiated online relationship with each friend. The viewer's friends are members of the Caffeine service whom the viewer 115 is following and/or who are following the viewer 115. The lobby of feeds includes streams that are currently live on Caffeine that the viewer's friends are either watching or broadcasting. In some embodiments, a lobby is a graphical presentation of a list of currently available streams for viewing in a user-specific order.
Continuing with the hypothetical, assume the streamer 105 and the viewer 115 are friends. The viewer 115 can thus find the streamer's feed in the viewer's lobby. Upon selecting the streamer's feed, the corresponding video and audio of the game play and the corresponding video and audio of the streamer 105 start streaming from the cloud infrastructure 110 on C2. In some embodiments, C2 does not need to run the Caffeine client software to access and to display live streams from the cloud infrastructure 110. Instead, live streams can be viewed from the web browser on C2. However, the Caffeine client software can also be launched to view live streams.
In some embodiments, as illustrated in
The second section 415 of the user interface 400 allows the viewer 115 to provide comments for the streamer 405 and other viewers in the audience to see, and to see comments from the other viewers in the audience. As soon as the viewer 115 enters a comment on C2, the comment 310 is displayed in real-time as an image (e.g., animated GIF) or as text on top of the game play 310 on C1, as well in real-time on other viewing devices used by the other viewers in the audience. In some embodiments, the viewer's comment appear in real-time in web browsers on the other viewing devices.
Briefly, in some embodiments, messages from the audience are filtered so that each user (e.g., either a viewer in the audience or the streamer) do not see all messages but only filtered and/or relevant messages based on one or more filters that are configured for that user. An exemplary filter is a Friends filter, which allows a user to see messages from those that the user is following. Other exemplary filters, such as a Location filter, a Content filter, a Top Fan filter, and an Upvoted filter, are contemplated. The Location filter allows a user to see messages from those viewers in the same or a selected geographical region. The Content filter allows a user to see messages that do not include content that the system or the user has determined as being inappropriate. The Top Fan filter allows a user to see messages from those who are “popular” in the audience such as those being followed by a predetermined number of viewers. The Upvoted filter allows a user to see a message that has been selected by sufficient viewers in the audience such that the message earns exposure to a greater number of users. Details of the real-time messaging are further discussed below.
In some embodiments, the user interface 400 includes a third section 420, which allows the viewer 115 to start/join at least one private chat, each with specific viewers from the audience that the viewer 115 wants to chat with. The second and third sections 415, 420 are simultaneously displayed in the user interface 400 such that the viewer 115 can engage in the private conversation while seeing a filtered view of what the audience is saying. In some embodiments, the streamer 105 can also participate in the private conversation.
In some embodiments, viewers are able to obtain virtual goods to enhance their experience on the social entertainment platform. For example, a viewer is able to purchase a “virtual microphone” to make one or two comments more prominent so that the comment(s) can be seen by the streamer and all viewers who are currently watching the streamer's game play. For another example, a viewer is able to sponsor (e.g., tip) or gift (e.g., flowers) a streamer.
As demonstrated herein, the social entertainment platform provides a people-centric approach for obtaining video streams and comments. Unlike traditional platforms that have a content-centric approach, a user can select which members to follow so that the user sees first and foremost comments from those in the audience the user wants to hear from.
Real-Time Video Broadcasting.
Referring back to
When the Caffeine client software recognizes that a compatible (e.g., “streamable”) game is running, the Caffeine client software inserts a DLL (dynamic link library) into the game. The DLL changes the APIs (application program interfaces) that the game is using to send drawing commands to the GPU so that the Caffeine client software is able to access them. By accessing these drawing commands first, the Caffeine client software is able to capture the video, frame buffer by frame buffer. The Caffeine client software sends the captured video, along with the live camera feed, to the cloud infrastructure 110 for distribution to viewers 115. At substantially the same time, the Caffeine client software modifies the captured video by writing comments from the streamer's audience back to the frame buffers to the GPU for rendering on the display unit 130 for the streamer 105 to see. The Caffeine client software essentially makes available two versions of the captured video. One of the versions is the original, unaltered video, which is to be forwarded to the cloud infrastructure 110 for real-time video broadcasting to the streamer's audience. The other version is an altered video, which is the original video modified with audience comments, that is rendered to the streamer 105.
The Caffeine client software is built out of different components. In some embodiments, one of them is the WebRTC (Web Real-Time Communication) framework. Other components could be substituted. WebRTC is a set of communications protocols and application programming interfaces that enable real-time communication over peer-to-peer connections. The WebRTC framework in the Caffeine client software takes the raw frames and encodes them into a compressed video stream (e.g., VP8, H.264) so that it can be transmitted in real-time to the cloud infrastructure 110 for broadcasting.
The cloud infrastructure 110 includes a fan-out architecture of servers 120 arranged in a plurality of repeater levels (Level 1 to Level N), wherein each of the repeater levels has more servers than the previous repeater level in the fan-out architecture. In some embodiments, each server 120 in the cloud infrastructure 110 runs Janus, which an open source software, and also implements WebRTC protocols. Each server 120 takes incoming compressed video stream (including the captured game play and the camera feed) and repeats it to the next repeater level servers 120 that are coupled to it. For example, the compressed video stream from the C1 is received by the server 1S1 at the first repeater level, which forwards it to the servers in the second repeater level, namely 2S1, 2S2 and 2S3, and each of those servers forwards it on to those servers coupled to it, and so forth. This repetition has a multiplicative effect, allowing the compressed video stream to reach all requesting viewers 115 simultaneously and, thereby, allowing all participants to engage (e.g., seeing and replying to comments) with one another in substantially real-time.
Eventually, the compressed video stream reaches the viewer's computing device C2 from server NSX. The web browser on C2 is configured to decode and render the compressed video stream received from one of the plurality of servers 120. The fan-out architecture of servers 120 prevents or, at least, limits time shifting of the distribution of the video stream such that everyone who is watching the game play is watching in sync. All requesting viewers 115 are seeing substantially the same content that the streamer 105 is seeing. In some embodiments, there is less than one second of delay between the actual game play and the webcam of the streamer 105 and what the viewers 115 are seeing, which is nearly real-time. In some embodiments, the delay is 190 milliseconds, which is substantially real-time. The substantially real-time aspect of the video broadcast enables instantaneous engagement between the streamer 105 and the viewers 115. For example, a viewer can be respond or react to what the streamer just said or to a move the streamer just made in the game play.
As discussed above, WebRTC is a collection of communications protocols and application programing interfaces that enable real-time communication over peer-to-peer connections. WebRTC's primary use case is video conferencing. However, the Caffeine client software and the cloud servers 120 utilize WebRTC to enable web broadcasting.
Real-Time Messaging.
The following discussion of the real-time messaging component is in the context of one broadcast message regarding a video stream. The real-time messaging component can be implemented on an application server and/or one of the previously discussed servers. It should be noted that all servers discussed herein are communicatively directly coupled or indirectly via network(s), and have access to necessary data stores.
Assume the predetermined threshold has been met for Client 1 and Client 1 has indicated that it is able to receive a message (e.g., comment), and Client 2 has output a message (e.g., comment) 520. Client 1 can either be a viewing device or a streaming device, while Client 2 is a viewing device. The message 520 output from Client 2 is received by a first annotator element 510. The sieve element 505, determining that Client 1 is open for communication, notifies the first annotator element 510 of such. Each annotator element 510, including the first annotator element, receives the message and other annotations, if any, generates its own annotations for the message, and passes the message and all annotations, including its own annotations, to other downstream element(s) 510 coupled with it. The message 520 and all of the annotations are eventually received by the sieve element 505, which makes a determination, based on all of the annotations generated by all of the annotator elements 510, as to whether to forward the message 520 to Client 1 or to drop the message 520. For example, when too many messages are generated for a particular stream, the sieve element 505 ensures that a user interact only with other users that the user should be interacting with and not necessarily with random person who commented on the particular stream. In other words, those messages forwarded to Client 1 from and by the sieve element 505 are typically more relevant to Client 1 than not.
Although it is shown in
In some embodiments, the Relationship Annotator generates annotations describing the relationship between the message generator or poster (e.g., Client 2) and the potential viewer (e.g., Client 1), such as whether Client 1 is a follower of Client 2. In some embodiments, the Graph Annotator generates annotations describing the social distance between Client 1 and Client 2. The closer the clients are grouped implies Client 1 may be more interested than not in Client 2's comments. The proximity of the clients is dependent on the social distance between them, such as the number of friends they have in common. In some embodiments, the Location Annotator generates annotations describing the physical distance between Client 1's location and Client 2's location. As discussed above, other annotator elements 510 are contemplated and can be included as part of the real-time messaging component 500. In some embodiments, each of the annotator elements 510 corresponds with one of the filters described above.
Assume that the sieve element 50 determines to forward the message (e.g., comment) 520 to Client 1 in
Integrated User Interface.
Unlike traditional systems where two separate screens or two separate windows on one screen are used (one to create content, such as game play, and another to consume content from viewers, such as comments), a streamer on the social entertainment platform uses only one window (e.g., game window) on one screen to create and consume content. This allows the streamer to be able to quickly, seamlessly and conveniently interact with the streamer's viewers. Comments can be filtered such that the streamer only sees relevant comments displayed as overlays on top of the game play, as illustrated in
Unlike traditional systems where content displayed for a viewer can be overwhelming, the viewer, upon log in, enters the viewer's own personalized lobby of feeds associated with the viewer's friends. These friends are members of the Caffeine service whom the viewer is following and/or who is following the viewer. The lobby of feeds includes streams that are currently live on Caffeine that the viewer's friends are either watching or broadcasting. The viewer can select a stream to watch with the viewer's friends. Upon a selection, the viewer can watch the streamer's live stream, enter comments, view comments entered by the viewer's friends who are also part of the audience watching the streamer's live stream, and privately chat with one or more of these friends who are in the audience, as illustrated in
Social Entertainment Platform.
The real-time messaging component 610, as described elsewhere herein, enables filtering of messages such that each of the viewers and the streamer sees his/her own filtered view of the messages. The messages from the viewers are filtered separately for each of the viewers and the streamer. The filtered view allows each of the viewers and the streamer to interact only with certain viewers in the audience, such as those who are friends or those whose messages are voted up.
The UI component 615, as described elsewhere herein, enables filtered messages to be displayed as overlays in a game. The comments are displayed as animated images or as text and disappear after a predetermined amount of time to make room for new comments. The display of comments on top of the game allows the streamer to look at one screen on one display unit so that the streamer can quickly interact with the audience, thus bettering the viewing experience.
The UI component 615 also enables a viewer to watch alongside with the viewer's friends. The viewer's personalized lobby includes streams that are currently live on Caffeine that the viewer's friends are either watching or broadcasting. Upon selecting the streamer's feed, the corresponding video and audio of the game play and the corresponding video and audio of the streamer start streaming to the viewer's computing device. The viewer is able to see a filtered view of comments and is also able to chat with one or more friends, thus bettering the audience experience.
At a step 710, the first software sends the captured video to a remote location for real-time broadcasting while simultaneously generating a modified video from the captured video for rendering by the processing unit on a display unit of the streaming device.
In some embodiments, after the video is captured, the first software encodes raw frames of the captured video into a compressed video stream to be transmitted in real-time by the first software to the remote location. In some embodiments, the compressed video stream also includes a live camera feed captured by the streaming device. The remote location includes a cloud infrastructure of a plurality of remote servers configured in a plurality of repeater levels such that the compressed video stream simultaneously reaches the plurality of viewing devices. In some embodiments, the compressed video stream is received simultaneously at each of the plurality of viewing devices. The compressed video stream is rendered by a web browser on each of the plurality of viewing devices, and a filtered view of comments is also displayed on each of the plurality of viewing devices.
In some embodiments, the first software generates the modified video by writing comments to the frame buffers such that the comments are displayed as overlays when the modified video is rendered on the streaming device. The comments are generated on web browsers of the plurality of viewing devices. The comments are filtered comments that are more relevant to a user associated with the streaming computing device than not.
One of ordinary skill in the art will realize other uses and advantages also exist. While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. Thus, one of ordinary skill in the art will understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.
This application claims benefit of priority under 35 U.S.C. section 119(e) of the U.S. Provisional Patent Application Ser. No. 62/428,349, filed Nov. 30, 2016, entitled “Social Entertainment Platform,” which is hereby incorporated by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20090327918 | Aaron | Dec 2009 | A1 |
| 20180091869 | Krishna | Mar 2018 | A1 |
| 20180288467 | Holmberg | Oct 2018 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 62428349 | Nov 2016 | US |
