Patent Grant
12309437
Aspects of the present disclosure relate to streaming video, specifically aspects of the present disclosure are related to adding viewer to broadcaster interaction to streaming video.
Streaming of media presentations, such as video games, movies and TV shows has become popular. Streaming websites such as Twitch and Facebook gaming are gaining increasing popularity. A unique aspect of streaming websites is interaction with the broadcaster. Generally, interactions of the viewer with the broadcaster with streaming websites is limited to applications running in tandem with the media presentation on the broadcasting device. As such any interaction features of the streaming application must be made compatible with the media presentation on a case-by-case basis. Thus, the most common interactions between viewers and a broadcaster are through a chat window. Broadcasters generally have multiple monitors and, in some cases, multiple devices for broadcasting the media presentation and interacting with viewers.
A broadcaster may have one monitor displaying the media presentation on a first application and a second monitor displaying user interactions on a second application. Here, the viewer interactions do not appear on the media presentation itself but instead on a second application which may include text or voice chat functions. The second application may also display a streaming window which displays captured image frames from the media presentation. The streaming window may also include user interactions in the image frames. Meanwhile the media presentation itself runs without any interactions from the user. In cases where the media presentation is a video game or other mentally taxing application, the broadcaster's full attention may be taken up by the media presentation on the first display and may be unable to look at a second display or application for user interaction. Additionally, there is often a large delay (e.g., one second to tens of seconds) between the media presentation and interactions displayed by the second application. In some cases, this delay is intentional to prevent cheating in multiplayer games but often it is due to system or network latency. Thus, it would be advantageous for user interactions to be displayed directly on a broadcaster's media presentation with low latency. It is within this context that aspects of the present disclosure arise.
The teachings of the present disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the invention. Accordingly, the exemplary embodiments of the invention described below are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.
Viewer interaction is a major draw for many viewers of streaming media. Viewers enjoy chatting with their favorite media broadcasters (also known as “Streamers”) in text chat or voice chat. Sometimes a streamer will ask their viewers for help on a media presentation such as a video game. The help from their viewers is often limited to text-based explanations or explanations over voice chat. Some applications have integration with streaming video platforms allowing specific user interactions enabled by application programming. There is currently no way to user interactions to media presentations that are not specifically designed to accommodate those interactions.
Additionally streaming media presentations has also become common between smaller friend groups. Oftentimes friends will stream media presentations with each other to share a new game, or get help on a game; friends may sometimes hold movie nights. It would be useful for friends to be able to help other friends with on-screen interactions such as arrows, circles, boxes, zoom-in areas, free-form lines and on-screen text messages.
Aspects of the present disclosure add application agnostic, low system latency, viewer interactions, to broadcaster media presentations. Operating system level media presentation streaming, and interaction overlay code may be used to provide application agnostic interaction and decreased latency.
Here the zoom function 201 may be activated by a viewer performing a ‘pinch to zoom’ function with their cursors 208. A square or other shape may be displayed over the area of the broadcaster's interaction interface that is being zoomed. The zoom 201 function may use pixel values from the media presentation in the zoom area 203 and apply an upscaling algorithm to the pixel values from the media presentation in the zoom area to upscale the image. The upscale image may be displayed by the broadcaster's interaction interface overlay, over the original location 203 or in a second location 201. The upscaling algorithm may be for example and without limitation, nearest neighbor interpolation, bicubic or bilinear scaling, sinc or lanczos resampling, box sampling, Fourier-transform based interpolation, edge directed interpolation, hqx, and similar. In some implementations the upscaling algorithm may apply machine learning based approaches.
Additionally other user interactions may be displayed by the broadcaster's interaction interface such as a free form arrow 207 or freeform line 205 or viewer cursors 208. The viewer's interactions may also be combined with the zoom function. As shown here the free form arrow 207 is drawn over the zoomed in castle tower 206 and the free form line 205 is drawn in the zoomed in area 201. Here, the zoom function reveals a person on the rampart that may have been too small for the broadcaster to see without the function. The free form line 205 and free form arrow 207 may be drawn by the same viewer who performed the zoom function or a different viewer. The zoom function may continue for as long as a viewer holds the pinched zoom 208 or the zoom may be on a timer. The zoom timer may have a length that is viewer selectable or broadcaster selectable.
In some implementations, the broadcaster's interaction interface overlay may allow the broadcaster to control other types of unwanted interactions, e.g., undesired interactions from bad-actor viewers. In some implementations, the broadcaster's interaction interface overlay may allow the broadcaster to prioritize interactions, e.g., by giving higher priority to interactions by VIP viewers or friends or influencers. In some implementations interactions may be prioritized based on the popularity of the feedback provider so their feedback appears before other feedback providers. In other implementations, some feedback may be excluded if the volume of feedback sufficiently high. In such situations, the system may create an alternative view of feedback, e.g., for “top feedback only”. Furthermore, the broadcaster may have the option to selectively disable viewer interactions for some or all viewers, e.g., if they think the feature is blocking their view of the media presentation, or if they wish to allow it only for a limited group of friends or subscribers on the platform.
The next layer is the system service layer 507, which provides a service interface to the operating system 503 for the application layer 508. The system service layer may include a file system which indexes files stored in hardware by the operating system, Graphical User Interface (GUI) which provides a graphical representation of the services provided by the OS 503 and allows a user to more easily operate the general-purpose computers and task management which manages services provided by the OS. Finally, the Application layer 508 uses services from the OS provided by the system service layer to generate additional computer functionality separate from the OS. Applications run at a less privileged state than the operating system and generally require an application programming interface API to interact with the OS. Generally, unless specifically programmed applications do not interact with other applications unless it is through the OS layer. The provision of screensharing native to the OS layer allows the general-purpose use of screensharing without the use of specialized application or customization of applications for screen sharing.
After a connection has been established with a viewing device the broadcaster may enable viewer-controlled events at 604. If viewer-controlled events are not enabled, the broadcasting device may not display viewer interactions even if viewer interaction parameters are received. A message may also be sent to the viewing device informing it that viewer-controlled events are enabled on the broadcasting device. The broadcasting device may include enable able limitations on the types of viewer interactions and placement of viewer interactions on the broadcasting device display. The broadcasting device may have user-controlled settings that control the placement of viewer interaction and type of user interactions at 605. After a connection has been established with the viewing device, the broadcasting device may send an image frame to the viewing device at 606. In some implementations image frames may be sent to a streaming server which then sends image frames to the viewing device.
After sending an image frame to a viewing device, the viewing device may generate viewer interaction parameters from interactions by the viewer with the image frame. These viewer interaction parameters may be sent back to the broadcasting device, where they are received by the broadcasting device at 607. In alternative implementations the viewing device may send the viewer interaction parameters to a streaming server and the streaming server sends the viewer interaction parameters to the broadcasting device. The broadcaster interface overlay may take the viewer interaction parameters and overlay the interactions over the media presentation at 608. Here, user interaction parameters may include, for example and without limitation, an interaction type and/or an interaction location. The interaction type may for example and without limitation describe the displayed interaction, e.g., a zoom, arrow, free form line, cursor etc. The interaction location may describe the location of the interaction in the media presentation e.g., a pixel location or coordinate grid location of the interaction or portions of the interaction (such as height, width, center, radius, diameter, edge locations etc.). By way of example, and not by way of limitation, interaction type and the locations of a viewer's interactions may be determined from inputs by the viewer through graphical user interface software and/or hardware associated with the viewer's device.
The viewer interactions may be displayed back to the viewer. This may allow the viewer to more easily understand broadcaster commentary or actions. The broadcasting device may generate subsequent image frames that include the viewer interactions from the broadcaster's interaction interface overlay, as indicated at 609. The subsequent image frame may be generated from a subsequent scene of the media presentation or a frame that is currently displayed on the broadcasting device as long as the frame index is always advancing, to avoid sending a frame more than once. These subsequent image frames may then be sent to the viewing device at 610. The viewer may then interact with the subsequent image frame(s) and send the interaction parameters to the broadcasting device repeating at 611 back to step 607 and through the rest of the method steps 608-610. Alternatively, the broadcaster may end streaming the media presentation at 612.
The image frame from the media presentation on the broadcasting device is displayed on the viewing device at 703. Additionally, a viewer interaction interface overlay is generated over the image frame at 704, this may occur concurrently with display of the image frame or before the image frame is displayed or after the image frame is displayed. The viewer interaction interface overlay may be invisible to the user or may display types of interactions the user may place within the image frame. In any case the viewer interaction overlay captures the viewer's interactions with the image frames and translates those interactions into interaction parameters at 705. By way of example, and not by way of limitation, the interaction overly may include a graphical user interface software and/or hardware associated with the viewer's device that allows the viewer to determine interaction parameters such as the interaction type and the interaction location.
The interaction parameters are then sent to the broadcasting device at 706. As discussed above the broadcasting device may include the user interaction in a subsequent image frame. That subsequent image frame including the viewer interaction is then transmitted to and received by the viewer device at 707. The subsequent image frame may then be displayed at 703, repeating at 708 the steps 704, 705, 706, 707 until the viewer or broadcaster ends streaming or stops allowing viewer interactions 709.
Interaction between the viewer and broadcaster according to aspects of the present disclosure may be understood by referring to the timing diagram depicted in
Once communication has been established with the viewing device 801, the broadcasting device 802 sends image frames to the viewing device at 809. The viewing device displays the image frames from the broadcasting device at 810. The viewing device may provide a viewer interaction overlay over the displayed image frames. Using the viewer interaction overlay, the viewing device may capture the viewer's interaction at 811 and translates the interaction into interaction parameters. The interaction parameters are then sent to the broadcasting device at 812. The broadcasting device 802 may generate a subsequent scene from the media presentation and display the subsequent scene at 813. Over the subsequent scene, the viewer interactions may be displayed and overlaid at 814 using the viewer interaction parameters. The broadcasting device 802 may then generate an image frame for streaming to the viewing device that includes the viewer overlaid interactions at 815. Including the viewer interaction in a subsequent image frame may allow viewers to understand the context of a broadcaster's action and also understand that their interactions are received by the broadcasting device and displayed on the broadcasting device. Broadcaster-specific information, e.g., number and names of viewers may be omitted from the image generated by the broadcasting overlay. The subsequent image frame including viewer interaction is sent to the viewer device at 816. The viewer device may display the image frame including the viewer interactions at 817 but omitting any broadcaster-specific information.
In some implementations, it may be possible for the locations of user interactions to follow the locations of objects within the image frame. For example, in
The system 1000 generally includes a central processor unit (CPU) 1003, and a memory 1004. The system 1000 may also include well-known support functions 1006, which may communicate with other components of the system, e.g., via a data bus 1005. Such support functions may include, but are not limited to, input/output (I/O) elements 1007, power supplies (P/S) 1011, a clock (CLK) 1012 and cache 1013.
The system 1000 includes a mass storage device 1015 such as a disk drive, CD-ROM drive, flash memory, solid state drive (SSD), tape drive, or the like to provide non-volatile storage for programs and/or data. The system 1000 may also optionally include a user interface unit 1016 to facilitate interaction between the system 1000 and a user. The user interface 1016 may include a keyboard, mouse, joystick, light pen, or other device that may be used in conjunction with a graphical user interface (GUI). The system 1000 may also include a network interface 1014 to enable the device to communicate with other devices over a network 1020. The network 1020 may be, e.g., a local area network (LAN), a wide area network such as the internet, a personal area network, such as a Bluetooth network or other type of network. These components may be implemented in hardware, software, or firmware, or some combination of two or more of these.
The CPU 1003 may include one or more processor cores, e.g., a single core, two cores, four cores, eight cores, or more. In some implementations, the CPU 1003 may include a GPU core or multiple cores of the same Accelerated Processing Unit (APU). The memory 1004 may be in the form of an integrated circuit that provides addressable memory, e.g., random access memory (RAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), and the like. The main memory 1004 may include application(s) 1021 used by the processor 1003 to generate for example, a chat program internet browser, video game, movie etc. The main memory 1004 may also include an operating system (OS) 1022 the operating system may run over a hardware abstraction layer as depicted in
According to aspects of the present disclosure the processor 1003 may carry out connection and viewer interaction, e.g., as discussed above with respect to
The Mass Storage 1015 may contain Application or Programs 1017 that are loaded to the main memory 1004 when processing begins on the application 1023. Additionally, the mass storage 1015 may contain data 1018 used by the processor during start up to load the OS 1009.
As used herein and as is generally understood by those skilled in the art, an application-specific integrated circuit (ASIC) is an integrated circuit customized for a particular use, rather than intended for general-purpose use.
As used herein and as is generally understood by those skilled in the art, a Field Programmable Gate Array (FPGA) is an integrated circuit designed to be configured by a customer or a designer after manufacturing—hence “field-programmable”. The FPGA configuration is generally specified using a hardware description language (HDL), similar to that used for an ASIC.
As used herein and as is generally understood by those skilled in the art, a system on a chip or system on chip (SoC or SOC) is an integrated circuit (IC) that integrates all components of a computer or other electronic system into a single chip. It may contain digital, analog, mixed-signal, and often radio-frequency functions—all on a single chip substrate. A typical application is in the area of embedded systems.
A typical SoC includes the following hardware components:
These components are connected by either a proprietary or industry-standard bus. Direct Memory Access (DMA) controllers route data directly between external interfaces and memory, bypassing the processor core and thereby increasing the data throughput of the SoC.
A typical SoC includes both the hardware components described above, and executable instructions (e.g., software or firmware) that controls the processor core(s), peripherals, and interfaces.
Aspects of the present disclosure allow for enhanced remote interaction among players of video games. Various implementations discussed herein go beyond allowing players to share what is present on their respective screens. Such implementations additionally allow viewer interactions to be displayed directly on a broadcaster's media presentation with low latency.
While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications and equivalents. Therefore, the scope of the present invention should be determined not with reference to the above description but should, instead, be determined with reference to the appended claims, along with their full scope of equivalents. Any feature described herein, whether preferred or not, may be combined with any other feature described herein, whether preferred or not. In the claims that follow, the indefinite article “A”, or “An” refers to a quantity of one or more of the item following the article, except where expressly stated otherwise. The appended claims are not to be interpreted as including means-plus-function limitations, unless such a limitation is explicitly recited in a given claim using the phrase “means for.”
This application is a continuation of U.S. patent application Ser. No. 17/852,993 filed Jun. 29, 2022, the entire contents of which are incorporated herein by reference. This application is a continuation of International Patent Application Number PCT/US2023/022854 filed May 19, 2023, the entire contents of which are incorporated herein by reference. International Patent Application Number PCT/US2023/022854 is a continuation of U.S. patent application Ser. No. 17/852,993 filed Jun. 29, 2022.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8559516 | Hardacker | Oct 2013 | B2 |
| 9195880 | Levoy | Nov 2015 | B1 |
| 10410376 | Waggoner et al. | Sep 2019 | B1 |
| 10785512 | Kambhatla et al. | Sep 2020 | B2 |
| 11792443 | Agoston | Oct 2023 | B1 |
| 20090271821 | Zalewski | Oct 2009 | A1 |
| 20180081509 | Varela et al. | Mar 2018 | A1 |
| 20190273954 | Evans | Sep 2019 | A1 |
| 20210099759 | Armstrong et al. | Apr 2021 | A1 |
| 20220132212 | Moehrle | Apr 2022 | A1 |
| 20220138902 | Lea | May 2022 | A1 |
| 20220207804 | Collins et al. | Jun 2022 | A1 |
| Entry |
|---|
| “Typical Software Architecture” ebrary.net, Academic library 2014-2022, Available at: https://ebrary.net/22045/computer_science/typical_software_architecture, (Accessed: 2022). |
| International Search Report and Written Opinion for International Application No. PCT/US2023/022854, dated Oct. 10, 2023. |
| Notice of Allowance for U.S. Appl. No. 17/852,993, dated Jun. 28, 2023. |
| Number | Date | Country | |
|---|---|---|---|
| 20240048781 A1 | Feb 2024 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/US2023/022854 | May 2023 | WO |
| Child | 18380579 | US | |
| Parent | 17852993 | Jun 2022 | US |
| Child | 18380579 | US | |
| Parent | 17852993 | Jun 2022 | US |
| Child | PCT/US2023/022854 | WO |
