1. Field of Invention
The present application relates to a video system. More particularly, the present application relates to a real-time video streaming system and a control method thereof.
2. Description of Related Art
Recently, technologies of digital cameras, video cameras, cam recorders and other video recorders are highly developed. Resolutions of aforesaid image/video cameras are advanced from VGA (640×480), HD (1280×720, 720p) to FHD (1920×1080, 1080p) and even 4K2K (3820×2160, 2160p). FHD and 4K2K resolutions can provide more details about the scene, such that the observer can have better experience while watching the images/videos with high resolutions. However, high resolutions mean more pixel data in each frame. The raw data of 4K2K video requires a large transmission bandwidth to be transmitted from a video camera and a monitor. This large transmission bandwidth can be established by a physical wiring, but hard to be realized over a wireless communication.
Users love to see the real-time scene (e.g., ballgames, fashion shows, children activities). The interested scene can be recorded and broadcasted under high resolution, but to transmit videos or streaming videos over wireless communication (e.g., cellular telephony network) will occupy a large bandwidth of the wireless connection.
An aspect of the present disclosure is to provide a real-time video streaming system and a control method thereof for optimizing the transmission bandwidth usage by selecting and transmitting a partial frame from the whole frame recorded or stored by the media source.
The disclosure provides a media streaming system, which includes a media source and a displayer device. The media source is configured to provide a media content with a first resolution. The displayer device communicatively connected with the media source. The displayer device includes a display panel and a gesture input module. The display panel is configured to display with a second resolution lower than the first resolution. The gesture input module is configured to generate a gesture command. The media source is configured to extract a partial frame with the second resolution from the media content according to the gesture command. A relative location of the partial frame in the media content is determined by the gesture command. Data of the partial frame is transmitted from the media source to the displayer device.
The disclosure provides a media streaming method, which includes the following operations. Media content is provided with a first resolution by a media source. A gesture command is generated by a displayer device. A partial frame is extracted with the second resolution from the media content according to the gesture command. A relative location of the partial frame in the media content being determined by the gesture command. The partial frame is transmitted from the media source to the displayer device.
The disclosure provides a non-transitory computer readable storage medium with a computer program to execute a method. The method includes the following operations. Media content is provided with a first resolution by a media source. A gesture command is received from a displayer device. A partial frame is extracted with the second resolution from the media content according to the gesture command. A relative location of the partial frame in the media content being determined by the gesture command. The partial frame is transmitted to the displayer device.
It is to be understood that both the foregoing general description and the following detailed description are by examples, and are intended to provide further explanation of the invention as claimed.
The disclosure can be more fully understood by reading the following detailed description of the embodiment, with reference made to the accompanying drawings as follows:
Reference will now be made in detail to the present embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
Reference is made to
The displayer device 120 includes a display panel 122, a gesture input module 124 and a communication module 126. In some embodiments, the displayer device 120 is a mobile phone, a tablet computer, a television, a personal computer or any equivalent electronic apparatus with a displaying component.
In some embodiments, the media source 140 can be a digital camera, a versatile camera, a sport camera, a 360-degree sphere camera or a combination of multiple cameras. In these cases, the media source 140 includes a camera module 142, a control module 144 and a communication module 146 as shown in
In some other embodiments, the media source 140 is a video storage database, and a storage unit (not shown in figures) of the media source 140 stores video files with the first resolution.
In other words, the media source 140 is configured to provide a media content with the first resolution by recording a streaming video or providing an existed video file.
The communication module 126 of the displayer device 120 and the communication module 146 of the media source 140 are communicatively connected to each other, such that data or information can be exchanged between the displayer device 120 and the media source 140. In some embodiments, the communication module 126 and the communication module 146 are local communication modules (e.g., WiFi direct, Zigbee, Bluetooth, etc) and the displayer device 120 and the media source 140 are wireless connected in a direct way. In some embodiments, the communication module 126 and the communication module 146 are telecommunication modules (e.g., WiFi, 3G, 4G, LTE etc) and the displayer device 120 and the media source 140 are wireless connected over a telecommunication network.
In some embodiments, the control module 144 of the media source 140 is a processor, a central processing unit, a graphic processing unit or any equivalent circuit component.
Reference is also made to
In some embodiments, the media source 140 is capable of recording a video or a streaming video as the media content MC. The media content MC is provided with the first resolution RES1, as shown in
It is noted that the second resolution and the first resolution are not limited to this specific values (1920×1080 and 3840×2160). These values are mentioned for demonstration. The first resolution RES1 is higher than the second resolution RES2, and any combination of the first resolution RES1 and the second resolution RES2 are suitable in this disclosure.
In a conventional way of streaming video broadcasting, raw data (the full frame) of the streaming video recorded or stored by the media source will be transmitted from the media source to the displayer device. It requires a large transmission bandwidth to transmit these data. The displayer device must receive, decode and converts the raw data of the full frame into a suitable format to be displayed on the displayer device. Bandwidth, time, computation resource is wasted to transmit and process these raw data in the conventional way.
As shown in
Reference is also made to
In some embodiments, the gesture input module 124 of the displayer device 120 includes at least one of a touch panel, a motion sensor (a gyro sensor, a g-sensor, an e-compass, etc), an eye-tracking unit and a microphone for generating the gesture command GCMD.
In some embodiments, the gesture input module 124 includes the touch panel, such that the user can slide on the touch panel to a right side, a left side, a top side or a bottom side, and correspondingly the gesture command GCMD will be generated. The gesture command GCMD is sent from the displayer device 120 to the media source 140. In response to the gesture command GCMD, the control module 144 is configured to extract the partial frame PF from the media content MC according to the gesture command GCMD, such that the gesture command GCMD triggers the control module 144 to adjust the partial frame PF move to the right side, the left side, the top side and the bottom side in the full frame of the media content MC.
In some embodiments, the gesture input module 124 includes the motion sensor (a gyro sensor, a g-sensor, an e-compass, etc), such that the user can rotate, shift, move or flip the displayer device 120 to different directions and the motion sensor can detect a movement of the displayer device 120, and correspondingly generate the gesture command GCMD. The gesture command GCMD is sent from the displayer device 120 to the media source 140. In response to the gesture command GCMD, the control module 144 is configured to adjust the partial frame PF, which is moved to a direction corresponding to the movement indicated by the gesture command GCMD. For illustration, the displayer device 120 is rotated to the right side, and then the partial frame PF is moved to the right side in the full frame of the media content MC.
In some embodiments, the gesture input module 124 includes the eye-tracking unit. For illustration, the eye-tracking unit is realized by a front camera of the displayer device 120, and the front camera is activated to capture and track the eye-balls of the user. When the user moves their eyes to stare at a specific point and the eye-tracking unit can track the eyes, and correspondingly generates the gesture command GCMD. The gesture command GCMD is sent from the displayer device 120 to the media source 140. In response to the gesture command GCMD, the control module 144 is configured to move a center of the partial frame PF toward where the eyes stare.
In some embodiments, the gesture input module 124 includes the microphone, which is utilized to record a voice-based order as the gesture command GCMD. The gesture command GCMD is sent from the displayer device 120 to the media source 140. In response to the gesture command GCMD the control module 144 adjusts the partial frame PF according to the voice-based order.
Based on aforesaid embodiments, a relative location of the partial frame PF in the media content MC is determined by the gesture command GCMD, Operation S206 is performed to extract the partial frame PF with the second resolution RES2 from the full frame of the media content MC according to the gesture command GCMD. It is noted that the partial frame PF is dynamically changed over time according to the gesture command GCMD.
After the partial frame PF is extracted by the control module 144, operation 5208 is performed to transmit data of the partial frame PF from the media source 140 to the displayer device 120. Operation 5210 is performed to display the data of the partial frame PF on the display panel 122 of the displayer device 120.
In some embodiments, when the media content MC is a static image, the data of the partial frame PF includes a portion of the static image. When the media content. MC is a video or a streaming video, the data of the partial frame PF includes a series of partial frames PF of the video or the streaming video over time, and the locations of these partial frames PF are dynamically adjusted according to the gesture command GCMD.
Once the gesture command GCMD is sent from the displayer device 120 to the media source 140, the control module 144 of the media source 140 is configured to adjust the partial frame PF to the position ordered by the gesture command GCMD, and the re-allocated partial frame PF will be transmitted in the streaming video to the displayer device 120 in real time. Therefore, the observer can always observe the scene of his interests on the displayer device 120.
In order to elevate the reaction speed of the partial frame PF adjustment, a surrounding area adjacent to the current partial frame PF is regarded as a buffer area BA (as shown in
Reference is made to
The displayer device 120 is suitable to display with a second resolution RES2, e.g., FHD (1920×1080). The media source 140, for illustration, is a 360-degree sphere camera, a panorama camera or a combination of multiple cameras for recording a full frame with the first resolution. The media source 140 is capable of recording a media content MC with a first resolution RES1, e.g., 4K2K (3840×2160). In this embodiment, the media source is able capture the scene along 360-degree visual degrees. The first resolution RES1 recorded by the media source is higher than the second resolution RES2 displayed on the displayer device 120. However, the first resolution RES1 and the second resolution RES2 are not limited to specific values (3840×2160 and 1920×1080).
In some other embodiments, the media source 140 can also be a video storage database, which stores video files with the first resolution RES1.
In some embodiments, a partial frame PF with the second resolution RES2 is extracted from the full frame of the media content MC with the first resolution RES1 according to a gesture command GCMG, and the extracted partial frame PF is transmitted from the media source 140 to the displayer device 120.
In some embodiments, the displayer device 120 includes the gesture input module 124 (referring to
In some embodiments, the gesture input module 124 includes the eye-tracking unit. For illustration, the eye-tracking unit is realized by a front camera of the displayer device 120, and the front camera is activated to capture and track the eye-balls of the user. When the user moves their eyes to stare at a specific point and the eye-tracking unit can track the eyes, and correspondingly generates the gesture command GCMD. The gesture command GCMD is sent from the displayer device 120 to the media source 140. In response to the gesture command GCMD, the control module 144 is configured to move a center of the partial frame PF toward where the eyes stare. In some embodiments, the motion sensor of the gesture input module 124 is able to detect that that user lifts, rotates or bows his/her head as well as the user steps forward, backward, to the right side or to the left side. In some embodiments, the gesture command GCMD is generated by a combination of a detection result of the eye-tracking unit and a detection result of the motion sensor. For example, the user stepping three steps forward, lifting up his head and rolling his eyes to the right side is detected by the motion sensor and the eye-tracking unit and integrated as the gesture command GCMD.
Further details about how to generate the gesture command GCMD by the motion sensor, the eye-tracking unit or the microphone are disclosed in aforesaid embodiments, and not to be repeated here.
Once the gesture command GCMD is sent from the displayer device 120 to the media source 140, the media source 140 can adjust the partial frame PF to the position ordered by the gesture command GCMD, and the re-allocated partial frame PF will be transmitted in the streaming video to the displayer device 120 in real time. Therefore, the observer can always observe the scene of his interests on the displayer device 120.
It is noted that, the displayer device 120 as illustrated in
In an embodiment, the partial frame PF displayed on the display panel 122 covers all visible angles of the observer wearing the displayer device 120. In response to the observer move this head or rotate his body, the partial frame PF will be moved or adjusted (e.g., enlarge, trim) on the whole sphere-shaped frame recorded or stored by the media source 140, such that the observer feel like being in the scene corresponding to the media source 140.
In order to elevate the reaction speed of the partial frame PF adjustment, surrounding area adjacent to the current partial frame PF is regarded as buffer area BA. In some embodiments, the pixel data in the buffer area BA will be transmitted from the media source 140 to the displayer device 120 along with the partial frame PF.
In some embodiments, the partial frame PF is real-time adjusted according to the gesture command GCMD, and the partial frame PF is streaming transmitted to the displayer device 120 immediately. This real-time experience is suitable to be utilized on high-end devices with powerful processors. For example, the partial frame PF is adjusted according to a head movement of the user detected by the motion sensor, a body movement (e.g., stepping forward, backward, to the right or to the left) of the user detected by the motion sensor, an eye movement (e.g., a staring position of user's eye) detected by the eye-tracking unit, a voice command detected by the microphone, or a combination of at least two of aforesaid detection results.
In some practical applications of some low-end devices with limited computation ability, adjusting the partial frame dynamically will consume too much electricity or overload their processors. On these low-end devices, the full frame of the media contents on the media source 140 is divided into several partial frame files, Reference is made to
As illustrated in
In response to the gesture command GCMD, one of the partial frame files is selected by the control module 144 according to the gesture command GCMD, the selected one of the partial frame files PF1-PF9 is transmitted from the media source 140 to the displayer device 120, such that the low end device is able to perform the media streaming method 200 in this disclosure. These partial frames PF1-PF9 is located side-by-side (as shown in
In the embodiments illustrated in
00611 As shown in
As shown in
For illustration, there are at least one radiation indicator 160 implemented at a fixed position in the space SPC. The positioning module 128 includes a plurality of radiation sensors disposed on different spots on the virtual reality headset. The radiation sensors are gapped from each others. The radiation sensors will receive a radiation from the radiation indicator 160 at different time stamps. The positioning module 128 is able to calculate the spatial relationship SR of the virtual reality headset within the space SPC according to the time stamps marked by different radiation sensors and the gapping distances between the radiation sensors. Aforesaid embodiment of the positioning module 128 is for illustration. In practices, a distance sensor, a GPS receiver or any equivalent positioning circuit can be utilized as the positioning module 128.
Reference is also made to
Operation 5302 is performed to provide media contents by the camera devices 140a-140d of the media source 140. The camera devices 140a-140d is located at different position in the scene SCN.
Operation S303 is performed to sense the spatial relationship SR of the displayer device 120 in the space SPC by the positioning module 128 as disclosed above.
Operation S304 is performed to generate a gesture command GCMD by the gesture input module 124 of displayer device 120. Details about how to generate the gesture command GCMD have been disclosed in aforesaid embodiments.
The spatial relationship SR and the gesture command GCMD are sent to the media source 140. A controller (not shown in figures) of the media source 140 or one coordinator of the camera devices 140a-140d receives the spatial relationship SR and the gesture command GCMD. Operation S306a is performed to select one of the camera devices 140a-140d according to the spatial relationship SR. The one of the camera devices 140a-140d in the scene SCN is selected to match the relationship SR of the displayer device 120 in the space SPC. If the displayer device 120 is moved to the front end on the right side of the space SPC, the camera device 140b is selected. If the displayer device 120 is moved to the near end on the left side of the space SPC, the camera device 140c is selected. The gesture command GCMD processed by the selected one of the camera devices 140a-140d.
Operation S306b is performed to extract the partial frame PF according to the gesture command GCMD. Operation S308 is performed to transmit the partial frame PF from the selected one of the camera devices 140a-140d to the displayer device 120, e.g., the virtual reality headset. Operation S310 is performed to display the partial frame PF.
Reference is also made to
Operation 5402 is performed to provide a media content MC with a first resolution RES1. Operation 5404 is performed to receive a gesture command GCMD from a displayer device 120. Operation 5406 is performed to extract a partial frame PF with the second resolution RES2 from the media content MC according to the gesture command GCMD. A relative location of the partial frame PF in the media content MC is determined by the gesture command. The second resolution RES2 is lower than the first resolution RES1. Operation 5408 is performed to transmit the partial frame PF to the displayer device 120.
Based on aforesaid embodiments, the media source in the media streaming system is able to transmit the partial frame with a specific resolution corresponding to the displayer device. Therefore, the bandwidth will not be wasted on transmitting some extra data which are not interested by the user of the displayer device. Once the user of the displayer device conducts a gesture input, the partial frame is adjusted accordingly.
Although the present invention has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.
It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.
This application claims the priority benefit of U.S. Provisional Application Ser. No. 62/128,520, filed Mar. 5, 2015, which is herein incorporated by reference.
Number | Date | Country | |
---|---|---|---|
62128520 | Mar 2015 | US |