Patent Application
20210203842
The disclosure relates in general to a panoramic video processing method, a panoramic video processing device and a panoramic video system.
In the conventional panoramic image processing method, all of the original images are stitched to obtain one panoramic image. Then, the panoramic image is divided for displaying according to the need of the end user. However, all of the original images are needed to be stitched. The stitches are performed sequentially and cannot be performed in parallel, such that the process cannot be speeded up.
Especially, for the application of the panoramic video, if the processing speed of panoramic video is low, the frame rate of the panoramic video will be reduced, and it cannot be applied for the real time application. Therefore, how to speed up the process of the panoramic video is an important issue.
Moreover, the end users have various processing capability, and the panoramic video may be more than 8 k or 16 k. Because the bandwidth is limited, the transmission time is needed to be reduced for all users to bring a good high-resolution panoramic viewing experience.
The disclosure is directed to a panoramic video processing method, a panoramic video processing device and a panoramic video system.
According to one embodiment, a panoramic video processing method is provided. The panoramic video processing method includes the following steps. A plurality of shooting ranges of a plurality of cameras and a field of view (FOV) are received. An amount of stitches of a plurality of original images is calculated according to the shooting ranges and the field of view. An amount of a plurality of adjacent parts of the original images is corresponding to the amount of stitches. The adjacent parts of the original images are stitched to obtain a plurality of partial panoramic images.
According to another embodiment, a panoramic video processing device is provided. The panoramic video processing device includes a controlling unit and a plurality of processing units. The controlling unit is used for receiving a plurality of shooting ranges of a plurality of original images and a field of view (FOV), and calculating an amount of stitches of the original images. An amount of a plurality of adjacent parts of the original images is corresponding to the amount of stitches. The processing units are used for stitching the adjacent parts of the original images some of the original to obtain a plurality of partial panoramic images.
According to an alternative embodiment, a panoramic video system is provided. The panoramic video system includes a panoramic photography device and a panoramic video processing device. The panoramic photography device includes a plurality of cameras and a panoramic video processing device. The panoramic video processing device includes a controlling unit and a plurality of processing units. The controlling unit is used for receiving a plurality of shooting ranges of a plurality of original images and a field of view (FOV), and calculating an amount of stitches of the original images. An amount of a plurality of adjacent parts of the original images is corresponding to the amount of stitches. The processing units are used for stitching the adjacent parts of the original images to obtain a plurality of partial panoramic images.
In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments may be practiced without these specific details. In other instances, well-known structures and devices are schematically shown in order to simplify the drawing.
Several embodiments are described here to illustrate the panoramic video processing method of the present application. The processing of the panoramic video is speeded up and the transmission time of all users are reduced to bring a good high-resolution panoramic viewing experience.
Please referring to
The partial panoramic images PP1 to PP8 are transmitted via a plurality of channels CH1 to CH8 respectively. Only one of the partial panoramic images PP1 to PP8, based on a view direction VD (shown in
In
Please refer to
The panoramic photography device 100 includes the cameras CAM1 to CAMS. The amount of the cameras may be larger than or equal to 2. The panoramic video processing device 200 includes a controlling unit CU and the processing units PU1 to PU8. The amount of the processing units may be less than or equal to the amount of the cameras. Each of the controlling unit CU and the processing units PU1 to PU8 may be a processor, a circuit, a chip, a circuit board, a processing core, a program module or a storage device storing program codes. The operation of those elements is illustrated by a flowchart as below.
Please referring to
The stitch strategy creating procedure (steps S101 to S104) is illustrated first. At step S101, the panoramic photography device 100 transmits a plurality of shooting ranges CR1 to CR8 of the cameras CAM1 to CAM8. For example, please refer to
Then, in the step S102, the controlling unit CU receives the shooting ranges CR1 to CR8 and the field of view (FOV) FV of the display device U1. Different display devices may have different FOV. In general, the FOV of the head mounted display may be 80 to 110 degrees. The user may zoom in or zooms out the frame to reduce or enlarge the FOV.
Next, in step S103, the controlling unit CU calculates an amount S of stitches of the original images IM1 to IM8 according to the shooting ranges CR1 to CR8 and the field of view FV. An amount of a plurality of adjacent parts of the original images IM1 to IM8 is corresponding to the amount of stitches. If the amount S is 2, it means that two original images of two cameras which are adjacent are stitched to be a partial panoramic image; if the amount S is 3, it means that three original images of three cameras which are adjacent are stitched to be a partial panoramic image; and so on. The amount S of stitches may be temporarily stored in a storage unit. The amount S of stitches is an integer which is larger than or equal to 2. For example, in the example of
As shown in
Besides, as shown in
Moreover, if the amount S of stitches is 2, N partial panoramic images can be obtained from N original images; if the amount S of stitches is 3, Nā1 partial panoramic images can be obtained from N original images; if the amount S of stitches is 4, Nā2 partial panoramic images can be obtained from N original images, and so on.
Afterwards, in step S104, the controlling unit CU obtains a relationship TB between the partial panoramic images PP1 to PP8 and a view direction VD according to the stitching angle ranges SR1, SR2, . . . . Base on the steps S101 to S104, the stitch strategy creating procedure is completed. Taken table I as an example, the table I illustrates the relationship TB. The relationship TB can be integrated into a meta-data packet.
Next, the stitch performing procedure, i.e. the steps S105 to S110, is illustrated. In the step S105, the cameras CAM1 to CAM8 of the panoramic photography device 100 capture the original images IM1 to IM8 respectively.
Then, in step S106, the panoramic photography device 100 transmits the original images IM1 to IM8 to the panoramic video processing device 200.
Next, in step S107, the processing units PU1 to PU8 obtain the original images IM1 to IM8 and the amount S of stitches.
Then, in step S108, the processing units PU1 to PU8 stitch the adjacent parts of the original images IM1 to IM8 to obtain the partial panoramic images PP1 to PP8. In this step, the processing units PU1 to PU8 are performed in parallel to obtain the partial panoramic images PP1 to PP8 at the same time. Therefore, the time for stitching can be reduced to ā .
Next, in step S109, the processing units PU1 to PU8 perform some processes, such as coding or rending, on the partial panoramic images PP1 to PP8.
Afterwards, in the step S110, the partial panoramic images PP1 to PP8 are transmitted via broadcast or multicast for the user to download. Through the steps S105 to S110, the stitch performing procedure is completed. In the present embodiment, due to the parallel processing, the processing speed of the stitch performing procedure can be greatly improved. Therefore, the frame rate of the panoramic video is not reduced and can be applied for the real time application.
Next, the download procedure, i.e. the steps S201 to S203, is illustrated. In step S201, the display device U1 detects the view direction VD of the user. For example, the display device U1 may equipped with a gyroscope, a compass, a G-sensor, an outside-in tracking system, or an inside-out tracking system to detect the view direction VD.
Then, in step S202, the relationship TB is looked up to establish a connection with one of the channels CH1 to CH8 according to the view direction VD, and one of the partial panoramic images PP1 to PP8 corresponding the view direction VD is downloaded.
Next, in step S203, the display device U1 displays the downloaded partial panoramic image (one of the partial panoramic images PP1 to PP8). Through the steps S201 to S203, the download procedure is completed. In the present embodiment, due to the partial panoramic images PP1 to PP8, each user does not download whole of the panoramic image P0. The bandwidth is not occupied to much and the transmission time is reduced to bring a good high-resolution panoramic viewing experience.
According to the embodiments described above, the parallel processing technology and broadcast/multicast technology are used to improve the processing speed of the panoramic video and reduce the transmission time for all users. As such, in the 360-degrees Virtual Reality Ultra High Resolution (16K) application, there is no encoding/decoding/display latency. The head mounted display can achieve 4K display capability and display 16K picture.
It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments. It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.
