Patent Application
20180115768
The invention relates to a system for releasing a stereoscopic video film, wherein the system has a data processing unit, which is configured to receive and to process a monoscopic video film and to release the stereoscopic video film, wherein the monoscopic video film has been recorded using a video recording device having only a single objective.
The invention further relates to a method for generating and replaying a stereoscopic video film from a monoscopic video film recorded using a video recording device having only a single objective.
According to prior art, the focus of the state-of-the-art display technology used, e.g., with TV sets, computer screens or portable smart phones or tablet computers, is placed on the two-dimensional or monoscopic presentation. Human vision, however, is based on spatial or stereoscopic vision. In order to replay stereoscopic images, there is made use of stereovision, this is image pairs having depth impression, which are composed of respectively one image for each of the two eyes, and of motion parallax, which enables the presentation of the position of various objects in space by means of image sequences of a moving viewer. Stereoscopic images or video films, hence, are composed of two two-dimensional images, respectively one for each of the two eyes of the viewer. The human brain receives these two different images, generating the spatial structure of the image or the video film therefrom.
The document US 2008/0080852 A1 discloses a system and a method for generating stereoscopic images. The system therein uses a camera in order to produce several multi-focus recordings of an object and to generate a stereoscopic image therefrom. In this way, by means of a data processing unit and a complex algorithm, there is calculated a combined depth impression from the multi-focus recordings. From the depth impressions of various recordings, there is identified, by means of a further complex algorithm, a single-focus image. By means of the so-called “depth-based rendering”, there is finally generated a stereoscopic image, which is composed of an image for the left eye and an image for the right eye of the viewer, which may be displayed via a stereoscopic display unit.
In the known system, it has proven to be disadvantageous that there are required several multi-focus recordings of an object for the generation of each individual stereoscopic image. This strains the data processing unit as well as a possibly required non-volatile storage likewise. In the case of a data transfer in real time, the very large amount of data may also lead to transfer problems. As there are performed several complex algorithms for the generation of a stereoscopic image, the requirements in regard to the performance of the data processing unit are very high. As the time required for the generation of the stereoscopic image is essentially determined by the time of data transfer as well as the performance and calculation of the algorithms, this may be too long for a practical application.
In addition, it has proven to be disadvantageous that the method described in the preceding paragraph has to be performed for every individual frame of the video film in order to generate a stereoscopic video film. This will inevitably lead to even bigger drawbacks in regard to the performance of the data processing unit as well as for the time required for the generation of the stereoscopic video.
The invention is based on the task to provide a system and an associated method for generating and releasing stereoscopic video films, in which the preceding disadvantages will not occur and in which the requirements in regard to the performance of the data processing unit as well as the time required for the generation of the stereoscopic video will be significantly reduced.
According to the invention, this task is solved in a system in that the data processing unit is configured to receive and evaluate a motion information allocated to the monoscopic video film or to determine the motion information to be allocated to the monoscopic video film, which motion information characterizes a motion direction of the video recording device in regard to a filmed object, wherein the data processing unit is configured to generate the stereoscopic video film from two content-identical and temporally delayed monoscopic video films.
According to the invention, this task is solved in a method in that the following process steps are carried out:
In this way, there is obtained the advantage that the stereoscopic video film may be generated directly from the monoscopic video film, which has been recorded using the video recording device having only a single objective. For this purpose, there is not required a technically complex and cost-intensive stereoscopic recording device. The associated method for processing the stereoscopic video film from the monoscopic video film and releasing it requires, in comparison to the prior art described above, no lengthy and complicated algorithms that are to be performed by the data processing unit.
Similarly, the system according to the invention does not require multiple multi-focus recordings or depth impressions of multiple recordings. These advantages will reduce the data amount produced, the required evaluation time as well as the pertaining development time and costs for the stereoscopic video film.
The system in an embodiment according to the invention requires exclusively the image information of the monoscopic video film in order to generate and release the stereoscopic video film from the monoscopic video film. From this, it may then determine the motion information allocated to the monoscopic video film, which is composed of the motion direction of the video recording device in regard to the filmed object.
In an advantageous embodiment, already after having received the data, the system has been provided with the motion information allocated to the monoscopic video film, and the data processing unit only has to evaluate this motion information.
From the monoscopic video film and the motion information allocated, the data processing unit may now generate two content-identical but temporally delayed monoscopic video films. These will be released side-by-side as the stereoscopic video film.
In order to evaluate the motion information, the data processing unit differentiates between a first motion component of the video recording device, in the direction of the optical axis of the single objective, and a second motion component of the video recording device, transversely to the direction of the optical axis of the single objective. If during a sequence of at least two successive frames of the recorded monoscopic video film exclusively the first motion component is available, this frame of the stereoscopic video film will be identified according to a first processing. If, however, during a sequence of at least two successive frames the second motion component is available, this frame of the stereoscopic video film will be identified according to a second processing.
The two different processings are advantageous insofar as they enable the system to determine, at any point of time of the motion and in every motion direction of the video recording device, a stereoscopic effect and, hence, also the stereoscopic video film.
In this way, the data processing unit in the first processing utilizes only the left half or only the right half of the frames of the monoscopic video film for processing the stereoscopic video film. For example, if the left half of the frames has been selected, each frame of the monoscopic video film is cut to the size of the left half thereof. This size-cut monoscopic video film will then be copied, and the two content-identical films will be released side-by-side, respectively for the left eye and the right eye of the viewer, as a stereoscopic video film. In order to make possible the stereoscopic effect, the monoscopic video film of the stereoscopic video film associated with the left eye of the viewer will be delayed by a determined amount of frames per second, and the monoscopic video film of the stereoscopic video film associated with the right eye of the viewer will be released without delay.
In the second processing, the data processing unit may utilize the entire, this is uncut, frames, or it may utilize only the left or the right half of the frames according to the first processing. The second motion component hereby available, hence, corresponds to a relative motion of the video recording device towards the film object from the left to the right, for example. In order to make possible the stereoscopic effect, in the present example the monoscopic video film of the stereoscopic video film associated with the left eye of the viewer will then be delayed by a determined amount of frames per second, and the monoscopic video film of the stereoscopic video film associated with the right eye of the viewer will be released without any delay.
In an advantageous embodiment the data processing unit will utilize, as soon as during a sequence of at least two successive frames within the recorded monoscopic video film the first motion component is available, also in the second processing always only the left half or only the right half of all frames of the monoscopic video film. This has the advantage that when viewing the stereoscopic video film no sudden adaptation of the video formats within the stereoscopic display unit has to be performed, which would be disturbing for the viewer.
Thereby, the data processing unit need not select exactly the left half or exactly the right half of the frames of the monoscopic video film. Instead, the data processing unit may, on the availability of the first motion component of the video recording device in the direction of the optical axis of the single objective during a sequence of at least two successive frames, in the first processing and in the second processing select only a left-weighted partial image as the left half or only a right-weighted partial image as the right half of the frames of the monoscopic video film.
In an advantageous embodiment of the system, the motion information allocated to the monoscopic video film comprises, in addition to the motion direction of the video recording device in regard to the filmed object, a motion speed and a recording rate of the video recording device. This has the advantage that by way of this information the data processing unit may determine automatically and in real time, which part of the stereoscopic video film needs to be delayed by which amount, in order to make possible the stereoscopic effect.
In a further advantageous embodiment of the system, a user may manually control which part of the stereoscopic video film is released by the data processing unit delayed by which amount in order to make possible the stereoscopic effect in an individual way.
In an advantageous embodiment of the system, the video recording device has a 3-axis-stabilisator. The 3-axis-stabilisator serves for stabilizing the video recording device during recording of the monoscopic video film. In this way, “blurring” of the recording will be prevented and a stereoscopic effect optimally enabled.
The provision of a non-volatile storage enables storing of the monoscopic video film recorded using the video recording device. This may then be received and evaluated locally and temporally independently by the data processing unit. On the other side, the provision of a data communication unit for the wireless communication will enable that the recorded monoscopic video film, and, preferably, also the appropriately allocated motion information, may be transferred within the frame of the radio range in a locally independent way and in real time.
In an advantageous embodiment of the system, there is provided an autonomous and unmanned transport means, preferably a drone, for recording and for guiding the video recording device. In the concrete case of the drone, this has the 3-axis-stabilisator and a GPS module. In this way, the monoscopic video film recorded by the video recording device may be transferred to the data processing unit in essentially real time.
If the system in addition has a stereoscopic display unit, which is connected to the data processing unit, the stereoscopic video film identified by the data processing unit may be viewed by the viewer essentially in real time.
In an embodiment according to the invention, the stereoscopic display unit is configured as a screen of a TV set or a computer, or as a projector, and has 3D glasses, preferably virtual reality 3D glasses.
In an advantageous embodiment, the data processing unit is part of a mobile telecommunication device, for example a smart phone, or of a mobile tablet computer. In combination with the stereoscopic display unit, which is composed, for example, of a screen of the device and virtual reality 3D glasses, monoscopic video films, which have been recorded using a video recording device having only a single objective, may thus be viewed in real time.
In a further advantageous embodiment of the system, the video recording device is held in a stationary position by the viewer or the transport means, while a monoscopic 360-degrees-video film is being recorded. From the recorded monoscopic 360-degrees-video film, the data processing unit processes a stereoscopic panoramic image, which is composed of two content-identical and temporally delayed monoscopic 360-degrees-video films. This stereoscopic panoramic image may be viewed by the viewer by means of the stereoscopic display unit, which is composed of, for example, a smart phone and passive virtual reality 3D glasses.
In a further advantageous embodiment of the system, the video recording device is configured as a 360-degrees-video recording device. In this way, the 360-degrees-video recording device comprises a single objective, which covers a recording area of 360 degrees transversely to the optical axis of the single objective. The 360-degrees-video recording device comprises preferably several single objectives, preferably each with an image sensor of its own. The data processing unit processes, by way of the monoscopic video films and the allocated motion information of all single objectives, the stereoscopic video film to be released within the recorded 360-degrees-surrounding. In this way, there is processed a smooth transition between the different recording areas of the single objectives so that the viewer wearing passive or active virtual reality 3D glasses may move essentially freely in a stereoscopic virtual surrounding.
Because of the configuration of the video recording device as a 360-degrees-video recording device there is obtained the advantage that, in comparison to known 360-degrees-systems, which record the 360-degrees-surrounding in several individual images, which they later on combine, there will not be necessary such a combination of images, thus no “stitching errors”, i.e. errors at the transition between two combined images, will occur. The method for producing and releasing the stereoscopic video film is thus simplified, and the quality of the released stereoscopic video film is improved.
Further advantageous embodiments of the system according to the invention will be explained in detail by means of the figures.
At any point of time, the motion of the video recording device 4 should be in a stabilized situation in order to make possible a stereoscopic effect 12 of the stereoscopic video film 2 in an optimal way. Hence, at no point of time the recorded monoscopic video film 7 should be blurred, as then also the stereoscopic video film 2 would be released in a blurred way, thus, reducing a stereoscopic effect 12.
The monoscopic video film 7 is transferred from the video recording device 4 to the non-volatile storage 5, for example, a memory card.
The data processing unit 3, in a process step A, receives an image information 13. Process step A as well as subsequent process steps B to E are illustrated in
As a function of the identified motion information 14, the data processing unit 3 performs a first processing 15 and a second processing 16, wherein the first processing 15 is allocated to a first motion component 17 and the second processing 16 is allocated to a second motion component 18. The
The first motion component 17 is schematically illustrated in
If, during a sequence of at least two subsequently recorded images 20 of the motion information 14 allocated to the monoscopic video film 7, exclusively the first motion component 17 is available, then the portion of the second motion component 18 in the motion direction 27 of the video recording device 4 is zero and the data processing unit 3 processes the stereoscopic video film 2 according to the first processing 15.
In the process, the motion direction 27 of the video recording device 4 always relates to the image 20 recorded by the image sensor 10, which has at least one filmed object 11. If actually only one characteristic filmed object 11 is in the recorded image 20, then there has to be available at least one characteristic background in the recorded image 20 in order to process a stereoscopic frame of the stereoscopic video film 2 therefrom. In order to make use of the invention, several filmed objects 11 are advantageous.
A process step C of this first processing 15 is illustrated in
If, during the sequence of at least two successive frames 21 of the monoscopic video film 7, at least one filmed object 11 is moving, then a motion speed of the video recording device 4 is appropriately adjusted in order to prevent double or blurred stereoscopic frames of the stereoscopic video film 2 to be released. The higher the motion speed of the filmed object 11 is, the higher the motion speed of the video recording device 4 has to be.
The data processing unit 3 in process step C selects either a left half 22 or a right half 23 of the recorded frames 21 of the monoscopic video film 7 and cuts these frames 21 to the size of, for example, their left halves 22. In a next step, the left halves 22 of all these frames 21 are copied. These two sequences of content-identical and size-cut frames 21 are successively released side-by-side, respectively for a left eye 24 and a right eye 25 of a viewer 26, with delay, forming the stereoscopic video film 7. In the present example, the two monoscopic video films of the stereoscopic video film 2 to be released are formed by the left halves 22 of the frames 21 of the recorded monoscopic video film 7.
Alternatively, the data processing unit 3 in process step C selects either a left-weighted partial image 38 as the left half 22 or a right-weighted partial image 39 as the right half 23 of the frames 21 of the monoscopic video film 7 and cuts these frames 21 to the size of the, for example, left-weighted partial image 38. In the next step, the left halves 22, formed by the left-weighted partial images 38, of all these frames 21 are copied. These two sequences of content-identical and size-cut frames 21 are released successively side-by-side, respectively for a left eye 24 and a right eye 25 of a viewer 26, temporarily delayed, forming the stereoscopic video film 2.
“Left-weighted” in this case means that more than the half, i.e. at least 50.1 percent, of the area or of the pixels in the selected image cut-out, to which size there is cut, are positioned to the left of the image centre, thus, in
In a process step D, which is not depicted in
The size of the amount of delay in frames per second depends, firstly, on the motion speed of the video recording device 4 in regard to the reference object, and, secondly, how strong the stereoscopic effect 12 of the stereoscopic video film 2 is to be. In this regard, the amount of the delay is to be selected the larger the higher the motion speed of the video recording device 4 is or the stronger the desired stereoscopic effect 12 is to be, respectively. Preferably, the adjusted delay in frames per second is between a third and two thirds of the recording rate of the video recording device 4, especially preferably the half of the recording rate of the video recording device 4.
The selection by which amount in frames per second the side that is to be presented with delay is to be released may also be made manually by the viewer 26.
The first processing 15, hence, advantageously uses a “lens effect”, this is the distorted display of the filmed object 11 as a function of its position on the projected monoscopic image area in regard to the central axis of the optical lens 9. Using this lens effect enables to overcome a well-established prejudice among experts, this is that there will be no motion parallax and thus no stereoscopic effect 12 if the video recording device 4 moves exclusively in the direction of the optical axis of the single objective 8 and the optical lens 9, this is along the first motion component 17.
According to the first processing 15, hence, even on the exclusive availability of the first motion component 17, there may be generated a motion parallax and thus a “genuine” stereoscopic video film 2 having a “genuine” stereoscopic effect 12. In this way, in the process step C, the distortion caused by the optical lens 9 of the respective recorded image 20 may be evaluated by the data processing unit 3 essentially in real time. By way of this evaluation, a motion parallax is then determined by the data processing unit 3, and after the process steps C and D have been performed, a stereoscopic video film 2 having a genuine stereoscopic effect 12 will be generated. “Genuine” in this case means, for example, that the data processing unit 3, a machine or a robot recognizes where an object 11 is located in the “space” of the stereoscopic video film 2, meaning whether it is located in front of or behind another object 11. A robot, for example, an autonomous and unmanned drone, may in this way autonomously head towards objects 11 or avoid these.
Similar systems or methods for producing a stereoscopic effect on exclusive availability of a motion direction of a video recording device comparable with the first motion component 17, for example as described in the document by Zhang, X. et al: “Visual video image generation . . . ”, IEICE Trans. Inf. & Syst., Bd. E83-D, No. 6, June 2000, pages 1266-1273,XP000976220, ISSN: 0916-8532, however, do not create a “genuine” stereoscopic effect 12, as only two shifted but identical images are released for the left eye and for the right eye of a viewer. In this way, only a “simulated” and “false” stereoscopic effect is produced, as it seems as if an object in the image for the left eye is at another location than in the image for the right eye. The image for the left eye and the image for the right eye, however, are identical images, and for this reason the method according to Zhang, X. et al does not provide any depth information. For this reason, e.g., a robot cannot recognize where an object is located in the space of a stereoscopic film.
If, during a sequence of at least two subsequently recorded images 20 of the motion information 14 allocated to the monoscopic video film 7, a portion of the second motion component 18 is available, then the data processing unit 3 will process the stereoscopic video film 2 according to the second processing 16. In this context, there is always performed the second processing 16, provided the portion of the second motion component 18 in the motion direction 27 of the video recording device 4 is not zero.
The second motion component 18 is schematically illustrated in
If, during a continuous sequence of at least two successive frames 21, several filmed objects 11 move in the same or in different directions, the data processing unit 3 selects at least on reference object, by way of which the associated motion information 14 of the video recording device 4 is being determined. If, during a continuous sequence of at least two successive frames 21, the motion speed of at least one filmed object 11 changes, the motion speed of the video recording device 4 is adjusted appropriately in order to prevent double or blurred images of the stereoscopic film 2 to be released. The higher the motion speed of the film object 11 is, the higher the motion speed of the video recording device 4 has to be.
The frames 21 of the monoscopic video film 7 need not be cut into size in the second processing 16. The data processing unit 3 in process step C doubles all recorded frames 21. These two sequences of content-identical frames 21 are subsequently released side-by-side and delayed to each other, respectively for the left eye 24 and the right eye 25 of the viewer 26, forming the stereoscopic video film 2. The two monoscopic video films of the stereoscopic video film 2 to be released are then formed by the frames 21 of the recorded monoscopic video film 7.
In an advantageous variant of the system 1 according to the invention, also in the second processing 16 all frames are cut to the size of their left halves 22 or their right halves 23. In the case that the motion information 14 of the video recording device 4 determined during the monoscopic video film 7 also requires the second processing 16, it may be prevented that during the replay of the stereoscopic video film 2 the side proportions of the two monoscopic video films change respectively for the left eye 24 and for the right eye 25 of the viewer 26.
In the second processing 16, however, in contrast to the first processing 15, the delay of the monoscopic video film 7 for the left eye 24 or for the right eye 25 of the viewer 26 is determined by way of the relative motion direction 27 of the video recording device 4 to the reference object. If, for example, the video recording device 4 moves from the left to the right in regard to the reference object, then in the stereoscopic video film 2 the monoscopic video film for the left eye 24 of the viewer 26 will be released with delay and the monoscopic video film for the right eye 26 of the viewer 26 will be released without delay.
The size of the amount of the delay in frames per second also in the second processing 16 depends, firstly, on the motion speed of the video recording device 4 in regard to the reference object and, secondly, on how strong the stereoscopic effect 12 of the stereoscopic video film 2 is to be. In this regard, the amount of delay is to be selected the higher, the higher the motion speed of the video recording device 4 in regard to the reference object is or how strong the desired stereoscopic effect 12 is to be. Preferably, the adjusted delay in frames per second is between a third and two thirds of the recording rate of the video recording device 4, especially preferably the half of the recording rate of the video recording device 4.
The selection, whether the monoscopic video film is to be released with delay for the left eye 24 of the viewer 26 or whether the monoscopic video film for the right eye 25 of the viewer 26 is to be released with a delay of a determined amount in frames per second, may also be made manually by the viewer 26.
As last process step E, the stereoscopic video film 2 processed in the process steps A to D is released by the data processing unit 3 to the stereoscopic display unit 6. The stereoscopic display unit 6 is composed in the embodiment according to the invention of a screen of a TV set or computer, or of a projector, and 3D glasses, preferably virtual reality 3D glasses.
In an advantageous variant of the system 1 according to the invention, the viewer 26 integrates the telecommunication device 33 directly into virtual reality 3D glasses. The viewer 26, in this way, may view the stereoscopic video film 2, which is processed by the data processing unit 3 from the monoscopic video film 7 recorded by the video recording device 4, essentially in real time by means of the system 1 depicted in
In an advantageous variant of the embodiment of the system 1 illustrated in the
In an advantageous embodiment, the viewer 26 wears active virtual reality 3D glasses, which comprise the display and the gyroscope in a housing. In this way, the viewer 26 may view the stereoscopic panoramic image 35 also without a telecommunication device 33, if the active virtual reality 3D glasses receive the stereoscopic panoramic image 35 from the data processing unit 3 or if they comprise this data processing unit 3 in the housing.
In a further advantageous embodiment of the system 1, the video recording device 4 is configured as a 360-degrees-video recording device. Therein, the 360-degrees-video recording device comprises a single objective 8, which covers a recording area of 360 degrees in three-dimensional space, horizontally and vertically transversally to the optical axis of the single objective 8. This recording area corresponds to a spherical surface. Preferably, the 360-degrees-video recording device comprises several single objectives 8, each having an image sensor 10 of its own; especially preferably the 360-degrees-video recording device comprises at least four single objectives 8, each having an image sensor 10 of its own. The several single objectives 8 each cover a recording area of at least 360 degrees divided by the number of all single objectives 8 available horizontally and vertically transversally to the optical axis of the respective single objective 8. The video recording device 4 moves while the monoscopic video film 7, which is composed of the individual parts of the monoscopic video films recorded by the single objectives 8, is being recorded.
Because of the configuration of the video recording device 4 as a 360-degrees-video recording device, the 360-degrees-surrounding is advantageously recorded in a monoscopic video film 7, which may subsequently be released as a stereoscopic video film 2 to the stereoscopic display unit 6 essentially in real time. Compared to other known 360-degrees-systems, which record the 360-degrees-surrounding in several individual images, which are later on combined, such a combination of images is not necessary. In this way, the system 1 according to the invention enables a simplified and improved generation and release of the stereoscopic video film 2 and thus prevents the occurrence of “stitching errors”, this is errors in the transition of two combined images, which may occur with the 360-degrees-systems already known.
If, for example, there is present a configuration of the system 1 according to
If the viewer 26 wears passive or active virtual reality 3D glasses and if he glances within the virtual 360-degrees-surrounding, the data processing unit 3, according to the motion of the viewer 26, selects the stereoscopic video film 3 corresponding to this motion of the single objective 8 associated with this viewing direction. In this way, the viewer 26 may glance in an essentially free way within a stereoscopic virtual surrounding. The viewer 26 then sees, while the stereoscopic video film 2 is being released, respectively the part of the virtual sphere surface, which corresponds to his viewing direction, or to the spatial direction of the passive or active virtual reality 3D glasses 38, respectively. The viewer 26 himself “adjusts his virtual motion” to the motion direction of the video recording device 4, while the monoscopic video film 7 is being recorded.
It may be noted that the system 1 according to the invention is also suited for releasing a stereoscopic image from the monoscopic video film 8. In this way, the stereoscopic image is composed of two images that are released side-by-side (a left one for the left eye 24 of the viewer 26 and a right one for the right eye 25 of the viewer 26). The stereoscopic image is herein generated by a so-called “screenshot” from the stereoscopic video film, meaning that a determined frame of the processed stereoscopic video film 2 is released to the stereoscopic display unit 6.
| Number | Date | Country | Kind |
|---|---|---|---|
| 15164966.2 | Apr 2015 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2016/058836 | 4/21/2016 | WO | 00 |
