Patent Application
20150312547
The present invention refers to an apparatus, a method and a software product for generating a video stream.
The present invention also refers to an apparatus, a method and a software product for rebuilding a video stream.
The present invention also refers to an electromagnetic signal incorporating a video stream generated by said apparatus, method or software product for generating a video stream.
The diffusion of the 3D technology, after having been an exclusive protagonist in the cinemas, is gaining more and more ground also in the TV field as stereoscopic HD television based on the use of a couple of views, the right one and the left one, depicting what respectively perceived by the right eye and by the left eye in the binocular vision.
The simplest technique for the distribution of the stereoscopic signal consists in the transmission of both the right view, and the left one (the so-called simulcast), however so doubling the used band. Such an increase of the used band is at least unwanted, as presently the frequency resources for TV transmission are now very limited.
To make up for such a drawback it is possible to resort to alternative approaches among which the 2D plus stereoscopic metadata (2D plus stereoscopic metadata), allowing a Full HD display of the stereoscopic video, wherein “2D” intends to be one of the two views, and “stereoscopic metadata” intend the additional information allowing to depict the other view exploiting the correlation between the two views.
The definition 2D plus stereoscopic metadata includes different approaches, each of them using different metadata to obtain the view which is not integrally transmitted:
There are many approaches for the transmission of the TV 3D which are different in what regards the bit-rate necessary for the view encoding; the best known of them are based on spatial arrangements of the two stereo views (Side-by-Side, Top-and-Bottom formats, etc.), use of the correlation between the views (Multiview Video Coding or MVC Stereo High Profile), use of the depth map and of the occlusion map.
Side-by-Side and Top-and-Bottom allow to reuse the existing infrastructure HDTV, but do not allow a Full HD display for both eyes, because the two views are spatially arranged into a frame HDTV, with consequent horizontal or vertical halving of the native resolution.
The Multiview Video Coding (MVC) allows the use of a Full HD content for both eyes as the stereoscopic pair is transmitted using the correlation among the views, but does not allow a significant compression efficiency with respect to the simulcast.
The approach 2D plus Depth, as said, provides the transmission of a view (2D signal, for instance the left view) and of the depth map computed starting from the two views.
The depth map depicts, in grey scale, the information providing the disparity between the two views, that is the distance in which the pixels of the regarded view (to be rebuilt on regeneration side) are located with respect to the reference one (i.e. that integrally transmitted). The conversion between disparity and depth occurs by means of an appropriate transform depending on the intrinsic parameters of the video-cameras, on the distance between the two video cameras, on the distance of the farthest scene plans and the nearest ones to the video-cameras. The rebuilding of one of the two views can occur using many techniques:
1) by moving the pixels of even value to the oddness, obtained from the depth map, starting from the view taken as reference;
2) by using so-called algorithms 3D warping and reverse 3D warping for the projection of the real space coordinates of the pixels on those of the view synthesized through appropriate matrices. Said matrices are got from the parameters defining the position and the orientation of the camera with respect to the real system. The quality of the rebuilt view is closely related to the quality of the depth map.
The so rebuilt view is however not complete because of the so-called occlusions, that is parts of the background and/or objects present only in one of the two views. To overcome this problem then it becomes necessary to transmit also a map of the occlusions, in order to cover the not rebuilt pixels. Using this method there is however a resulting overall band occupation comprised between 180% and 220% with respect to a 2D Full HD stream with the same resolution and frame frequency, making very heavy the transmission and/or the storage of a 3D signal implemented in this way.
It is an object of the present invention to depict a method, an apparatus and a software product solving some problems of the prior art.
Specifically, it is an object of the invention to provide a method, an apparatus and a software product allowing to generate a stereoscopic video stream, as well as an apparatus, a method and a software product allowing to rebuild a stereoscopic video stream, capable to restrict in significant way the band occupation in a transmit/receive step, and the occupation of memory places in storing step, without in the meantime reducing the quality of the same video stream.
A further object of the invention is to indicate a method, an apparatus and a processing software of a three dimensional video stream which is highly scalable with respect to the required complexity especially in the rebuilding side, whereby it is easily possible to rebuild a same stereoscopic 3D video stream entering in 2D mode with a simple change to the operating mode of the rebuilding apparatus.
A further object of the invention is to indicate a method, an apparatus and a generating and rebuilding software of a three dimensional video stream which is easily extendible from the case of a stereoscopic video stream, comprising two views, to so-called “multi-view” systems employing a view number greater than two.
These and also other objects are substantially met by a method, an apparatus and a generation software product, as well as by a method, an apparatus and a rebuilding software product, according with what disclosed in the enclosed claims.
Further objects and advantages of the present invention will further appear from the detailed description of a preferred and not exclusive embodiment of the same invention.
Such a description is later provided with reference to the enclosed drawings, having they-selves just exemplary purpose and so not restrictive, wherein:
a shows a block diagram of a generating apparatus according to the present invention;
b shows a block diagram of a rebuilding apparatus according to the present invention;
a shows a block diagram of a possible embodiment of a rebuilding apparatus according to the present invention;
b-4c schematically show two operating modes of the apparatus of
a-5b show flow charts representing the rebuilding method according to the present invention;
a-10d schematically show image synthesis steps used in an apparatus and in a method according to the invention;
a-11d schematically show image rebuilding steps used in an apparatus and in a method according to the invention.
Referring to the enclosed
It is to be noted that, in the present context, the terms “image” and “frame” will be considered as synonymous between them and will be able to be used in an interchangeable way, while the term “map” indicates without distinction a disparity map or a depth map.
The generation apparatus 1 (
The first input sequence L and the second input sequence R, when reproduced through a proper playing apparatus (as, for instance, a TV set with 3D prearrangement) allow to display contents in such a way that the user has the feeling of the depth at which the various depicted elements stay, thereby giving the feeling of a 3D depiction of said contents.
As an example, the first input sequence L can be related to the left view of the stereoscopic stream, while the second input sequence R can be related to the right view of such a flow. It is however to be noted that the invention can be carried out also in the diametrically opposed situation, in which the first sequence L is depicting the right view and the second sequence R is depicting the left view.
In particular, the communication interface 10 receives as input one or more images of the first input sequence L, and one or more images of the second input sequence R.
The images of the first input sequence L can be stored, for instance, in a first memory area M1; the images of the second input sequence R can be stored, for instance, in a second memory area M2.
The first and/or second memory areas M1, M2 can belong to a non-volatile memory, wherein the first and/or the second input sequence L, R can be stored in a stable way, till a subsequent erase command, or to a volatile memory, wherein the first and/or the second input sequence L, R, or their parts, are stored just for the time strictly necessary for processing thereof.
The communication interface 10 receives as input one or more maps allowing, starting from one or more of the images of the first input sequence L, to rebuild substantially correspondent images of the second input sequence R.
In practice, by combining among them a determined image of the first input sequence L and one corresponding of said maps, it is possible to substantially obtain the respective image belonging to the second input sequence R, that is the image of the second input sequence R associated to the same time reference of the image of the first input sequence L.
As will be clearer later on, such a rebuilding can be not complete because of the so-called “occlusions”.
The communication interface 10 receives in the input one or more D maps allowing, starting from one or more of the images of the second input sequence R, and respectively from the first input sequence L, to substantially rebuild corresponding images of the first input sequence L, and respectively of the second input sequence R. Also these rebuilds could be not complete, because of the already above explained reasons.
The maps can be depth maps or disparity maps. Said maps can be produced through per se known techniques, which will not be disclosed herein in detail. It is however to be noted that, for the aims of the implementation of the present invention, both depth maps and disparity maps can be used. This observation refers not only to the above mentioned maps, but also to other “maps” which are mentioned in the present description and in the following claims.
In an embodiment, the apparatus 1 can be arranged to receive as input just the first and the second input sequence L, R, and to autonomously compute the necessary maps. The suitable processing module generating the maps in this embodiment, provides to supply as input of the communication interface 10 the generated maps, so that they can be processed by the processing unit 11 for the creation of the encoded video stream CVS. Advantageously the maps can be stored in a proper memory register, belonging to a volatile memory or a non-volatile memory.
The generation apparatus 1 further comprises a processing unit 11 operatively associated to the communication interface 10 and, preferably, to the mentioned first and second memory areas M1, M2.
The processing unit 11 provides to generate as output an encoded video stream CVS which, although presents on the whole a significantly lesser memory occupation with respect to the input stereoscopic stream VIN and so can be transmitted using lesser band resources, contains all the essential information in order that the contents of the essential stream can be faithfully reproduced.
Going in more detail, the processing unit 11 is configured to determine at least a first image L1 of the first input sequence L.
Preferably such a first image L1 is selected among the images belonging to the first input sequence L.
It is to be noted that, in the present context and in the following claims, with “first image” it is not necessarily designed the initial image of the first input sequence L, but simply one of the images received by processing unit 11 through the communication interface 10.
The same observation also holds for both the “first image” of the second input sequence R, and the term “second image”, which will be later mentioned.
The processing unit 11 is further configured to determine a first differential map D1.
The first map D1 can be selected among the maps received as input through the cited communication interface 10.
The first map D1 is such that, by combining the same with the first image L1 of the first input sequence L, it is possible to substantially rebuild a first image R1 of the second input sequence R. As said, such a rebuilding can be not complete.
The processing unit 11 is further configured to determine a second image R2 of the second input sequence R.
The second image R2 of the second input sequence can be selected among the images of the second input sequence R.
The processing unit 11 is further configured to determine a second map D2.
The second map D2 (which can be, for instance, a depth map or a disparity map) is such that, by combining the same with the second image R2 of the second input sequence R it can be substantially possible to rebuild a second image L2 of the first input sequence L.
Preferably, the second image L2 and the second image R2 are associated to a same time reference.
Preferably, the second images L2, R2 are time following and time adjacent, respectively, to the first images L1, R1.
The processing unit 11 so can prepare as output the encoded video stream CVS; the latter comprises at least: the first image L1 of the first input sequence L, the first map D1, the second image R2 of the second input sequence R and the second map D2.
In this way, the encoded video stream CVS can allow a use of the initial stereoscopic video stream VIN so avoiding that the last one is stored and/or integrally transmitted.
The encoded video stream CVS can be broadcasted, for instance for a use through TV sets able to decode the encoded video stream CVS, or associated to proper external decoders for said decoding.
The encoded video stream CVS can also be recorded on a proper magnetic and/or optical and/or electronic support. Such a support can then be associated to a decoder to allow a use of the contents of the stored video stream.
As above mentioned, not always by using the first image L1 of the first input sequence L and the respective map D1 (or, similarly, the second image R2 of the second input sequence R and the respective second map D2) it is possible to completely rebuild the first image R1 of the second input sequence R (or the second image L2 of the first input sequence L).
So-called “occlusions” can occur, i.e. pixels or pixel groups, representing background parts and/or object parts, present just in one of the two views. These pixels are present in particular on the edges of the objects of the scene, particularly in presence of relatively near objects, superimposed and moving. In other words, the first image R1 of the second input sequence R can contain some pixels and/or areas which do not have any correspondence in the first image L1 of first input sequence L and consequently which cannot be rebuilt using just L1 and D1.
The same applies to the rebuilding of the second image L2 of the first input sequence L starting from R2 and D2.
To avoid this drawback, which practically is an incomplete reproduction of some images, and a corresponding reduction of the use quality of the video stream, it is provided that the processing unit 11 identifies one or more occluded pixels, as a function of the first image L1 of the first input sequence L, and the corresponding first map D1. So descriptive data of one or more replacement pixels to substitute said one or more occluded pixels are determined as a function of one or more images of the second input sequence R.
In an embodiment, to locate the occluded pixels, the processing unit 11 can combine the first image L1 of the first input sequence L with the first map D1, obtaining a respective combination. By comparing such a combination, which can be similar but not completely identical to the first image R1 of the second input sequence R, exactly with the first image R1 of the second input sequence R, the processing unit 11 locates the occluded pixels and the related replacement pixels.
In an embodiment, the descriptive data of the replacement pixels can be determined in function of one or more images belonging to the second input sequence R different from (i.e. other with respect to) said first image R1.
Preferably, said descriptive data can be determined in function of an image time adjacent and time preceding said first image R1.
As an example, in the second input sequence R the image immediately preceding the first image R1 can be used.
Preferably specific replacement pixels are located which cannot be synthesized as a function of just one preceding image and/or a subsequent image to said corresponding image R1 belonging to said second input sequence R. In practice the processing unit 11 provides to locate those occluded pixels which cannot be rebuilt by the decoder on the basis of the single preceding image and/or of the only subsequent image with respect to the image to be rebuilt.
The processing unit 11 so determines the main data descriptive of said determined replacement pixels.
The main data are advantageously inserted in the encoded video stream CVS, so that they can be used in the decoding step.
In an embodiment, the main data can comprise motion vectors associated to the occluded pixels.
Said motion vectors are per se known and included in some coding standards (for instance H264/AVC). It is however envisaged that proper parameters can be used also not coinciding with those defined in the current video coding standards which describe the movement of the occluded pixels to improve their rebuilding at the encoding time.
As an example, the mentioned describing parameters of the replacement pixels can be obtained by means of a view synthesis system using the only information present in the depth map to locate the occluded areas.
As above mentioned, the first image R1 of the first input sequence R can be used to assess the value of the occluded pixels: a part of these pixels can be obtained from the preceding image belonging to the same sequence R, while the remaining ones, particularly regarding movement areas, are sought between the preceding images and the following images (with respect to the first image R1) belonging to the second sequence R.
As regards the image (of either the first or the second input sequence L, R) which has to be encoded and which, according to the time sequence with which the images are arranged, is preceding all the other ones, there will not be images preceding the same which can be used for the assessment of the occlusions and for determining the relative replacement pixels. In such a particular case, it will be advantageously used one or more of the following images. In an embodiment, it will be used the thereto adjacent image and that immediately following in the time.
As will be later clearer, the replacement pixels determinable through images preceding and/or following that to be encoded can be computed by the decoder without that in the encoded video stream CVS are inserted specific data identifying and describing said replacement pixels.
What can be advantageously inserted in the encoded video stream CVS are the mentioned main data, allowing to determine the replacement pixels not determinable through the only preceding image and/or the only image following that in question.
It is to be noted that, as the occluded pixels which cannot be obtained from the preceding (o the following) frame are typically just a small percentage of the total, the information overhead to send due to the presence of the movement vectors is normally negligible or however very reduced with respect to that necessary in case of absence of this feature.
What above described with reference to the identification of the information useful to the rebuilding of the first image R1 of the second input sequence R applies in a fully similar way to the synthesis of the second image L2 of the first input sequence L.
It is to be noted that what above described referring to the only first and second images L1, L2, R1, R2 can be in practice embodied with a well higher image number so to make video streams of short films, movies, and so on.
The communication interface 10 in fact can receive an input sequence L comprising a well higher image number. In particular, the first input sequence L comprises a first plurality of images Li each one of them being associated to a respective first time reference TLi, identifying the position inside the first input sequence L.
The first plurality of images Li comprises the above mentioned first image L1.
The first input sequence L can further comprise images alternated with the images of said first plurality Li, that is images associated to time references (which, as will be later clearer, will be identified as TRi) alternated with the first time references TLi.
The second input sequence R received by the communication interface 10 comprises a second plurality of images Ri each one associated with a respective second time reference TRi, identifying the location inside the second input sequence R.
The second plurality of images Ri comprises the above mentioned second image R2.
The second input sequence R can further comprise images alternated with the images of said second plurality Ri, that is images associated to time references alternated with to the second time references TRi. In practice, said further images, of the second input sequence R, are associated to the above mentioned first time references TLi.
Advantageously the first time references TLi are timely alternated to the second time references Tri. In other terms, in the preferred embodiment, the images of the first plurality of images Li are time alternated with the images of the second plurality Ri.
Preferably the first input sequence L is stored in the first memory area M1, and the second input sequence R is stored in the second memory area M2.
The communication interface 10 is configured to receive a first plurality of maps D1i belonging the input stream Vin.
The maps of the first plurality of maps D1i are preferably depth maps or disparity maps.
The maps of the first plurality of maps D1i are such that, by combining each one of said maps with the respective image of said first plurality of images Li, the corresponding image of the second input sequence R is substantially obtained (except for occlusions or other like phenomena).
The first plurality of maps D1i comprises the above mentioned first map D1.
The communication interface 10 is further configured to receive a second plurality of maps D2i forming part of the input stream Vin.
The maps of the second plurality of maps D2i are preferably depth maps or disparity maps.
The maps of the second plurality of maps D2i are such that, by combining each one of said maps with the respective image of said second plurality Ri, the corresponding image of the first input sequence L is substantially obtained (except for occlusions or other like phenomena).
The second plurality of maps D2i comprises the above mentioned second map D2.
The processing unit 11 is then configured to operate on the images of the first plurality Li, on the images of the second plurality Ri and on the respective maps D1i, D2i according to the same above described technique with reference to the first image L1 of the first input sequence L, to the first map D1, to the second image R2 of the second input sequence R and to the second map D2.
The processing unit 11 is configured for inserting in the encoded video stream CVS the first plurality of images Li, the first plurality of maps D1i, the second plurality of images Ri and the second plurality D2i of maps.
The encoded video stream CVS so contains, for each one time moment, an image of the first plurality Li associated to the respective first map D1i (first time references TLi), or an image of the second plurality Ri associated to the respective second map D2i (second time references TRi).
Preferably, the processing unit 11 is configured for associating, to one or more maps of the first and/or the second plurality D1i, D2i, descriptive data of replacement pixels. Said descriptive data can be advantageously determined through the above described techniques.
In particular, the descriptive data can consist of the above mentioned main data.
The input interface 10, the processing unit 11 and preferably the first and/or the second memory areas M1, M2 form a generator, making part of the generating apparatus 1 and indicated by the reference number 3 in
Advantageously it is provided a pre-processing of the images initially provided to the generating apparatus 1, particularly in the case in which the two views depicted by the two input sequences present remarkable differences in terms of colorimetry and/or brightness. In such a circumstance, in fact, the display of the decoded video stream could be troublesome owing to alternation of images with features so different from each other.
To this aim the generating apparatus 1 can optionally be provided with a pre-processing module 12, pre-arranged upstream of the processing unit 11.
The pre-processing module 12 operates based on the first input sequence L and on the initial sequence R0.
The initial sequence R0 contains a plurality of images R0i representative of the second view of the input stereoscopic stream; each of such images is associated with a corresponding image of the first input sequence L.
In practice the first input sequence L and the initial sequence R0 are the sequences originally received by the apparatus 1.
The pre-processing module 12 provides to compare one or more images R0i of the initial sequence R0 with the corresponding images of the first input sequence L, namely with images of the first input sequence L associated to the same time references of the images R0i of the initial sequence R0.
In function of said comparisons, the images forming part of the above mentioned second input sequence R are obtained.
In greater detail, as a function of each comparison a second corresponding map Di is made; then, by combining the images of the first input sequence L with the respective maps Di, the images of the second input sequence R are rebuilt.
Preferably, in the image rebuilding of the second input sequence R also the possible occlusions are taken into account, preventing a complete rebuilding starting from the images of the first input sequence L and the related maps D1. The replacement pixels to be used for said occlusions can be decisive, according to per se known techniques, as a function of initial sequence R0. As an example, the above disclosed algorithms can be used with reference to the generation of the encoded video stream CVS.
It is supposed that are well known to the generator the sequences of the alternate frames comprising the images of the Left View and the Right View, as well as the related maps (of disparity or depth); as an example, said maps are presented as depth maps of the left view with respect to the right (Depth Left) and of the right view with respect to the left (Right Depth).
In general the sequences of the images associated to the left view and to the right one are obtained in real time through proper catching instruments (TV cameras) during a stereoscopic shot, or off-line through computing techniques and instruments borrowed from the computer graphics. The disparity/depth maps can be computed through any known technique starting from the images of the two views and/or from the knowledge of the shot conditions of the images, or artificially generated by proper computing instruments.
Such a generator 3 generates schematically the stream depicted in
As said it is supposed that the generator is provided with, both the images of the input sequences L, R, and the related maps.
A proper View selector provides to alternate the frames starting from those setting the two input views, for instance starting from the left view. In such a case the view images L1, R2, L3, R4, etc., for the time periods t1, t2, t3, t4, etc. are selected in this order, obtaining a stereoscopic stream according the upper part of the
The maps D1, D3, . . . belong to the above mentioned first plurality of maps D1i. The maps D2, D4, . . . belong to the above mentioned second plurality of maps D2i.
The two sequences of the view images and of the alternated maps are respectively encoded by a View encoder and a Depth encoder which can operate in cooperating way taking into account the information contained at the input of the other encoder and also the encoding techniques adopted by the other encoder.
The two encoders can be of known kind; for instance they can adopt well known video code standards such as MPEG-2, MPEG-4 AVC, VC-1, etc., so as to use already existing instruments o devices for the compression of the images. Alternatively they can operate according to future code systems still under standardization, such as MPEG-4 AVC/SVC/MVC with the appropriate extensions necessary to include depth maps, HEVC and relative extensions.
The two streams Encoded view stream and Encoded depth stream respectively consisting of alternated views and maps, both compressed, exiting from the two encoders, are merged in the stereoscopic video encoded stream CVS by a Multiplexer, providing a parallel-series conversion of the two input streams.
The encoded video stream CVS will be possibly compounded with other information streams such as audio streams and/or data (subtitles, metadata, etc.) and stored in a storing device for subsequent play in a system, designed to allow the display, or to be transmitted according to the state of the art through cable, radio, satellite, IP (Internet Protocol), and so on.
It is to be noted that the diagram of
The generating apparatus 1′ is similar to the apparatus 1 shown in
The block “View selector” of
The inputs are processed in such a way that one of the two views (for instance the right one) is synthesized starting from the depth map and from the other view and then transmitted instead of the original one, while the other (for instance the left one) is sent in original form. In such a way the brightness and colorimetry difference present in the adjacent frames is reduced. The synthesis is effected using at first sight the disparity information present in the depth map, and the other view; instead the occlusions, after having been identified, are obtained from the same original stereoscopic view. In substance it is a procedure similar to that used by the rebuilding apparatus to obtain the frames of the missing view starting from the received ones.
a-10d show the necessary steps for the synthesis; in particular
a and 10c show the first step of the synthesis algorithm for the ith frame of the stream; from the view Li and from the depth map Di(Li→Ri) the view R0i* is computed through a View Synthesis block of known kind which can also be the same used in rebuilding step. The frames R0i* contain some unknown pixels at the occluded regions which are not obtainable from the View Synthesis algorithm. In contrast with what happens in the rebuilding (
In case the right view is taken as a reference, the same steps are carried out starting from the sequences R, L0, computing the incomplete views L0i* and, from these, the rebuilt virtual views Li to be used in the generation of the stereoscopic stream exiting from the View Synthesis and View Selector block (
As said above, the encoded video stream CVS can be transmitted and/or stored on a proper storing support, in order to be provided later to an apparatus able to rebuild the initial video stream so to allow a use of the same.
Such an apparatus for the rebuilding of a video stream, or decoder apparatus, is indicated by the numeral 2 in the enclosed figures.
The rebuilding apparatus 2 (
Such an encoded video stream CVS has the above described structure. In synthesis, the encoded video stream CVS comprises at least: a first image L1 of a first input sequence L, a first map D1 associated to such a first image L1, a second image R2 of a second input sequence R and a second map sequence associated to such a second image R2.
The rebuilding apparatus 2 further comprises an operating module 21 configured to carry out, in general, steps specular to the above described ones with reference to the generation step of the video stream.
In particular, the operating module 21 provides to rebuild a first image R1 of the second input sequence R as a function of the first image L1 of the first input sequence L and of the first map D1 associated therewith; in this way a first rebuilt image R1′ is obtained.
The operating module 21 further reconstructs a second image L2 of the first input sequence L as a function of the second image R2 of the second input sequence R and the second map D2 associated thereto; in such a way a second rebuilt image L2′ is obtained.
Following the above described steps the operating module 21 can thus prepare as output a decoded stereoscopic video stream DVS; such a decoded video stream DVS comprises:
In particular, in the first output sequence L′ the second rebuilt image L2′ is time following and time adjacent to the first image L1 of the first input sequence L and, in the second output sequence R′, the second image R2 of the second input sequence R is time following and time adjacent to the first rebuilt first image R1′.
The first and second output sequences L′, R′ are respectively representative of a first and second view of a decoded stereoscopic video stream DVS.
Therefore, by associating the rebuilding apparatus 2 to a proper display device, as for instance a TV set provided with 3D display function, it will be possible to display the decoded video stream DVS giving the user the proper depth representation of the contents of such a stream.
As said above, preferably the encoded video stream CVS received by the rebuilding apparatus 2 comprises in general, a first plurality of images Li belonging to the first input sequence L, a first plurality of maps D1i each one associated to a respective image of said first plurality of images Li, a second plurality of images Ri belonging to the second input sequence R, and a second plurality of maps D2i each one associated to a respective image of the second plurality of images Ri.
Referring to what already described with respect to the generation apparatus 1, the first plurality of images Li comprises a first image Li of the first input sequence L, the first plurality of maps D1i comprises a first map D1, the second plurality of images Ri comprises the second image R2 of the second input sequence R, and the second plurality of maps D2i comprises the second map D2.
Preferably each one image Li of the first plurality is associated to a respective first time reference TLi, and each image Ri of the second plurality is associated to a respective second time reference TRi.
Advantageously the first time references TLi are time alternated to the second time references TRi.
In practice, the images of the first plurality of images Li are time alternated to the images of the second plurality Ri.
Once the input interface 20 receives such an encoded stereoscopic video stream CVS, the operating module 21 provides to operate on the images forming such a stream in the above described manner with reference to the first image L1 and the second image R2, in order to rebuild the missing images and to generate as output the decoded video stream DVS.
In particular, the operating module 21 is configured to rebuild images of the second input sequence R as a function of the first plurality of images Li and the first plurality of maps D1i, obtaining corresponding first rebuilt images Ri′.
The operating module 21 further provides to rebuild images of the first input sequence L as a function of the second plurality of images Ri and the second plurality of maps D2i, obtaining corresponding second rebuilt images Li′.
The output decoded video stream DVS will thus comprise:
The first and second output sequence L′, R′ are respectively representative of a first and a second view of the decoded stereoscopic video stream DVS.
The operating module 21 is preferably configured to manage the presence of one or more occluded pixels which, for instance, can prevent a complete rebuilding, as a function of the first image L1 of the first input sequence L and the related first map D1, of the first rebuilt image R1′.
In particular, as a function of the first image L1 of the first input sequence L and the related first map D1, the operating module 21 identifies one or more occluded pixels with respect to the corresponding first image R1 of the second input sequence R. The operating module 21 so provides to determine, as a function of one or more determined images of the second input sequence R, one or more replacement pixels to replace one or more occluded pixels.
In more detail, the images of the second input sequence R, used by the operating module 21 to determine said replacement pixels, are images belonging to the mentioned plurality of images Ri comprised in the encoded video stream CVS.
Preferably the image immediately preceding that to be rebuilt is taken into consideration.
In addition or in alternative the following one can be used. For the initial image, that is that image not having by definition a previous image, the following image is necessarily used.
This operation can be carried out for each one of the images to be rebuilt, should they belong to the first or second output sequence L′, R′.
Advantageously the encoded video stream CVS can comprise describing data of replacement pixels to be used in some cases in the rebuilding of the missing images.
As above described, said describing data are inserted in the encode step wherein the only images and maps available to the decoder would not be enough for a complete and satisfactory rebuilding of the missing images.
As an example, the describing data of replacement pixels, which preferably can comprise or consist of the mentioned main data, can comprise motion vectors.
The operating module 21 is thus configured to detect in the encoded video stream CVS, the descriptive data of replacement pixels, if present, and to use the same for the rebuilding of the occluded areas.
a-11d schematically show how the missing incomplete views Li*, Ri*, that is the rebuilt views in which are still present some occlusions can be rebuilt at each step “i” greater than 1, and how the complete missing views Li′, Ri′ can be rebuilt from these, i.e. the views in which the occlusions have been substituted, starting from the actually received frames.
The two video decoders can be of known kind, such as, for instance MPEG-2 or MPEG-4 AVC, or one of those future ones under standardization such as MPEG-4 AVC/SVC/MVC and HEVC with the extension of depth maps, or to be a their modified version optimized for treating the video streams present on their input. The two decoders operate in synergy in coordinate way and can possibly exchange control signals and data with each other for exploiting time and video content information of the streams adapted to warrant the appropriate operation and an optimal decoding, based on the compression system adopted for the generation of the complex stream.
Also on the rebuilding side hold the same considerations made for the encoding in generation side: alternatively to the scheme proposed in
The input Front-end stage can so comprise an only video decoder device providing, according to the cases, to de-multiplex the only input stream or to treat the two not de-multiplexed input streams to provide as output the two separated streams.
Later, these two streams are processed by a block of view synthesis and occlusion filling View Synthesis and occlusion filling, making part of the mentioned operating module 21, dealing with generating the missing view, i.e. the not transmitted view, using where possible the information contained in the map. The parts that can not be rebuilt through the map are obtained using the particular alternate set of the stream received in accordance with the above described technique.
As output of the block View Synthesis and occlusion filling the two sequences relating the Left View and the Right View are obtained, which can be thus provided to a display apparatus for their 3D yield (or also just 2D yield) according to any prior art, such as the line interleave or frame alternate technique.
In the case it is wanted to display a 3D video in 2D mode, it is possible to proceed in different ways.
In a first solution it is possible to use the block View Synthesis and occlusion filling to rebuild the missing view as it happens for the 3D case, leaving to the display device the discard operation of all right and left images not necessary for the 2D vision. In such a case the resulting frame rate of the video content displayed in 2D is equal to the original one present on the generation side. In substance a complete regeneration of the 3D stream is performed according to the functional scheme depicted in
As a second solution only the frames transmitted for just one of the two views are used and the video is displayed at a frame rate which is one half with respect to the first solution, as shown in example way in a 3D reconfigurable stream regenerator shown in
This second solution allows to considerably simplifying the operations on the regeneration side as the rebuilding steps of the missing view frames and the related occlusions are no more necessary.
b depicts the operating mode of regenerator of
a shows the iterative structure of the process cyclically repeated for each frame pair received at a determined time ti, while
Back to
The following control checks if the received frame (of reference) belongs to the right or left views: in the first case the left frame is rebuilt starting from the immediately preceding left frame and from the related depth map, while the right frame is stored in a buffer to be used during the rebuilding of the following right frame, as supposed in
This way to extract the occluded pixels from the missing views is particularly simple to embody, as it requires the storage of just a frame for every step of the rebuilding procedure and takes into account pixels belonging to one or two frames at most. It however represents just one of the multiple possible solutions: in fact it is possible to take into account also pixels belonging to many view frames preceding and/or following the one to be rebuilt.
In such a last case, the contemporary storage of more frames is necessary in order to select the pixels of the adjacent view frames which better approximate the occlusions of the frame to be rebuilt. In the case in which the algorithm cannot identify the above mentioned pixels it is possible to resort to occlusion filling techniques as for instance the in-painting in order to obtain the highest possible quality of the rebuilt view.
Then a filter is applied for reducing the noise connected to the rebuilding (referred to as de-noising filter) of already known kind; during the tests carried out by the applicant the use of a bilateral filter as de-noising filter has been noticed as particularly efficient as it allows to preserve the outlines.
At the end of the operations performed for the current frame it is checked whether frames to be processed still exist: in the affirmative the operations are stopped, while in the (i+1)th negative the process continues with the reading of the frame pair of the subsequent time.
The explained invention can be extended to the case with more than two views using the modularity according to the existing requirements. On this subject two embodiments are proposed which exemplify what explained, referring to the 3D stream with more than two views present at the output of a video stream generator in a similar way to what schematically shown in
In an embodiment, described in
In a different embodiment, shown in
In the case in which the view number is odd, the odd view is processed according to the state of the art of the existing algorithms, so maintaining in such a case the requirement of the 2D retro-compatibility.
It is to be noted that the apparatus and the above described modules can be embodied through dedicated hardware, or through general purpose hardware duly programmed to carry out the various above described functions.
The invention in fact regards also a software for generating a video stream, comprising the necessary instructions for carrying out the operations of the generating apparatus 1, and a software for rebuilding a video stream, comprising the instructions needed for carrying out the operations of the rebuilding apparatus 2.
Further, it is to be noted that the subdivision in modules or functional blocks of the above described apparatus is to be considered just formal and directed to a clear exposition of the functions of the invention. Such a subdivision, moreover, does not necessarily reflect the hardware structure of the described devices.
The invention attains important advantages.
First of all, the generation technique according to the present invention allows to encode the initial video stream reducing in significant way the size preventing, in the meantime, the loss of essential information for a complete and reliable rebuilding of the same.
In particular the encoded video stream can be stored using storing supports of limited capacity.
Further, the encoded video stream can be transmitted and received occupying limited network resources with respect to a corresponding 3D Full HD stream, just by virtue of the reduced size of the encoded stream.
Further, the encoded stereoscopic video stream can be easily adapted to be represented also in 2D mode according to two different modes, with different quality and computing complexity.
The generation and rebuilding technique of the 3D video stream according to the present invention can be applied not only to a transceiver system or a stereoscopic 3D filing-playing system, but also to corresponding 3D systems having more than two views.
In addition to what above, the decoded stream presents a substantial identity with the initial stream (i.e. with the stream not yet encoded) and allows a high quality use of 3D video contents.
| Number | Date | Country | Kind |
|---|---|---|---|
| TO2012A001073 | Dec 2012 | IT | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/IB2013/060856 | 12/12/2013 | WO | 00 |
