Patent Application
20150093019
The invention relates to a method and an apparatus for generating depth maps for a sequence of images, and more specifically to a method and an apparatus for generating depth maps for a sequence of images, which are temporally consistent.
Today there is a trend to create and deliver richer media experiences to consumers. In order to go beyond the ability of either sample based (video) or model-based (CGI) methods novel representations for digital media are required. One such media representation is SCENE media representation (http://3d-scene.eu). Therefore, tools need to be developed for the generation of such media representations, which provide the capturing of 3D video being seamlessly combined with CGI.
The SCENE media representation will allow the manipulation and delivery of SCENE media to either 2D or 3D platforms, in either linear or interactive form, by enhancing the whole chain of multidimensional media production. Special focus is on spatio-temporal consistent scene representations. The project also evaluates the possibilities for standardizing a SCENE Representation Architecture (SRA).
In the SCENE project a professional video camera is developed, which integrates a time of flight camera to generate depth maps. Since the time of flight camera uses the same optics as the video camera, both cameras record material from the same perspective. As a consequence, the registration of video material and depth maps is much simpler as depth pixel and color pixel are consistent.
Some approaches to improve the quality of depth or disparity maps employ an over-segmentation of the video frames, making the assumption that depth or disparity values within one segment are similar. If this assumption holds, missing depth or disparity values are generated by “extrapolation” of the good values from the same segment.
European Patent Application EP 13171832.2 describes a method for temporally consistent video over-segmentation with superpixels. A superpixel can be considered as a cluster of adjacent image pixels, which share a set of common properties, e.g. color. The generation of superpixels can be conceived as a clustering problem, as described, for example, in R. Achanta et al.: “SLIC Superpixels Compared to State-of-the-Art Superpixel Methods”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43 (2012), pp. 2274-2282.
It would be desirable to achieve comparable consistency also for depth maps associated to a sequence of images.
It is thus an object of the present invention to propose a solution for generating temporally consistent depth maps for a sequence of images.
According to the invention, a method generating temporally consistent depth maps for a sequence of images for generating temporally consistent depth maps for a sequence of images comprises:
Accordingly, an apparatus configured to generate temporally consistent depth maps for a sequence of images comprises:
Similarly, a computer readable storage medium has stored therein instructions enabling generating temporally consistent depth maps for a sequence of images, which when executed by a computer, cause the computer to:
According to the invention three different types of input data are combined to generate temporally consistent depth maps. The first input data are video frames, for which a representation by segmentation regions resulting from a temporally consistent over-segmentation is available. An example of such segmentations regions are superpixels. The second input data are depth map frames. The solution assumes that there is a function F, which assigns every pixel of a video frame to one pixel in the related depth map. By means of this function the segmentation regions resulting from the temporally consistent over-segmentation for a video frame are projected easily onto the related depth map. Once this projection is complete, depth values of the depth map are modified within one or more of the projected segmentation regions.
The proposed solution has the advantage that temporally consistent and more reliable depth maps are generated for video material for which a temporally consistent over-segmentation is available. As real-time implementations are available for temporally consistent over-segmentation, the solution can be easily be incorporated into workflows requiring real-time processing.
Advantageously, the depth values are modified by fitting a regression plane to a projected segmentation region and replacing one or more depth values in the projected segmentation region with depth values represented by the regression plane. The regression plane is, for example, fitted to the projected segmentation region using a random sample consensus algorithm. The use of regression planes allows removing outliers, e.g. noise, very easily.
Advantageously, only those depth values are modified in the projected segmentation region, whose distance to the regression plane is larger than a threshold. This ensures that the depth information in the depth map is not modified more than necessary, i.e. that the details in the depth map are maintained.
Preferably, the quality of a potential regression plane is controlled by a first parameter defining a maximum allowed distance of a depth value from the plane to be considered located in the plane and a second parameter defining the minimal percentage of depth values of the projected segmentation region that are located in the plane or within the allowed distance from the plane. These quality criteria allow easily determining a group of good matching regression planes, among which the best matching plane is later determined.
Favorably, a missing regression plane for a projected segmentation region of the image is interpolated from corresponding regression planes of temporally adjacent images. For this purpose motion data for the segmentation region is preferably taken into account, like the average motion vector. This data is computed during the temporally consistent over-segmentation. The interpolation is useful in case for a certain segmentation region in a frame no plane can be determined, which fulfills the above quality criteria.
For a better understanding the invention shall now be explained in more detail in the following description with reference to the figures. It is understood that the invention is not limited to this exemplary embodiment and that specified features can also expediently be combined and/or modified without departing from the scope of the present invention as defined in the appended claims.
In the following the proposed solution shall be explained for a temporally consistent superpixel over-segmentation. Of course, the solution works with any temporally consistent over-segmentation approach. Temporally consistent superpixels are only one example. In addition, it is assumed that depth values, which belong to the same superpixel, can be approximated by a plane in the three dimensional space. In other words, the superpixels are assumed to represent an over-segmentation of a real object.
By means of the function F the superpixel segmentation for video frame i can be projected easily onto the related depth map 3. Thus, the function F(Si,j) determines the pixels in the depth map 3 which belong to the superpixel j in the video frame i.
Now, regression planes are determined for every projected superpixel in the depth map 3. Every plane can be defined by the selection of three points of the superpixel in the depth map 3. The only condition is that the direction vectors of the selected points are linearly independent. The quality of a regression plane Z is defined by the percentage of points of the superpixel in the depth map 3, which lie directly on the plane Z or have a deviation which is smaller than a certain threshold. Thus, the quality of good matching planes is controlled by two parameters. Firstly, the parameter d, which defines a maximum allowed deviation, i.e. distance, of a point from a plane. All points with a distance that is smaller than d, are considered to be in the plane. Secondly, the parameter l is the minimal percentage of points of F(Si,j), which lie directly on Z or have a maximal distance d from the plane Z.
Then, using a RANSAC algorithm (RANSAC: RANdom SAmple Consensus) for every projected superpixel j in the depth map 3 of frame i the best fitting regression plane Z is determined out of the set of “good matching” planes. Further explanations on how to determine the best fitting plane Z for a superpixel in the depth map 3 utilizing the temporal consistency of superpixels will be given further below. Of course, the RANSAC algorithm is only one possible solution for finding the best fitting regression plane Z. Other mathematical ways to construct and interpolate regression planes are well known to the skilled person.
In a next step all depth values belonging to F(Si,j), which have a distance to Z that is greater than a certain threshold t, will be replaced by the values represented by Z at the respective positions in order to generate the temporally consistent depth map 4. It is assumed that such values are outliers or noise in the depth map 3. Thus, the improvement of this step is threefold. Firstly, outliers will be removed. Secondly, temporally consistent over-segmentation in the image and in the depth map 4 becomes consistent. And thirdly, the temporal consistency of the superpixels in the depth map 4 is improved. As will be explained further below, the temporal consistency can be further improved if the determination of the best matching regression plane Z also takes the past and the future of a superpixel into account.
Basically two cases have to be distinguished for the calculation of the plane Z. If the superpixel j is marked as a static superpixel during the temporally consistent over-segmentation, i.e. if the superpixel j is marked as a superpixel without motion, the calculation of the best fitting plane Z can advantageously consider also the superpixel j in the previous and following frames F(St,j) with t={i−n, . . . , i, . . . i+m}, as long as j remains static in these frames. So the set of supporting points for the regression plane can be extended, which allows a more robust estimation of the plane in temporal direction. If, however, the superpixel j is not marked static, only the supporting points which are defined by F(Si,j) are used.
It may happen that for a certain superpixel j in frame i no plane can be determined, which fulfills the requested quality criteria. In this case a plane might be interpolated, if planes for this superpixel j exist in the past and future frames. If at frame i−n (past) and i+m (future) a plane can be determined with the method above, the missing planes (between i−n and i+m) can be approximated with different mathematical methods. In the following a geometrical method using normal vectors is described, with reference to
It is assumed that for frame i−1 (past) and i+1 (future) an optimal fitting regression plan Z can be determined with the method described above. Depending on the chosen mathematical method to define the plane the normal vector of the plane is known or can be determined easily. A special normal vector is the one through the origin. The intersections of these normal vectors through the origin with the planes at times i−1 and i+1 are S1 and S2.
The plane for frame i can now be interpolated by the computation of S3 . In this case S3 is defined by S3=½(S1+S2). The position vector to S3 can be considered as normal vector for the plane at frame i. Of course, this method can also be used to interpolate k planes. In this case the intersection points Sk for the normal vectors are defined as
with l={1, . . . , k}.
Without loss of generality, in the previous example a linear interpolation of the planes is assumed, as apparent from the factor
since the temporal distance between consecutive frames is assumed to be constant.
If also in past and future frames no good matching plane could be found, the depth values of the related superpixel cannot be improved.
A method according to the invention for generating temporally consistent depth maps 4 for a sequence of images is schematically shown in
| Number | Date | Country | Kind |
|---|---|---|---|
| 13306367.7 | Oct 2013 | EP | regional |
