Patent Application
20120062699
This invention concerns monitoring of video material to detect the presence of stereoscopic images.
There is increasing interest in stereoscopic television, in which a pair of images of the same scene from closely-spaced, horizontally-adjacent viewpoints are respectively presented to the left and right eyes of a viewer so as to obtain a ‘three dimensional’ image. One popular way of using existing transmission equipment to deliver stereoscopic images is to divide each frame into two halves and to use one half-frame for the left-eye image, and the other half-frame for the right-eye image. If the frame is divided horizontally, the horizontal scale of the two images is (approximately) halved; alternatively, if the frame is divided vertically, the vertical scale is (approximately) halved. The rescaling is, of course, reversed prior to display of the image.
This modified use of a transmission system requires careful monitoring and control to avoid presenting double images to 2D viewers, or simultaneously presenting different parts of the same image to the left and right eyes of stereoscopic viewers. There is thus great commercial utility for a system that processes image files or data and determines whether conventional two-dimensional images are being distributed, or whether, and in what manner, two component images of a stereoscopic image are being sent in the same frame. It is particularly helpful if image files or data can be monitored in this way whenever they are processed or transmitted. Therefore it is highly beneficial for the method of monitoring, and the monitoring apparatus, to be simple and inexpensive, so that the monitoring process can be included at many points in a transmission chain without undue additional cost.
The invention consists in a method and apparatus for detecting the presence of the two constituent images of a stereoscopic image within an image frame by analysis of columns of horizontally-averaged pixel values and rows of vertically-averaged pixel values.
In a preferred embodiment, the ordered set of values of a row of vertically-averaged pixels is analysed to detect the similarity of average values corresponding to respective image regions within the said frame horizontally separated by half the width of the said frame and the presence of a side-by-side arrangement of constituent images of a stereoscopic image is inferred from said similarity.
Suitably, the ordered set of values of a column of horizontally-averaged pixels is analysed to detect the similarity of average values corresponding to respective image regions within the said frame vertically separated by half the height of the said frame and the presence of over-and-under arrangement of constituent images of a stereoscopic image is inferred from said similarity.
Advantageously, the average difference between members of a set of a row or column of averaged pixel values is evaluated.
In some embodiments the correlation between the members of a set of a row or column of averaged pixel values is evaluated, and phase correlation may be used.
In an alternative embodiment a first row of vertically-averaged pixels is derived from the upper half of the said frame and compared with a second row of vertically-averaged pixels derived from the lower half of the said frame and the presence of over-and-under arrangement of constituent images of a stereoscopic image is inferred when these first and second rows of vertically-averaged pixels are found to be similar.
Suitably, a first column of horizontally-averaged pixels is derived from the left half of the said frame and compared with a second column of horizontally-averaged pixels derived from the right half of the said frame and the presence of side-by-side arrangement of constituent images of a stereoscopic image is inferred when these first and second rows of vertically-averaged pixels are found to be similar.
Advantageously, the average difference between respective members of a pair of rows of averaged pixels or between respective member of a pair of columns of averaged pixels is evaluated.
Alternatively, the correlation between two rows of averaged pixels or between two columns of averaged pixels is evaluated, and phase correlation may be used.
In preferred embodiments a measure of similarity between pixel values or combinations of pixel values is biased in the direction of dissimilarity when a measure of the range of values within a row or column of pixel values or combinations of pixel values is small.
In some embodiments pixels close to the edge of the said frame or close to the vertical centre line, or horizontal centre line of the said frame are disregarded.
An example of the invention will now be described with reference to the drawings in which:
An exemplary image frame (1) is shown in
If the frame (1) is used to carry two constituent images of a stereoscopic image in ‘side-by-side’ format, then the portion of the frame to the left of the vertical centre-line (2) will carry one constituent image, and the portion of the frame to the right of the vertical centre-line (2) will carry the other constituent image. Alternatively, if two constituent images are carried in ‘over-and-under’ format, then the portion of the frame above the horizontal centre-line (3) will carry one constituent image, and the portion of the frame below the horizontal centre-line (3) will carry the other constituent image.
The boundaries of equivalent top-edge regions, Ay vertical pixel-pitches wide, for the frame and the bottom half of the frame are shown by the lines (8) and (9) respectively. And, the boundaries of equivalent bottom-edge regions, By vertical pixel-pitches wide, for the frame and the bottom half of the frame are shown by the lines (10) and (11) respectively.
Although the two constituent images of a ‘stereo pair’ of images are taken from different viewpoints, the close proximity of these viewpoints results in significant similarity between the respective images. The identification of stereoscopic material is thus possible by evaluating the correlation between the pixel values of the left and right sides of the frame, and the correlation between the pixel values of the top and bottom halves of the frame. However, the calculation of two-dimensional cross-correlation functions, possibly requiring multi-dimensional transforms of sets of pixel values, requires significant processing resources. This approach is generally not cost-effective when monitoring large numbers of video data streams or files at many points in a distribution chain.
The inventor has appreciated that over-and-under and side-by-side stereoscopic material can be identified by simple processing applied to one or two rows of vertically averaged pixel values and one or two columns of horizontally averaged pixel values derived from each frame.
A suitable row, that is representative of the horizontal variation of pixel values, and comprises W values is given by:
R(x)=(1/H)×ΣP(x,y) [1 ]
Where:
A suitable column, that is representative of the vertical variation of pixel values, and comprises H values is given by:
C(y)=(1/W)×ΣP(x,y) [2]
Where:
Note that, in each case, the division by the height or width respectively gives similarity in the number range to be processed regardless of the size of the frame.
One method of detecting constituent images of a stereoscopic image is to evaluate two, one-dimensional cross-correlation parameters:
If only one of these two cross-correlation evaluations returns a high measure of correlation, then over-and-under carriage of stereo can be detected if measure (a) is high and measure (b) is low; and, side-by-side carriage of stereo can be detected if measure (b) is high and measure (a) is low. If neither measure shows high correlation, then the absence of stereoscopic component images can be detected.
The correlation can be evaluated mathematically from the known cross-correlation integral, derived either: in the spatial domain by convolution of the two sets of pixel values taken from different halves of the image; or, in the frequency domain from respective one-dimensional transforms of the sets of pixel values. A suitable frequency domain technique is the known method one-dimensional phase-correlation where the phases of respective frequency components are compared and the set of phase differences is transformed from the frequency domain back to the spatial domain and analysed to detect peaks.
Alternatively pixel-value differences can be evaluated to detect correlation between halves of the frame. This is much simpler than the above correlation methods and therefore more cost effective for a system that requires many different image files or streams to be monitored at many different locations. A system using this principle to analyse an array of pixel values to detect the presence of two component images of a stereo pair is shown in
Vertical columns of pixel values are respectively averaged according to equation [1] above by an averager (21) to give a row of W average values R(x). And, horizontal rows of pixel values are respectively averaged according to equation [2] above by a second averager (22) to give a column of H average values C(y).
The magnitudes of the differences between pairs of values from the row of average values R(x) that are separated by half the image width are evaluated and averaged in a horizontal difference averager (23) according to equation [3] below:
Δx=G×Σ{R(x)−R(x+W/2)}÷(W/2−Ax−Bx+1) [3]
Where:
The horizontal-difference measure Δx is thus a measure of the average difference between respective, vertically-averaged columns of pixels at equivalent horizontal positions in the left and right halves of the frame. Image edge regions, including the edges of constituent stereo images filling half of the frame, are prevented from contributing to the difference measure by suitable choice of the constants Ax and Bx.
Similarly, the magnitudes of the differences between pairs of values from the column of average values C(y) that are separated by half the image height are evaluated and averaged in a vertical-difference averager (24) according to equation [4] below:
Δy=G×Σ{C(y)−C(y+H/2)}÷(H/2−Ay−By+1) [4]
Where:
The vertical-difference measure Δy is thus a measure of the average difference between respective, horizontally-averaged rows of pixels at equivalent vertical positions in the top and bottom halves of the frame. Image edge regions, including the edges of constituent stereo images filling half of the frame, are prevented from contributing to the difference measure by suitable choice of the constants Ay and By.
A suitable value for the constant G is 8, for typical 8-bit pixel values; and, 10 is a suitable value for the constants Ax, Ay, Bx and By.
When the whole frame is filled with a typical single image, it is unlikely that the left and right halves of the frame will be similar, and also unlikely that the top and bottom halves of the frame will be similar; both Δx and Δy can therefore be expected to be large.
If the frame is used for over-and-under transmission of the two constituent images of a stereoscopic image, the horizontal-difference measure can be expected to be large because the respective left and right halves of the constituent images are likely to be different. However, the vertical-difference measure is very likely to be small because the left-eye and right-eye views of a scene are acquired by cameras at the same height.
Alternatively, if the frame is used for side-by-side transmission of the two constituent images of a stereoscopic image, the vertical-difference measure can be expected to be large because the top and bottom halves of the constituent images are likely to be different. However, the horizontal difference measure is likely to be small—but not zero, because of the horizontal disparity between the left-eye and right-eye positions of objects intended to be portrayed in front of, or behind, the display.
Thus, provided the frame contains images having varying pixel values, it should be possible to detect whether or not side-by-side or over-and-under representation of stereoscopic is being used. However, if the pixel values do not vary over the image, perhaps following a fade to black, both difference measures will be zero and detection stereo images will not be possible. This difficulty can be avoided by adding a correction factor to the difference measures as will now be described.
The maximum pixel-value for each of the W columns of pixels is determined by a maximum value detector (25); and, the minimum value for each of the W columns of pixels is determined by a minimum value detector (26). The respective differences between these maxima and minima are then calculated by a subtractor (27) to give a set of W pixel-value range values according to equation [5] below:
Range(x)={Max(P(x,y))−Min(P(x,y))} [5]
Where:
These range values are averaged over the frame width in an averager (28) to give an average column range value according to equation [6] below:
RangeCol=(1/W)×Σ{Range(x)} [6]
Where:
Analogous processing is applied to the H rows of pixel values to obtain an average row range value as follows. The maximum value of each of the rows of pixels is determined by a maximum value detector (29); and the minimum value of each of the rows of pixels is determined by a minimum-value detector (30). A subtractor (31) determines the set of H range values according to equation [7] below:
Range(y)={Max(P(x,y))−Min(P(x,y))} [7]
Where:
These range values are averaged over the frame height in an averager (32) to give an average row range value according to equation [8] below:
RangeRow=(1/H)×Σ{Range(y)} [8]
Where:
The two average range values, RangeCol and RangeRow are summed in an adder (33) to give an overall measure of the variation in pixel values for the image. The output from the adder (33) will be small when there is little variation in the values of pixels over the image; in this situation low values of Δx and Δy give unreliable indication of the presence of stereoscopic images, as explained above. The output from the adder (33) is input to a weighting function (34) to obtain dynamic range correction factor that is added to Δx and Δy. This correction factor should be small when the range of pixel values sufficiently large to give representative values of Δx and Δy and become large otherwise. A suitable weighting function is:
δ=J×exp(−(z/K)2) [9]
Where:
The value of J sets the maximum value of δ when the input range z is zero; a value of 120 has been found to be suitable. The value of K determines how rapidly the weighting decreases as the input range z increases; a value of 2048 has been found suitable.
Returning to
Frames carrying the two constituent images of a stereo image will usually result in one of the comparator outputs being TRUE and the being other FALSE. Frames carrying a single image will usually result in both comparators giving TRUE outputs because both the horizontal difference measure Δx and the vertical difference measure Δy are above the threshold. Thus the FALSE condition at the output of the comparator (40) is indicative of side-by-side constituent images; and the FALSE condition at the output of the comparator (41) is indicative of over-and-under constituent images.
The logic block (42) ensures that the output flags (43) and (44) are mutually exclusive, and provides additional temporal averaging when processing sequences of images. A suitable algorithm for this latter case is to count the number of frames for which each of the comparators (40) and (41) give FALSE outputs over, say, 200 frames and to set the output flag corresponding to the comparator giving the higher number of FALSE outputs exceeding a suitable threshold number, say 50. Thus if the comparator (40) gives a FALSE output for more than 50 frames in the last 200 frames, and the comparator (41) gives fewer FALSE outputs for these frames, then the side-by-side flag (43) is set. And, similarly if the comparator (41) gives a higher number of FALSE outputs exceeding 50 for these frames, and the comparator (40) gives a fewer number of FALSE outputs, then the over-and-under flag (44) is set. If neither comparator gives more than 50 FALSE outputs in the most recent 200 frames, then neither output flag is set.
This logical processing can be improved by making use of the Δx and Δy values from the vertical-difference averager (23) and the horizontal-difference averager (24) when neither of the comparators (39) and (40) gives more than 50 FALSE outputs in the most recent 200 frames. The Δx and Δy values can be summed respectively over these 200 frames and the respective summations compared both with each other and with a threshold value, say 14,000. If the lower sum is less than the threshold, the appropriate flag is set; neither flag is set if both sums exceed the threshold. Other variations in this logic are possible, for example different thresholds can be set for the two comparisons, and the thresholds may depend on the currently detected flags so as to provide hysteresis that prevents a change in an output flag until a significant change in the measured horizontal or vertical difference values occurs.
The system shown in
R
Upper(x)=(2/H)×ΣP(x,y) [10]
Where:
R
Lower(x)=(2/H)×ΣP(x,y) [11]
Where:
C
Left(y)=(2/W)×ΣP(x,y) [12]
Where:
C
Right(y)=(2/W)×ΣP(x,y) [13]
Where:
The presence of over-and-under stereoscopic images can be inferred when the correlation between RUpper(x) and RLower(x) is high, and the correlation between CLeft(y) and CRight(y) is low.
Similarly, the presence of side-by-side stereoscopic images can be inferred when the correlation between CLeft(y) and CRight(y) is high, and the correlation between RUpper(x) and RLower(x) is low.
The complexity of correlation processing can be avoided by forming average horizontal and vertical pixel value difference measures as follows:
Δx′=G×Σ{CLeft(y)−CRight(y)}÷(H−Ay−By+1) [14]
Where:
And:
Δy′=G×Σ{RUpper(x)−RLower(x)}÷(W−Ax−Bx+1) [15]
Where:
The presence of over-and-under stereoscopic images can be inferred when Δy′ is low, and Δx′ is high. Similarly, the presence of side-by-side stereoscopic images can be inferred when Δx′ is low, and Δy′ is high. As in the system of
The invention can be applied to sets of values that describe any convenient image attribute at a respective set of pixel locations; or, to sets of combined pixel values where each member of the set is formed by combining the values for different image attributes at the same pixel location. The operations of taking averages and of taking differences may in appropriate cases be reversed in order. Typically luminance values are used but other pixel values may be employed. Images may be subsampled or oversampled prior to processing. Image-edge values, including values close to the horizontal or vertical centre lines of the image, may or may not be included in any of the summation and averaging processes. The invention can be applied to single frames or sequences of frames; of course the temporal filtering of horizontal difference values and the summation of values over multiple frames cannot be applied when processing unrelated images. The above description has been in terms of hardware blocks, other implementations, including processor-based implementations are possible, and will be apparent to the skilled person from the above description.
| Number | Date | Country | Kind |
|---|---|---|---|
| 1015089.4 | Sep 2010 | GB | national |
