1. Field of the Invention
The present invention relates to a method of establishing depth of field (DOF) data, and more particularly to a method of establishing DOF data and a system thereof, applicable to calculate offset values between two visual images in two different visual angles to obtain a depth map.
2. Related Art
In general, a three-dimensional (3D) image is formed by two sets of image data in different visual angles, in which one set of image data corresponds to a left-eye visual angle, and the other set of image data corresponds to a right-eye visual angle. The image corresponding to the left-eye visual angle is referred to as a left-eye visual image, and the image corresponding to the right-eye visual angle is referred to as a right-eye visual image.
The prior art includes three modes of establishing a 3D image. In a first mode, a 3D scene, including virtual characters, virtual objects, virtual buildings, and the like, is established through virtual reality software, and then the 3D scene is shot with a camera kit of the virtual reality software in different visual angles. However, an image produced by the virtual reality software already has depth information (i.e., the image has included 3 axial data perpendicular to one another, and the shot object or scene can be rotated under the control of the virtual reality software).
In a second mode, two camera devices are used to shoot the same scene, so as to generate images of the scene in two visual angles, and the two visual images are respectively a left-eye visual image and a right-eye visual image. When the image is displayed, the left eye of a viewer is made to merely see the left-eye visual image, and the right eye of the viewer is made to merely see the right-eye visual image. Accordingly, a stereoscopic vision is generated in the brain of the viewer, such that the viewer feels that a real 3D object is viewed.
In a third mode, a camera device with an infrared sensor is used to shoot a scene, in which the infrared sensor emits an infrared ray, the infrared ray is reflected when encountering the object under shot, and the infrared sensor receives the reflected infrared ray and determines a distance between the scene and the camera device according to conditions such as time and frequency of receiving the infrared ray, so as to determine the depth change of an outline of the real scene, thereby calculating the depth data of the scene to be integrated in the shot image.
However, the mode of establishing a 3D scene through the virtual reality software before shooting an image needs to firstly design a virtual scene and shoot the virtual scene to produce 3D animations, which is rather time-consuming and cannot be applied to shoot real objects (including characters or articles).
Furthermore, in the mode of shooting the same scene to generate two different images corresponding to different visual angles and combining the two images into a 3D image, although viewers can all have a stereoscopic sensation about the object from the 3D image, DOF data or DOF signals cannot be obtained from the 3D image.
Moreover, when images are shot with the camera device having the infrared sensor, relevant DOF data can be calculated by using the infrared ray to sense the depth and distance from the scene. However, the sensing distance of the infrared sensor is quite limited, and when the camera device is too far away from the real scene, the infrared sensor is unable to sense the depth change of the outline of the real scene, that is, unable to obtain valid DOF data correctly.
Therefore, how to effectively obtain DOF data of a 3D image has become a task for different manufacturers.
Accordingly, the present invention is directed to a method and a system capable of obtaining DOF data of a 3D image rapidly and effectively.
To achieve the objective, the present invention provides a method of establishing depth of field (DOF) data of a three-dimensional (3D) image, applicable to a 3D image comprising a first visual image and a second visual image. In the method, an offset vector matrix is established. The offset vector matrix includes a plurality of data fields, and each data filed is corresponding to n first pixels of the first visual image and the n is a natural number. An ath first pixel of the first visual image is obtained and the a is an integer between 1 and n. A reference frame in the first visual image is established according to a pixel selection block by taking the ath first pixel as a center, and the reference frame includes a plurality of first pixels. A target frame in the second visual image is searched for according to the reference frame where the ath first pixel belongs to, wherein the target frame has a minimum grayscale difference value with the reference frame. An offset vector value of the ath first pixel is calculated according to the minimum grayscale difference value. Therefore, the offset vector values corresponding to all of the ath first pixels are found and recorded in the offset vector matrix. The offset vector matrix is converted into a depth map.
To achieve the objective, the present invention provides a system of establishing depth of field (DOF) data of a three-dimensional (3D) image, applicable to a 3D image comprising a first visual image and a second visual image. The system includes a storage module, an offset calculator, and a comparator.
The storage module is used for recording an offset vector matrix. The offset vector matrix includes a plurality of data fields, and the data fields are corresponding to n first pixels of the first visual image and the n is a natural number. The offset calculator is used for establishing a reference frame including a plurality of first pixels in the first visual image according to a pixel selection block by taking an ath first pixel as a center. A target frame image is searched for from the second visual image according to the reference frame where the ath first pixel belongs to, and the target frame has a minimum grayscale difference value with the reference frame. An offset vector value is calculated according to the minimum grayscale difference value. The comparator is used for recording the offset vector value corresponding to each first pixel into the data fields of the offset vector matrix. When it is determined that the offset vector values of the ath first pixels have all been recorded, the comparator converts the offset vector matrix into a depth map.
In the method and the system according to the present invention, the above depth map is generated rapidly when a conventional 3D left-eye visual image and a conventional 3D right-eye visual image are converted into 2D images, such that an image displaying device displays a 3D image having a stereoscopic sensation according to the 2D images and the depth map and displays 3D effects corresponding to a plurality of viewing points of the 3D image. Moreover, an offset vector matrix records offset vector values of all the first pixels in a second visual image, so as to be converted to form the depth map. Thus, when the depth map is combined with the original 3D image, the synthesis effect of the 3D image can be effectively improved. Furthermore, the method and the system according to the present invention not only processes images generated by camera devices, but also processes animation pictures or static pictures that are not obtained through a photographing process, thereby further expanding the application range, applicable situation, and application layer of the present invention.
The present invention will become more fully understood from the detailed description given herein below for illustration only, and thus is not limitative of the present invention, and wherein:
In order to make the objects, structural features, and functions of the present invention more comprehensible, the present invention is described below in detail through relevant embodiments and accompanying drawings.
The first imaging module 21 shoots a scene 1 to generate a first visual image 11, and the second imaging module 22 shoots the same scene 1 to generate a second visual image 12. The storage module 25 records an offset vector matrix 13, and the offset vector matrix 13 includes a plurality of data fields. The number of the data fields is the same as that of the first pixels of the first visual image 11 on which an offset calculation is to be performed, which is set as n herein and n is a natural number.
The offset calculator 23 establishes a reference frame 31 in the first visual image 11 according to a pixel selection block by taking an ath first pixel 41 (as shown in
The comparator 24 is used to record each offset vector value to the data fields of the offset vector matrix 13, that is, the offset vector value of the ath first pixel 41 is recorded in the ath data field. After that, the comparator 24 sets an (a+1)th first pixel as the ath first pixel to return to the offset calculator 23 when it is determined that the data fields of the offset vector matrix 13 are not all filled with values, and then requests the offset calculator 23 to re-calculate and record the related offset vector values; on the contrary, the comparator 24 converts the offset vector matrix 13 into a depth map when it is determined that the offset vector values of the ath first pixels have all been recorded.
It should be noted that, the above pixels may be commonly known pixels or sub-pixels.
An offset vector matrix 13 is established (Step S110). The offset vector matrix 13 includes a plurality of data fields corresponding to n first pixels of the first visual image 11, and n is a natural number. As shown in
An ath first pixel 41 of the first visual image 11 is obtained (Step S120), in which a is an integer between 1 and n. In this step, the first pixels of the first visual image 11 are arranged in a sequence from left to right and from top to bottom, so that the top-left first pixel is considered as the 1st first pixel of the first visual image 11, and the bottom-right first pixel is considered as the last first pixel of the first visual image 11.
By taking the ath first pixel 41 as the center, a reference frame 31 is established in the first visual image 11 according to a pixel selection block (Step S130). The reference frame 31 further includes a plurality of first pixels for performing grayscale comparison. The reference frame 31 may be in a square shape having a length of 3 pixel length, 5 pixel length, 7 pixel length, or 9 pixel length, that is, odd-numbered pixel length.
A target frame having a minimum grayscale difference value with the reference frame 31 is searched for from the second visual image 12 according to the reference frame 31 where the ath first pixel 41 belongs to (Step S140).
The offset calculator 23 divides a plurality of pre-selection frames 32 in the second visual image 12 according to the pixel selection block by taking the pre-selected second pixels 43 as centers, and each of the pre-selection frames 32 includes a plurality of second pixels (Step S142).
The pre-selection frame 32 where the ath second pixel 42 belongs to includes second pixels with pixel coordinates in a range from L(i−2,j−2) to L(i+2,j+2), and the second pixels are arranged in a sequence from left to right and from top to bottom. It is assumed that the current ath second pixel is the 1st second pixel with the pixel coordinates of L (0,0), so that the pixel coordinates of the second pixels in the pre-selection frame 32 where the 1st second pixel belongs to is in a range from L(−2,−2) to L(2,2).
Likewise, when the ath second pixel is the 2nd second pixel with the pixel coordinates of L(1,0), the pixel coordinates of all the second pixels included in the pre-selection frame 32 are in a range from L(−1,−2) to L(3,2). When the ath second pixel is the 10th second pixel with the pixel coordinates of L(10,0), the pixel coordinates of all the second pixels included in the pre-selection frame 32 are in a range from L(8,−2) to L(12,2). When the ath second pixel is the (−9)th second pixel with the pixel coordinates of L(−10,0), the pixel coordinates of all the second pixels included in the pre-selection frame 32 are in a range from L(−12,−2) to L(−8,2).
However, any one of the pre-selection frames 32 may exceed the boundary of the second visual image 12, and in this case, the pixel values of the second pixels at the boundary of the second visual image 12 may be used for compensation. For example, the pixel length and the pixel width of the second visual image 12 are set as (p,q), and if the pre-selection frame 32 exceeds the top boundary of the second visual image 12, the second pixels of (0,0) to (p,0) are used to perform the compensation; if the pre-selection frame 32 exceeds the left boundary of the second visual image 12, the second pixels of (0,0) to (0,q) are used to perform the compensation; if the pre-selection frame 32 exceeds the bottom boundary of the second visual image 12, the second pixels of (0,q) to (p,q) are used to perform the compensation; and if the pre-selection frame 32 exceeds the right boundary of the second visual image 12, the second pixels of (p,0) to (p,q) are used to perform the compensation.
The offset calculator 23 matches positions of all the first pixels of the reference frame 31 respectively with that of all the second pixels in each pre-selection frame 32, calculates grayscale differences between the first pixels and the second pixels with matched positions and sums up the grayscale difference values thereof, so as to obtain a plurality of grayscale difference sums corresponding to the pre-selection frames 32 individually (Step S143).
For example, the offset calculator 23 obtains grayscale values corresponding to all the first pixels, i.e., R(−2,−2) to R(2,2), in the reference frame 31 where the 1st first pixel belongs to. The offset calculator 23 selects any pre-selection frame 32 where a pre-selected second pixel 43 belongs to, for example, when the pre-selection frame 32 where the 11th second pixel belongs to (i.e., the offset pixel value x=10) is selected, the offset calculator 23 obtains the grayscale values of all the second pixels in the pre-selection frame 32 where the 11th second pixel belongs to.
D(x)=[L(i−2+x,j−2)−R(i−2,j−2)]2+[L(i−1+x,j−2)−R(i−1,j−2)]2+K+[L(i+x,j)−R(i,j)]2+A+[L(i+2+x,j+2)−R(i+2,j+2)]2
In this embodiment, the grayscale difference sum of the reference frame 31 of the 1st first pixel and the pre-selection frame 32 of the 11th second pixel is listed as follows:
D(10)=[L(i−2+10,j−2)−R(i−2,j−2)]2+[L(i−1+10,j−2)−R(i−1,j−2)]2+K+[L(i+10,j)−R(i,j)]2+A+[L(i+2+10,j+2)−R(i+2,j+2)]2
Likewise, the grayscale difference sums of the reference frame 31 of the 1st first pixel and the pre-selection frames 32 of the other pre-selected second pixels 43 (i.e., the 10th second pixel to the (−9)th second pixel, having offset pixel values in a range of −10 to 9) are respectively listed as follows:
The offset calculator 23 obtains a minimum grayscale difference value from all the grayscale difference sums, and the pre-selection frame 32 where the minimum grayscale difference value belongs to is the target frame (Step S144).
The offset calculator 23 calculates the offset vector value of the 1st first pixel in the second visual image 12 according to the obtained minimum grayscale difference value (Step S145). In this embodiment, it is assumed that D(−8) is the minimum grayscale difference value, and −8 is the offset vector value of the 1st first pixel in the second visual image 12. That is, the offset vector value is x, and each offset vector value is an integer between −x and x.
The comparator 24 records the offset vector value in an ath data field of the offset vector matrix 13 (Step S150). In this embodiment, the ath first pixel 41 refers to the 1st first pixel, and the obtained offset vector value also refers to the offset value of the 1st first pixel in the second visual image 12, so the comparator 24 records the offset vector value (e.g., −8 as described above) corresponding to the 1st first pixel in the 1st data field of the offset vector matrix 13.
The comparator 24 determines whether the offset vector values of all the ath first pixels 41 have been recorded in the offset vector matrix 13 (Step S160). In this embodiment, the comparator 24 determines whether the ath first pixel 41 currently used to perform the calculation of the offset vector value is the last first pixel of the first visual image 11, i.e., the nth first pixel.
When the comparator 24 determines that the current ath first pixel 41 is not the nth first pixel, the offset vector values of the first pixels in the first visual image 11 are not all recorded. The comparator 24 sets an (a+1)th first pixel as the ath first pixel 41 (Step S163).
In the above embodiment, the ath first pixel 41 is the 1st first pixel, and the (a+1)th first pixel is the 2nd first pixel. After Step S163, the comparator 24 considers the 2nd first pixel as the ath first pixel 41, the 3rd second pixel as the (a+1)th first pixel, the 1st first pixel as the (a−1)th first pixel, and so forth. Thereafter, the comparator 24 performs Step S130 to Step S163 once again until the offset vector values of all the ath first pixels 41 are recorded in the offset vector matrix 13.
When the comparator 24 determines that the ath first pixel 41 is the nth first pixel, it indicates that the offset vector values of the first pixels are all recorded in the offset vector matrix 13. The comparator 24 converts the offset vector matrix 13 into a depth map (Step S162).
Z(i,j)=[A(i,j)+x]*(255/2x),
in which x indicates the offset pixel value, and the Z(i, j) indicates the offset vector matrix Z converted from the offset vector matrix A. Each offset vector value is converted into a grayscale difference value satisfying the grayscale value recording rule, and each grayscale difference value is an integer between 0 and 255. Thereafter, the comparator 24 performs Step S162 to convert the offset vector matrix Z into a Z depth map. However, generally speaking, the offset vector matrix Z may be considered as a numeric Z depth map.
As shown in
Referring to
In this embodiment, the first visual image 11 is a right-eye visual image, and the second visual image 12 is a left-eye visual image. According to the above method of establishing DOF data and the system thereof, the offset vector values of all the first pixels of the first visual image 11 in the second visual image 12 may be calculated and then recorded to form the offset vector matrix A. In order to become available for other manufacturers or image displaying devices, the offset vector matrix A is converted into the offset vector matrix Z satisfying the grayscale format, and then, the offset vector matrix Z is converted into the Z depth map as shown in
Referring to
As shown in
The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.