Patent Application
20170309055
The present technology relates to an image processing apparatus, an image processing method, and a program. In particular, a three-dimensional display and a two-dimensional display are to be carried out naturally and more comfortably.
In recent years, an image display apparatus is provided with a function of carrying out a three-dimensional display. When a three-dimensional image displayed by the above-mentioned image display apparatus is viewed, a focal distance varies even when an angle of convergence is the same as the real world, which becomes a cause of visual fatigue. In particular, if a change of parallax is large, for example, in a case where a certain part on a screen protrudes too much or a case where an object suddenly jumps out during a movie display, a viewer is strained. For this reason, to reduce the strain of the viewer and display the three-dimensional image naturally and more comfortably, a technology of shifting the image in a horizontal direction to adjust the parallax is proposed (for example, see Japanese Unexamined Patent Application Publication No. 2011-055022).
Incidentally, when a two-dimensional display is carried out by using the three-dimensional display material where the parallax is adjusted through the horizontal shift, a movement based on the parallax adjustment may be generated in the displayed image in some cases. For example, in a case where both a still object and a moving object exist on the screen, if a depth of the moving object is largely changed, the image is shifted in the horizontal direction so as to display the three-dimensional image naturally and more comfortably. For this reason, the still object is also moved through the shift of the image in the horizontal direction. Consequently, when the two-dimensional display is carried out by using the parallax-adjusted three-dimensional display material, the still object is displayed with a movement in accordance with the change in the depth of the moving object.
In view of the above, according to an embodiment of this technology, it is desirable to provide an image processing apparatus and an image processing method, and a program with which a three-dimensional display and a two-dimensional display can be carried out naturally and more comfortably.
According to an embodiment of this technology, there is provided an image processing apparatus including: an information obtaining unit configured to obtain parallax adjustment information with regard to a parallax-adjusted three-dimensional display material; and an image transform unit configured to generate a material before the parallax adjustment as a two-dimensional display material from the parallax-adjusted three-dimensional display material on the basis of the parallax adjustment information obtained by the information obtaining unit.
According to the embodiment of this technology, the parallax adjustment information with regard to the parallax-adjusted three-dimensional display material is obtained, and the inverse transform processing is conducted on the parallax-adjusted three-dimensional display material to be restored to the material before the parallax adjustment on the basis of this parallax adjustment information to generate the two-dimensional display material. For example, in a case where an entire image is shifted in the parallax adjustment, the entire image may be shifted in an opposite direction in the inverse transform processing. Also, in a case where an object is shifted in the parallax adjustment, the object may be shifted in the opposite direction in the inverse transform processing.
Also, in the inverse transform processing, an image may be enlarged, or an interpolation processing may be conducted on an area where the image does not exist to eliminate the area without the existence of the image through the inverse transform processing. In a case where the image is enlarged, the image may be enlarged in accordance with an adjustment amount of the parallax adjustment to eliminate the area without the existence of the image. Alternatively, the image may be enlarged in accordance with a maximum adjustment amount of the parallax adjustment to eliminate the area without the existence of the image. Alternatively, the image may be enlarged in accordance with a maximum adjustment amount of the parallax adjustment for each scene to eliminate the area without the existence of the image.
Furthermore, identification information indicating whether or not the three-dimensional display material is the parallax-adjusted material may be obtained, and in a case where a three-dimensional display is carried out, when the identification information obtained by the information obtaining unit indicates that the parallax adjustment is not conducted, the parallax adjustment may be conducted.
According to another embodiment of this technology, there is provided an image processing method including: obtaining parallax adjustment information with regard to a parallax-adjusted three-dimensional display material; and performing an inverse transform processing on the parallax-adjusted three-dimensional display material to be restored to a material before the parallax adjustment on the basis of the obtained parallax adjustment information and generating a two-dimensional display material.
According to still another embodiment of this technology, there is provided a program for causing a computer to execute an image processing on a three-dimensional display material, the image processing including: obtaining parallax adjustment information with regard to a parallax-adjusted three-dimensional display material; and performing an inverse transform processing on the parallax-adjusted three-dimensional display material to be restored to a material before the parallax adjustment on the basis of the obtained parallax adjustment information and generating a two-dimensional display material.
It should be noted that the program according to the embodiment of the present technology is a program that can be provided, for example, to a general-use computer that can execute various program codes by a storage medium or a communication medium in a computer-readable format, for example, a storage medium such as an optical disc, a magnetic disc, or a semiconductor memory or a communication medium such as a network. By providing the above-mentioned program in the computer-readable format, the processing in accordance with the program is realized on the computer.
According to the embodiment of this technology, the parallax adjustment information with regard to the parallax-adjusted three-dimensional display material is obtained, and on the basis of this obtained parallax adjustment information, the material before the parallax adjustment is conducted is generated as the two-dimensional display material from the parallax-adjusted three-dimensional display material. For this reason, in the two-dimensional display, the movement of the still object caused by the parallax adjustment can be avoided during the display, and the three-dimensional display and the two-dimensional display can be carried out naturally and more comfortably.
Hereinafter, embodiments for carrying out the present technology will be described. It should be noted that the description will be given in the following order.
First, a parallax adjustment will be described. A pixel at coordinates (x, y) is set as p(x, y), attention pixels on left and right images are set as p(xL, YL) and p(xR, YR), and pixels on the left and right images after a parallax control are set as p(x′L, y′ L) and p(x′R, y′R).
Herein, in an image transform for a parallax adjustment, a shift processing of shifting relative positions of the left and right images is defined as Expression (1). At this time, a parallax control parameter is a shift amount s.
d=x
R
−x
L (2)
That is, in a case where the three-dimensional image exists in back of the display surface of the three-dimensional display, the parallax d is a positive number, and in a case where the three-dimensional image exists in front of the display surface of the three-dimensional display, the parallax d is a negative number.
Also, in
L
d
:L
s
=d
e:(de−d) (3)
L
d
=L
s
·d
e/(de−d) (4)
Herein, when a consideration is given while the interocular interval de and the sight distance Ls are set as fixed values, the three-dimensional image distance Ld depends on the parallax d as represented by Expression (4). That is, it may be apparent that a three-dimensional appearance (three-dimensional image distance Ld) changes depending on the parallax d.
(B) of
(C) of
Herein, when the pixel at the coordinates (x, y) is set as p(x, y) and the target pixels on the left and right images are set as p(xL, yL) and p(xR, yR), the pixels p(x′L, y′L) and p(x′R, y′R) on the left and right images after the parallax adjustment by the shift amount s can be calculated from Expressions (5) and (6).
p(x′LL, y′L)=p(xL−s/2, yL) (5)
p(x′R, y′R)=p(xR+s/2, yR) (6)
In a case where the shift processing is conducted by the shift amount s, since the parallax changes to (d+s), a three-dimensional image distance L′d after the shift processing is represented by Expression (7).
L′
d
=L
s
·d
e/(de−(d+s)) (7)
That is, in a case where the shift processing is conducted in the direction in which the left and right images are separated from each other (s>0), it is perceived that the three-dimensional image is further moved into the back. At this time, as the shift amount is increased, it is perceived that the three-dimensional image is further moved into the back, and the object further in the back is moved more into the back. On the other hand, in a case where the shift processing is conducted in the direction in which the left and right images are coming closer to each other (s<0), it is perceived that the three-dimensional image is further moved towards the front. At this time, as an absolute value of the shift amount is increased, it is perceived that the three-dimensional image is further moved towards the front, and the object further in the back is moved towards the front more.
Also, it is conceivable that a comfortable range of a three-dimensional vision is a range of a parallax angle at ±1° from a parallax angle on the display surface of the three-dimensional display.
|α−β|≦1°(=π/180) (8)
α=β±(π/180) (9)
A relationship between the sight distance Ls and the parallax angle β corresponds to Expression (10). Therefore, the parallax angle β can be calculated from Expression (11).
(de/2)/Ls=tan(β/2) (10)
β=2tan−1(de/(2Ls)) (11)
Furthermore, a relationship between the three-dimensional image distance Ld and a corresponds to Expression (12).
L
d
=d
e/2tan(α/2) (12)
Therefore, the range in the depth direction where the viewing can be comfortably conducted corresponds to Expression (13).
de/2tan(αmax/2) to de/2tan(αmin/2) (13)
It should be noted that parallax angles αmax and αmin correspond to values in Expressions (14) and (15).
αmax=β+π/180 (14)
αmin=β−π/180 (15)
In this manner, when the shift amount s is adjusted so that the three-dimensional image distance Ld falls within the range indicated by Expression (13), the change in the parallax is suppressed, and the comfortable viewing can be carried out.
Also, in the above-mentioned parallax adjustment, the case has been described in which the left and right images are shifted in the horizontal direction, but the parallax adjustment may be carried out in units of object. For example, by using an image pickup element provided with a ranging function, it is possible to detect distances to respective objects included in an image pickup range. Therefore, the parallax adjustment may also be carried out in units of object on the basis of the ranging results of the respective objects. In addition, parallax amounts with regard to the respective objects are determined from the left and right images through an image processing, and the parallax adjustment may also be carried out in units of object on the basis of the determination results.
When the two-dimensional display is carried out by using the three-dimensional display material where the parallax adjustment for shifting the image in accordance with the three-dimensional image distance is conducted in order that the comfortable viewing of the three-dimensional image can be carried out, the still object is caused to make a movement because of an influence from the transform processing. In view of the above, according to this technology, in a case where the two-dimensional display is carried out by using the three-dimensional display material after the parallax adjustment, a display material before the parallax adjustment is generated on the basis of the parallax adjustment information with regard to the parallax-adjusted three-dimensional display material to be used as the two-dimensional display material.
The information obtaining unit 11 obtains the parallax adjustment information with regard to the three-dimensional display material after the parallax adjustment which is used for the two-dimensional display. For example, in a stream of image data generated by using an encoding system such as H.264/AVC or HEVC, SEI (Supplemental enhancement information) indicating additional information is provided as a syntax element. In a case where parallax adjustment information indicating the shift amount of the horizontal shift is included in this SEI, the information obtaining unit 11 obtains the parallax information from the SEI of the stream. Also, in a case where the parallax adjustment information is filed in a separated manner from the three-dimensional display material, the parallax information is obtained from a file corresponding to the three-dimensional display material used for the two-dimensional display. It should be noted that, for example, in a case where the parallax adjustment is conducted in units of frame, the parallax adjustment information indicates the shift amount of the horizontal shift carried out for each frame, and in a case where the parallax adjustment is conducted in units of GOP (Group Of Pictures), the parallax adjustment information indicates the shift amount of the horizontal shift carried out for each GOP. The information obtaining unit 11 outputs the obtained parallax adjustment information to the image transform unit 12.
The image transform unit 12 performs an inverse transform processing on the parallax-adjusted three-dimensional display material to be restored to a material before the parallax adjustment on the basis of the parallax adjustment information. Herein, the pixel at the coordinates (x, y) is set as p(x, y), the target pixels on the left and right images are set as p(xL, yL) and p(xR, yR), and the pixels on the left and right images after the parallax adjustment are set as p(x′Ly′ L) and p(x′R, y′R). In the parallax adjustment, the processing of shifting the relative positions of the left and right images in the horizontal direction is defined as Expression (1). In this case, the parallax adjustment information indicates the shift amount s. On the basis of the parallax adjustment information, in an inverse manner to Expression (1), the image transform unit 12 calculates the target pixels p(xL, yL) and p(xR, yR) on the left and right images before the parallax adjustment from the pixels p(x′L, y′L) and p(x′R, y′R) on the left and right images after the parallax adjustment to generate the material before the parallax adjustment. Also, the image transform unit 12 outputs the material generated through the inverse transform processing, that is, one of a left view point material and a right view point material before the parallax adjustment as the two-dimensional display material.
When the image transform unit 12 performs the horizontal shift on an entire image in the opposite direction to the direction at the time of the parallax adjustment in the inverse transform processing, an area where the image does not exist is generated on one of the left end and the right end of the image. Therefore, the image transform unit 12 performs an image interpolation processing with respect to the area where the image does not exist. The image transform unit 12 sets the area where the image does not exist, for example, as a black strip display for the image interpolation processing. Also, the image transform unit 12 may perform an image enlargement processing to avoid the generation of the area where the image does not exist for the image interpolation processing. Furthermore, the image transform unit 12 may perform the interpolation of the area where the image does not exist by utilizing an image of a different view point or an image at a different time for the image interpolation processing.
When the image transform unit 12 performs the horizontal shift on an object in the image in the opposite direction to the direction at the time of the parallax adjustment in the inverse transform processing, it is assumed that an area where the image does not exist on an object border (occlusion area) may be generated. Therefore, the image transform unit 12 performs the image interpolation processing on the occlusion area. The image transform unit 12 interpolates the occlusion area by utilizing an image of a different view point or an image at a different time for the image interpolation processing.
Furthermore, in the inverse transform processing, the image transform unit 12 may perform the image enlargement processing so that the area where the image does not exist is not generated even when the horizontal shift is conducted.
In a first operation by the image processing apparatus 10, an operation in a case where the image display is carried out by using the material on which the horizontal shift processing is conducted will be described.
In step ST2, the image transform unit 12 performs the two-dimensional display processing. The image transform unit 12 performs the inverse transform processing on the basis of the parallax adjustment information supplied from the information obtaining unit 11 and generates the material before the parallax adjustment from the three-dimensional display material after the parallax adjustment. Furthermore, the image transform unit 12 outputs the generated material before the parallax adjustment, for example, one of the left view point material and the right view point material as the two-dimensional display material.
In step ST3, the image transform unit 12 performs the three-dimensional display processing. The image transform unit 12 outputs the three-dimensional display material after the parallax adjustment.
When the horizontal shift is conducted in the opposite direction to the direction at the time of the parallax adjustment through the inverse transform processing, the area where the image does not exist is generated. Therefore, in the inverse transform processing, the interpolation is conducted with respect to the area where the image does not exist.
Also, in a case where the parallax adjustment for each object is conducted, for example, an object illustrated in
Furthermore, the image transform unit may avoid the generation of the area where the image does not exist by enlarging the image. In a case where the image enlargement is conducted, the image transform unit 12 sets an enlargement factor in accordance with the shift amount. For example, by increasing the enlargement factor as the shift amount is higher, it is possible to eliminate the area where the image does not exist. Also, if the enlargement factor does not suddenly fluctuate, the user regards the image enlargement as an effect added at the time of editing or the like, it is possible to avoid a reaction that the image involves a sense of discomfort.
In addition, the image transform unit 12 may decide the enlargement factor in accordance with a maximum shift amount. In this case, the area where the image does not exist can be eliminated, and also the image size is unchanged even when the shift amount is changed. The maximum shift amount may previously be set as a maximum amount of the shift amount at the time of the generation of the three-dimensional display material, and also the maximum shift amount may be stored at the time of the generation of the three-dimensional display material. Furthermore, the enlargement factor may be decided in accordance with the maximum shift amount in units of scene as well as in units of material of the three-dimensional display material.
Also, in a case where the parallax adjustment is conducted in units of GOP (Group Of Pictures) and the parallax adjustment information indicates the shift amount conducted for each GOP, the image transform unit 12 performs the inverse transform processing on the basis of the parallax adjustment information and generates the material before the parallax adjustment in units of GOP. In addition, in a case where the parallax adjustment is conducted in units of frame and the parallax adjustment information indicates the shift amount in units of GOP, a linear interpolation is conducted by using the shift amount that is indicated by the parallax adjustment information corresponding to at least one of proceeding and subsequent GOPs to calculate the shift amounts of the respective frames in the GOP. The inverse transform processing may be conducted on the respective frames by using this calculated shift amount to generate the material before the parallax adjustment.
As described above, in a case where the three-dimensional display is carried out, since the parallax-adjusted three-dimensional display material is output, if the three-dimensional display is carried out by using the image data that is output from the image processing apparatus 10, it is possible to display the three-dimensional image more naturally and comfortably. Also, in a case where the two-dimensional display is carried out, one of the left view point material and the right view point material is output which corresponds to the material before the parallax adjustment generated by conducting the inverse transform processing on the three-dimensional display material after the parallax adjustment on the basis of the parallax adjustment information. Therefore, even when the parallax-adjusted three-dimensional display material is used, it is possible to avoid the display of the object that is still in the two-dimensional display without the movement. Also, since the image interpolation processing is conducted in the two-dimensional display, it is possible to eliminate the area without the existence of the image. It should be noted that in a case where the parallax adjustment is conducted by switching to or selecting the entire image or the units of object, the information indicating whether the parallax adjustment is conducted by selecting the entire image or the units of object and the information on the object on which the parallax adjustment is conducted are included in the parallax adjustment information.
Incidentally, in the first operation, the operation in a case where the image display is carried out by using the three-dimensional display material after the parallax adjustment has been described. In the image processing apparatus 10, not only the three-dimensional display material after the parallax adjustment but also the three-dimensional display material before the parallax adjustment may be utilized. In a second operation, a case in which both the three-dimensional display material before the parallax adjustment and the three-dimensional display material after the parallax adjustment can be utilized will be described. It should be noted that the material used by the image processing apparatus 10 has identification information indicating whether or not the three-dimensional display material is a material where the parallax adjustment is conducted. The identification information is provided, for example, as a flag in a header of the three-dimensional display material. Herein, in a case where the flag is “0”, the three-dimensional display material is a material where the parallax adjustment is not conducted. On the other hand, in a case where the flag is “1”, the three-dimensional display material is the material where the parallax adjustment is conducted. Also, in a case where the flag is “1”, the parallax adjustment information indicating an adjustment amount for the parallax adjustment is provided.
In step ST12, the image transform unit 12 determines whether or not the two-dimensional display is carried out. In a case where the two-dimensional display is carried out, for example, in a case where the two-dimensional display is instructed from the user or a case where the display apparatus used for the image display does not have the three-dimensional display function, the image transform unit 12 proceeds to step ST13. Also, for example, in a case where the three-dimensional display is instructed from the user or a case where the display apparatus has the three-dimensional display function, the image transform unit 12 proceeds to step ST14.
In step ST13, the image transform unit 12 performs the two-dimensional display processing. The image transform unit 12 performs the inverse transform processing on the basis of the parallax adjustment information supplied from the information obtaining unit 11 and generates the material before the parallax adjustment from the three-dimensional display material after the parallax adjustment. Furthermore, the image transform unit 12 outputs one of the left view point material and the right view point material corresponding to the generated material before the parallax adjustment as the two-dimensional display material.
In step ST14, the image transform unit 12 performs the three-dimensional display processing. The image transform unit 12 outputs the three-dimensional display material after the parallax adjustment.
When the image transform unit 12 proceeds from step ST11 to step ST15, the image transform unit 12 determines whether or not the two-dimensional display is carried out. In a case where the two-dimensional display is carried out, for example, in a case where the two-dimensional display is instructed from the user or a case where the display apparatus used for the image display does not have the three-dimensional display function, the image transform unit 12 proceeds to step ST16. Also, for example, in a case where the three-dimensional display is instructed from the user or a case where the display apparatus has the three-dimensional display function, the image transform unit 12 proceeds to step ST17.
In step ST16, the image transform unit 12 performs the two-dimensional display processing. Since the three-dimensional display material is a material where the parallax adjustment is not conducted, the image transform unit 12 outputs one of the left view point material and the right view point material of the three-dimensional display material as the two-dimensional display material.
In step ST17, the image transform unit 12 performs the three-dimensional display processing. Since the three-dimensional display material is the material where the parallax adjustment is not conducted, so that the natural and more comfortable three-dimensional display can be carried out, the image transform unit 12 performs the parallax adjustment on the three-dimensional display material. Furthermore, the image transform unit 12 outputs the three-dimensional display material after the parallax adjustment.
When the above-mentioned processing is conducted, the image processing apparatus 10 can output the material suitable to the two-dimensional display or the three-dimensional display in accordance with whether or not the three-dimensional display material used for the image display is the parallax-adjusted material. Therefore, irrespective of whether or not the three-dimensional display material used for the image display is the parallax-adjusted material, in a case where the three-dimensional display is carried out, the three-dimensional image can be displayed naturally and more comfortably. Furthermore, in a case where the two-dimensional display is carried out, it is possible to avoid a situation in which the still object is displayed with a movement.
Also, the series of processings described in the specification can be executed by hardware, software, or a composite structure of the both. In a case where the processing is executed by the software, a program where a processing sequence is recorded is installed into a memory within a computer that is incorporated in dedicated-use hardware for execution. Alternatively, the program can also be installed into a general-use computer that can execute various processings for execution.
A RAM 803 (Random Access Memory) appropriately stores a computer program executed by the CPU 801, data, and the like. The CPU 801, the ROM 802, and the RAM 803 are mutually connected by a bus 804.
An input and output interface 805 is also connected to the CPU 801 via the bus 804. An input unit 806 such as a touch panel, a key board, a mouse, and a micro phone and an output unit 807 composed of a display or the like are connected to the input and output interface 805. The CPU 801 executes various processings corresponding to instructions that are input from the input unit 806. Then, the CPU 801 outputs results of the processings to the output unit 807.
The recording unit 808 connected to the input and output interface 805 is composed, for example, of a hard disk and records the computer program executed by the CPU 801 and various pieces of data. A communication unit 809 communicates with an external apparatus by being connected to the internet or a local area network via a wired or wireless communication medium. Also, the computer apparatus 80 may obtain a computer program via the communication unit 809 to be recorded in the ROM 802 or the recording unit 808.
When a removable medium 85 such as a magnetic disc, an optical disc, an opto-magnetic disc, or a semiconductor disc is mounted to a drive 810, the drive 810 drives the removable medium to obtain the recorded computer program, data, and the like. The obtained computer program and data are transferred to the ROM 802, the RAM 803, or the recording unit 808 as appropriate.
The CPU 801 reads out and executes the computer program for carrying out the above-mentioned series of processings, and the above-mentioned series of processings are conducted on the material recorded on the recording unit 808 and the removable medium 85 and the material supplied via the communication unit 809.
It should be noted that the present technology is not to be considered limited to the above-mentioned embodiments. The embodiments of the present technology disclose the present technology by way of exemplifications, and it should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof. That is, to determine the gist of the present technology, the scope of the appended claims should be referred to.
Also, the image processing apparatus of the present technology can adopt the following configurations.
(1) An image processing apparatus including: an information obtaining unit configured to obtain parallax adjustment information with regard to a parallax-adjusted three-dimensional display material; and an image transform unit configured to perform an inverse transform processing on the parallax-adjusted three-dimensional display material to be restored to a material before the parallax adjustment on the basis of the parallax adjustment information obtained by the information obtaining unit and generate a two-dimensional display material.
(2) The image processing apparatus according to (1), in which the image transform unit enlarges an image or performs an interpolation processing on an area where the image does not exist to eliminate the area without the existence of the image through the inverse transform processing.
(3) The image processing apparatus according to (2), in which the image transform unit enlarges the image in accordance with an adjustment amount of the parallax adjustment to eliminate the area without the existence of the image.
(4) The image processing apparatus according to (2), in which the image transform unit enlarges the image in accordance with a maximum adjustment amount of the parallax adjustment to eliminate the area without the existence of the image.
(5) The image processing apparatus according to (4), in which the image transform unit enlarges the image in accordance with a maximum adjustment amount of the parallax adjustment for each scene to eliminate the area without the existence of the image.
(6) The image processing apparatus according to any one of (1) to (5), in which in a case where an entire image is shifted in the parallax adjustment, the image transform unit shifts the entire image in an opposite direction in the inverse transform processing.
(7) The image processing apparatus according to any one of (1) to (5), in which in a case where an object is shifted in the parallax adjustment, the image transform unit shifts the object in an opposite direction in the inverse transform processing.
(8) The image processing apparatus according to any one of (1) to (7), in which the information obtaining unit obtains identification information indicating whether or not the three-dimensional display material is the parallax-adjusted material, and in which in a case where a three-dimensional display is carried out, when the identification information obtained by the information obtaining unit indicates that the parallax adjustment is not conducted, the image transform unit performs the parallax adjustment.
The present disclosure contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2012-011063 filed in the Japan Patent Office on Jan. 23, 2012, the entire contents of which are hereby incorporated by reference.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2012-011063 | Jan 2012 | JP | national |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 13742547 | Jan 2013 | US |
| Child | 15452421 | US |
