IMAGE GENERATION APPARATUS, IMAGE GENERATION METHOD, AND PROGRAM

Abstract

Provided is an image generation technology capable of suppressing unpleasant feelings caused by fluctuation in image quality caused by the viewer's viewpoint movement. The image generation technology includes an image generation unit configured to generate a pseudo viewpoint image Iφk and a pseudo viewpoint image I−φk, by using disparity inducing edge Dφk having a phase difference φk from a viewpoint image I, where φk(1≤k≤K) is set as a real number satisfying 0<φ1< . . . <φK≤π/2, an output image generation unit configured to generate an output image Outm(1≤m≤2K−1), from the pseudo viewpoint image I(m)(1≤m≤2K+1), the viewpoint image I is set as a pseudo viewpoint image Iφ0, where the pseudo viewpoint images Iφk, I−φk (0≤k≤K) are arranged in a sequence of Iφk, Iφ(K−1), . . . , Iφ1, Iφ0(=I), I−φ1, . . . , and I−φK is set as I(1), I(2), . . . , I(K), I(K+1), I(K+2), . . . , I(2K+1), and the output image Outm and the output image Outm+1(1≤m≤2K−2) include a phase modulation component that is canceled out when synthesized and visually recognized.

Description

TECHNICAL FIELD

The present disclosure relates to an image generation technology, and particularly to an image generation technology for a viewer to perceive an image from a viewpoint located at an intermediate position between adjacent viewpoints.

BACKGROUND ART

A technology described in Non-Patent Literature 1, for example, for synthesizing luminance of the images (viewpoint images) from adjacent viewpoints smoothly based on the position of an intermediate viewpoint between the adjacent viewpoints (intermediate viewpoint position), and enabling a viewer to perceive an image from an intermediate viewpoint, is called “linear blending”.

The linear blending is advantageously used to reduce the number of projectors required for image display, because there is no need to prepare the image from the intermediate viewpoint position. In other words, the cost associated with image display can be reduced.

CITATION LIST

Non Patent Literature

Non-Patent Literature 1: Andrew Jones, Jonas Unger, Koki Nagano, Jay Busch, Xueming Yu, Hsuan-Yueh Peng, Oleg Alexander, Mark Bolas, Paul Debevec, “An Automultiscopic Projector Array for Interactive Digital Humans”, SIGGRAPH' 15, ACM SIGGRAPH 2015 Emerging Technologies Article No. 6, 2015.

SUMMARY OF THE INVENTION

Technical Problem

However, with the current linear blending, image quality may be degraded due to image multiplexing caused by overlapping images at the intermediate viewpoint position. In addition, while images at adjacent viewpoints that correspond to the positions of the projectors show no degradation in quality, images at the intermediate viewpoint positions show degradation in quality. When a viewer moves its viewpoint, a fluctuation in image quality occurs, causing the viewer to experience unpleasant feelings.

A description will be made with reference to FIG. 1. At a viewpoint A, a viewer visually recognizes only single viewpoint image (image 1) and therefore experiences a good image quality. However, at a viewpoint B, which is the intermediate viewpoint position, the viewer visually recognizes an image that is synthesized (visible at the same time) from the two viewpoint images (image 1 and image 2) and thus experiences a poor image quality due to overlapping. As a result, when the viewer moves in a sequence of A→B→C, the image quality fluctuates from good→poor→good.

Thus, an object of the present disclosure is to provide an image generation technology capable of suppressing unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Means for Solving the Problem

An aspect of the present disclosure includes: an image generation unit configured to generate, from a viewpoint image I and a disparity map Dmap corresponding to the viewpoint image I, a disparity inducing edge D_φk having a phase difference φk from the viewpoint image I for k=1, . . . , K, generate a pseudo viewpoint image I_φk, by adding the disparity inducing edge D_φk to the viewpoint image I, and generate a pseudo viewpoint image I_−φk by adding a polarity-inversed image of the disparity inducing edge D_φk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2; and

an output image generation unit configured to generate an output image Out_m (1≤m≤2K−1) from the pseudo viewpoint image I^(m) (1≤m≤2K+1), where the viewpoint image I is set as the pseudo viewpoint image I_φ0, and the pseudo viewpoint images I_φk and I_−φk (0≤k≤K) being arranged in a sequence of I_φK, I_φ(K−1), . . . , I_φ1, I_φ0 (=I), I_−φ1, . . . , and I_−φK are set as I⁽¹⁾, I⁽²⁾, . . . , I^(K), I^(K+1), I^(K+2), . . . , and I^(2K+1);wherein the output image Out_m and output image Out_m+1 (1≤m≤2K−2) include a phase modulation component that is canceled out when synthesized and visually recognized.

Another aspect of the present disclosure includes: an image generation unit configured to generate, from a viewpoint image I and a viewpoint image I′ captured from a viewpoint position adjacent to a viewpoint position at which the viewpoint image I is captured, a disparity inducing edge D_φk having a phase difference φk from the viewpoint image I for k=1, K, generate a pseudo viewpoint image I_φk, by adding the disparity inducing edge D_φk to the viewpoint image I, and generate a pseudo viewpoint image I_−φk by adding a polarity-inversed image of the disparity inducing edge D_φk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2; and

an output image generation unit configured to generate an output image Out_m (1≤m≤2K−1) from the pseudo viewpoint image I^(m) (1≤m≤2K+1), where the viewpoint image I is set as a pseudo viewpoint image I_φ0, and the pseudo viewpoint images I_φk and I_−φk (0≤k≤K) being arranged in a sequence of I_φK, I_φ(K−1), I_φ1, I_φ0 (=I), I_−φ1, . . . , and I_−φK are defined as I⁽¹⁾, I⁽²⁾, . . . , I^(K), I^(K+1), I^(K+2) and I^(2K+1);wherein the output image Out_m and output image Out_m+1 (1≤m≤2K−2) include a phase modulation component that is canceled out when synthesized and visually recognized.

Another aspect of the present disclosure includes: an output image generation unit configured to generate an output image Out_m (1≤m≤M−1) from a viewpoint image I^(m) and a disparity map Dmap^(m) (1≤m≤M), where I⁽¹⁾, I⁽²⁾, . . . , I^(M) are set as viewpoint images arranged in a sequence of viewpoint positions, and Dmap^(m) (1≤m≤M) is set as a disparity map corresponding to the viewpoint image I^(m),

wherein the output image Out_m and output image Out_m+1 (1≤m≤M−2) include a phase modulation component that is canceled out when synthesized and visually recognized.

Another aspect of the present disclosure includes: an output image generation unit configured to generate an output image Out_m (1≤m≤M−1) from a viewpoint image I^(m) (1≤m≤M+1), where I⁽¹⁾, I⁽²⁾, . . . , I^(M) are set as viewpoint images arranged in a sequence of viewpoint positions,

wherein the output image Out_m and output image Out_m+1 (1≤m≤M−2) include a phase modulation component that is canceled out when synthesized and visually recognized.

Another aspect of the present disclosure includes: an output image generation unit configured to generate an output image Out_j (1≤j≤J−1) from a viewpoint image IO) (1≤j≤2J+1), where I⁽¹⁾, I⁽²⁾, . . . . I^(2J+1) are set as viewpoint images arranged in a sequence of viewpoint positions, wherein the output image Out_j and output image Out_j+1 (1≤j≤J−2) include a phase modulation component that is canceled out when synthesized and visually recognized.

Another aspect of the present disclosure includes: an image generation unit configured to, from a viewpoint image I and a disparity map Dmap corresponding to the viewpoint image I, generate a disparity inducing edge D_φk having a phase difference φk from the viewpoint image I for k=1, . . . . K, generate a pseudo viewpoint image I_φk, by adding the disparity inducing edge D_φk to the viewpoint image I, and generate a pseudo viewpoint image I_−φk by adding a polarity-inversed image of the disparity inducing edge D_φk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2,

wherein the viewpoint image I is set as a pseudo viewpoint image I_φ0, and for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, a pseudo viewpoint image I_φk and a pseudo viewpoint image I_φ(k+1) are viewpoint images at viewpoint positions adjacent to each other, and are images used for presenting an image obtained by synthesizing luminance of at least two images of the pseudo viewpoint image I_φk and the pseudo viewpoint image I_φ(k+1) as an image at a viewpoint located at an intermediate position between the viewpoint positions adjacent to each other.

Another aspect of the present disclosure includes: an image generation unit configured to generate, from a viewpoint image I and a viewpoint image I′ captured from a viewpoint position adjacent to a viewpoint position at which the viewpoint image I is captured, a disparity inducing edge D_φk having a phase difference φk from the viewpoint image I for k=1, . . . , K, generate a pseudo viewpoint image I_φk, by adding the disparity inducing edge D_φk to the viewpoint image I, and generate a pseudo viewpoint image I_−φk by adding a polarity-inversed image of the disparity inducing edge D_φk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2,

wherein the viewpoint image I is set as a pseudo viewpoint image I_φ0, and, for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, the pseudo viewpoint image I_φk and the pseudo viewpoint image I_φ(k+1) are viewpoint images in viewpoint positions adjacent to each other, and are images used for presenting an image obtained by synthesizing luminance of at least two images of the pseudo viewpoint image I_φk and the pseudo viewpoint image I_φ(k+1) as an image at a viewpoint located at an intermediate position between the viewpoint positions adjacent to each other.

Effects of Invention

According to the present disclosure, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a related art (linear blending).

FIG. 2 is a block diagram illustrating a configuration of an image generation apparatus 1.

FIG. 3 is a flowchart illustrating an operation of the image generation apparatus 1.

FIG. 4 is a block diagram illustrating a configuration of an image generation unit 10.

FIG. 5 is a diagram illustrating a state of fluctuation in image quality associated with the viewpoint movement.

FIG. 6 is a block diagram illustrating a configuration of an image generation apparatus 2.

FIG. 7 is a flowchart illustrating an operation of the image generation apparatus 2.

FIG. 8 is a block diagram illustrating a configuration of an image generation unit 20.

FIG. 9 is a block diagram illustrating a configuration of an image generation apparatus 3.

FIG. 10 is a flowchart illustrating an operation of the image generation apparatus 3.

FIG. 11 is a block diagram illustrating a configuration of an output image generation unit 30.

FIG. 12 is a flowchart illustrating an operation of the output image generation unit 30.

FIG. 13 is a block diagram illustrating a configuration of an image-for-synthesis generation unit 31.

FIG. 14 is a diagram illustrating the technology of the present application.

FIG. 15 is a block diagram illustrating a configuration of an image generation apparatus 4.

FIG. 16 is a flowchart illustrating an operation of the image generation apparatus 4.

FIG. 17 is a block diagram illustrating a configuration of an image generation apparatus 5.

FIG. 18 is a flowchart illustrating an operation of the image generation apparatus 5.

FIG. 19 is a block diagram illustrating a configuration of an image generation apparatus 6.

FIG. 20 is a flowchart illustrating an operation of the image generation apparatus 6.

FIG. 21 is a block diagram illustrating a configuration of an image generation apparatus 7.

FIG. 22 is a flowchart illustrating an operation of the image generation apparatus 7.

FIG. 23 is a diagram illustrating a relationship between a viewpoint image and an output image when J=5.

FIG. 24 is a block diagram illustrating a configuration of an image-for-synthesis generation unit 71.

FIG. 25 is a diagram illustrating a result of an experiment.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that components having the same function are given the same reference numerals, and redundant descriptions are omitted.

First Embodiment

Hereinafter, an image generation apparatus 1 will be described with reference to FIG. 2 and FIG. 3. FIG. 2 is a block diagram illustrating a configuration of the image generation apparatus 1. FIG. 3 is a flowchart illustrating operations of the image generation apparatus 1. As illustrated in FIG. 2, the image generation apparatus 1 includes an image generation unit 10 and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 1.

The image generation apparatus 1 receives, as an input, the viewpoint image I and a disparity map Dmap corresponding to the viewpoint image I, generates 2K pseudo viewpoint images (K is an integer of 1 or greater), I_φk, I_−φk (1≤k≤K) from the viewpoint image I and the disparity map Dmap, and outputs the pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents a phase difference between the viewpoint image I and the disparity inducing edge D_φk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. The disparity inducing edge D_φk is an image obtained by modulating a phase to make a phase difference from the viewpoint image I to be φk. Further, the pseudo viewpoint image I_φk, I_−φk are images both at the viewpoint positions different from the viewpoint position of the viewpoint image I (see FIG. 5). In the following, it is presented that φ0=0, I_φk=I, and the viewpoint image I may be referred to as a pseudo viewpoint image I_φ0.

Note that the disparity in the disparity map Dmap corresponding to the viewpoint image I may not be a disparity obtained at the viewpoint position where the viewpoint image I is captured. For example, the desired disparity may be used.

The operation of the image generation apparatus 1 will be described in accordance with FIG. 3.

In step S10, the image generation unit 10 receives, as an input, the viewpoint image I and a disparity map Dmap of the viewpoint image I, generates pseudo viewpoint image I_φk, I_−φk, from the viewpoint image I and the disparity map Dmap for k=1, . . . , K, and outputs the pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I. A procedure for generating the pseudo viewpoint images I_φk, I_−φk will be described. First, the image generation unit 10 generates a disparity inducing edge D_φk, which is a phase modulated image having a phase difference φk from the viewpoint image I, from the viewpoint image I and the disparity map Dmap. Next, the image generation unit 10 generates a pseudo viewpoint image I_φk by adding the disparity inducing edge D_φk to the viewpoint image I (I_φk=I+D_φk), and generates a pseudo viewpoint image I_−φk by adding the polarity-inversed image of the disparity inducing edge D_φk to the viewpoint image I (I_−φk=I−D_φk). The image generation unit 10 sequentially generates the pseudo viewpoint images I_φk, I_−φk (1≤k≤K).

Note that the image generation apparatus 1 may further include a projection unit (not illustrated) that projects the pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I which are output at S10. The projection unit arranges the pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I in the sequence of phase differences from I_φk, I_φk(K−1), . . . , I_φ1, I, I_−φ1, . . . , I_−φK, that is, in the sequence of the viewpoint positions, and projects the arranged images onto a screen.

Further, for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, the pseudo viewpoint image I_φk and the pseudo viewpoint image I_φ(k+1) are viewpoint images at viewpoint positions adjacent to each other, and the synthesized luminance of at least two images of the pseudo viewpoint image I_φk and the pseudo viewpoint image I_φ(k+1) may be used to present the images at viewpoints located at an intermediate position between the viewpoint positions adjacent to each other.

Hereinafter, the configuration and operation of the image generation unit 10 will be described in detail with reference to FIG. 4. Here, a case in which the phase difference is π/2 will be described. In addition, in the description, a viewpoint image is referred to as an original image, a disparity inducing edge is referred to as a phase modulated image, and a pseudo viewpoint image is referred to as an added image.

Image Generation Unit 10

As illustrated in FIG. 4, the image generation unit 10 includes decomposition units 111, 112, spatial frequency domain conversion units 121, 122, spatial domain conversion units 123, 124, a weight calculation unit 13, a phase operation unit 14, a weighting unit 15, a reconstruction unit 16, and superposition units 17, 18.

The processing by the image generation unit 10 will be described.

The original image I_C and the disparity map Dmap are input to the image generation unit 10. The original image I_C is a two-dimensional array having I_C (x, y) as elements (pixels). The original image I_C may be a still image, an image of individual frame of a moving image, an image read from an external recording device, or an image read from the recording unit 90. The disparity map Dmap is a two-dimensional array (two-dimensional grayscale image) having d(x, y) as elements (pixels). Here, the size of Dmap is the same as the size of the original image I_C, x represents the horizontal coordinate of the spatial domain, y represents the vertical coordinate of the spatial domain, and it is satisfied that x_min≤x≤x_max, y_min≤y≤y_max, x_min<x_max, y_min<y_max. d(x, y) is one of a positive value, a negative value, and zero. In Dmap, in a region where d(x, y)>0, a disparity (crossed disparity) causes the region to be seen nearer than the display surface or the screen surface, and in a region where d(x, y)<0, a disparity (uncrossed disparity) causes the region to be seen farther than the display surface or the screen surface. The larger the absolute value |d(x, y)|, the greater the disparity. Such a Dmap can provide a desired disparity to each image region.

The original image I_C is input to the spatial frequency domain conversion unit 121. The spatial frequency domain conversion unit 121 converts the original image I_C into the original image I_C{tilde over ( )} in the spatial frequency domain, and outputs original image I_C{tilde over ( )}. Here, the original image I_C{tilde over ( )} in the spatial frequency domain is a two-dimensional array having I_C{tilde over ( )}(ω_x, ω_y) as elements. ω_x represents the spatial frequency in the horizontal direction, and ω_y represents the spatial frequency in the vertical direction. For conversion from the original image I_C to the original image I_C{tilde over ( )}, for example, discrete Fourier transform can be used. It should be noted that the superscript “{tilde over ( )}” of “I_C{tilde over ( )}” should be given directly above “I”. However, “{tilde over ( )}” may be given at the upper right of “I_C” due to restrictions on the description in the specification (step S121).

The original image I_C{tilde over ( )} in the spatial frequency domain is input to the decomposition unit 112. The decomposition unit 112 applies a complex steerable filter sequence Ψ to the original image I_C{tilde over ( )} and obtains and outputs a complex steerable pyramid. Here, the steerable filter sequence Ψ includes a steerable filter Ψ_{λ, μ}, corresponding to the spatial frequency band λ and the orientation band μ. Here, λ is an integer index corresponding to a spatial frequency band having a predetermined width, and μ is an integer index corresponding to an orientation band having a predetermined width. The conditions, λ_min≤λ≤λ_max, μ_min≤μ≤μ_max, λ_min<λ_max, μ_min<μ_max, are satisfied. A smaller λ corresponds to a lower frequency band. As described below, for all combinations of λ and μ, the decomposition unit 112 multiplies the steerable filter Ψ_{λ, μ} to the original image I_C{tilde over ( )} and obtains and outputs a complex steerable pyramid S_{Cλ, μ}{tilde over ( )} corresponding to each spatial frequency band λ and each orientation band μ.

[Equation 1]

{tilde over (S)} _{C λ,μ} =Ĩ _CΨ_λ,μ  (1)

It should be noted that the superscript “{tilde over ( )}” of “S_{Cλ, μ}{tilde over ( )}” should be given directly above “S” (see Equation (1)). However, “{tilde over ( )}” may be given at the upper right of “S_{Cλ, μ}” due to restrictions on the description in the specification. Further, the subscript “Cλ, μ” of “S_{Cλ, μ}{tilde over ( )}” should be given as in “C_{λ, μ}{tilde over ( )}” (see Equation (1)). However, it may be given as “Cλ, μ” due to restrictions on the description in the specification (step S112).

The complex steerable pyramid S_{Cλ, μ}{tilde over ( )} is input to the spatial domain conversion unit 123. The spatial domain conversion unit 123 converts the complex steerable pyramid S_{Cλ, μ}{tilde over ( )} into the complex steerable pyramid S_{Cλ, μ} in the spatial domain, and outputs a complex steerable pyramid S_{Cλ, μ}. For example, discrete inverse Fourier transform can be used for the conversion from the complex steerable pyramid S_{Cλ, μ}{tilde over ( )} to the complex steerable pyramid S Each complex steerable pyramid S_{Cλ, μ} is a two-dimensional array having S_{Cλ, μ}(x, y) as elements (pixels) (step S123).

The complex steerable pyramid S_{Cλ, μ} in the spatial domain is input to the phase operation unit 14. The phase operation unit 14 extracts an imaginary part Im [S_{Cλ, μ}(x, y)] of each complex steerable pyramid S_{Cλ, μ}, and obtains and outputs each phase shift image S_{C′λ, μ} corresponding to each spatial frequency band λ and each orientation band μ as follows.

$\begin{array}{cc}\left[\mathrm{Equation}2\right]& \\ S_{C_{\lambda ,\mu }^{\prime }}\left(\varkappa ,y\right)=\{\begin{array}{cc}\mathrm{Im}\left[S_{C_{\lambda ,\mu }}\left(\varkappa ,y\right)\right]& \mathrm{if}{\theta }_{\mu }\le \pi /2\\ -\mathrm{Im}\left[S_{C_{\lambda ,\mu }}\left(\varkappa ,y\right)\right]& \mathrm{if}{\theta }_{\mu }>\pi /2\end{array}& \left(2\right)\end{array}$

Here, each phase shift image S_{C′λ, μ} is a two-dimensional array having S_{C′λ, μ}(x, y) as elements (pixels). The subscript “C′λ, μ” of “S_{C′λ, μ}” should be given as “C′_{λ, μ}” (see Equation (2)). However, it may be given as “C′λ, μ” due to restrictions on the description in the specification. θ_μ refers to the orientation (peak orientation) corresponding to the component having the strongest power among the orientations included in the orientation band μ. An example of θ_μ is the orientation of the center of the orientation band μ. θ_μ is the angle formed by the lattice in the spatial domain consisting of certain spatial frequency components with respect to the vertical orientation, and satisfies 0≤θ_μ≤π. The phase of the phase shift image S_{C′λ, μ}, is obtained by shifting the phase of the sine wave component corresponding to the spatial frequency band λ and the orientation band μ of the original image I_C by 0.5π [rad] in the positive direction. In Equation (2), the reason why the polarity is inversed when θ_μ>π/2 is because the phase is shifted by 0.5π [rad] in the same direction (in the positive direction) for both the range of 0≤θ_μ≤π/2 and the range of π≥θ_μ>π/2 (step S14).

The disparity map Dmap which has been input to the image generation unit 10 is input to the decomposition unit 111. The decomposition unit 111 applies a Gaussian filter sequence to the disparity map Dmap to obtain and output a Gaussian pyramid G_Dλ(x, y) of the disparity map Dmap. The Gaussian pyramid G_Dλ includes spatial frequencies corresponding to spatial frequency band λ and lower spatial frequency bands than the spatial frequency band λ. That is, G_Dλ(x, y) represents the value of each coordinate (x, y) of the Gaussian pyramid of the disparity map Dmap corresponding to the spatial frequency band λ corresponding to ωλ and the spatial frequency band lower than the spatial frequency band λ. The subscript “Dλ,” of “G_Dλ” should be given as “D_λ”. However, it may be given as “Dλ” due to restrictions on the description in the specification (step S111).

The Gaussian pyramid G_Dλ(x, y) input to the weight calculation unit 13. The weight calculation unit 13 obtains and outputs a weight image A_{λ, μ} which is a two-dimensional array having each weight A_{λ, μ}(x, y) as an element as follows.

$\begin{array}{cc}\left[\mathrm{Equation}3\right]& \\ A_{\lambda ,\mu }\left(\varkappa ,y\right)=\tan \frac{G_{D_{\lambda }}\left(\varkappa ,y\right){\omega }_{\lambda }\cos {\theta }_{\mu }}{2}& \left(3\right)\end{array}$

Here, ωλ means a spatial frequency (peak spatial frequency) corresponding to a component having the strongest power among the spatial frequencies included in the spatial frequency band λ. An example of ωλ is the spatial frequency at the center of the spatial frequency band λ (step S13).

The phase shift image S_{C′λ, μ} and weight image A_{λ, μ} are input to the weighting unit 15. As follows, the weighting unit 15 multiplies each A_{λ, μ}(x, y) of the weight image A_{λ, μ} to each element S_{C′λ, μ}(x, y) of the phase shift image S_{C′λ, μ} and obtains and outputs a weighted image S{circumflex over ( )}_{C′λ, μ} which is a two-dimensional array having each S{circumflex over ( )}_{C′λ, μ}(x, y) as an element.

$\begin{array}{cc}\left[\mathrm{Equation}4\right]& \\ {\hat{S}}_{C_{\lambda ,\mu }^{\prime }}\left(\varkappa ,y\right)=A_{\lambda ,\mu }\left(\varkappa ,y\right)S_{C_{\lambda ,\mu }^{\prime }}\left(\varkappa ,y\right)& \left(4\right)\end{array}$

It should be noted that the superscript “{circumflex over ( )}” of “S{circumflex over ( )}” should be given directly above “S” (see Equation (4)). However, “{circumflex over ( )}” may be given at the upper right of “S” due to the restrictions on the description in the specification (step S15).

The weighted image S{circumflex over ( )}_{C′λ, μ} is input to the spatial frequency domain conversion unit 122. The spatial frequency domain conversion unit 122 converts the weighted image S{circumflex over ( )}_{C′λ, μ} into the weighted image S{circumflex over ( )}{tilde over ( )}_{C′λ, μ} in the spatial frequency domain, and outputs the weighted image S{circumflex over ( )}{tilde over ( )}_{C′λ, μ}. Note that, it may be described as “S{circumflex over ( )}{tilde over ( )}_{C′λ, μ}” due to restrictions on the description in the specification, but “S{circumflex over ( )}{tilde over ( )}_{C′λ, μ}” is synonymous with [Equation 5].

$\begin{array}{cc}\left[\mathrm{Equation}5\right]& \\ {\tilde{\hat{S}}}_{C_{\lambda ,\mu }^{\prime }}& \end{array}$

Note that, for example, a discrete Fourier transform can be used for the conversion from the weighted image S{circumflex over ( )}_{C′λ, μ} to the weighted image S{circumflex over ( )}{tilde over ( )}_{C′λ, μ} (step S122).

The weighted image S{circumflex over ( )}{tilde over ( )}_{C′λ, μ} is input to the reconstruction unit 16. The reconstruction unit 16 applies the above-described steerable filter sequence Ψ to the weighted image S{circumflex over ( )}{tilde over ( )}_{C′λ, μ} and obtains and outputs the phase modulated image I_D{tilde over ( )} in the spatial frequency domain as follows (step S16).

$\begin{array}{cc}\left[\mathrm{Equation}6\right]& \\ {\tilde{I}}_D=\sum _{\lambda ,\mu }{\Psi }_{\lambda ,\mu }{\tilde{\hat{S}}}_{C_{\lambda ,\mu }^{\prime }}& \left(5\right)\end{array}$

The phase modulated image I_D{tilde over ( )} in the spatial frequency domain is input to the spatial domain conversion unit 124. The spatial domain conversion unit 124 converts the phase modulated image I_D{tilde over ( )} in the spatial frequency domain into the phase modulated image I_D in the spatial domain, and outputs the phase modulated image I_D. Here, the phase modulated image I_D is a two-dimensional array having I_D (x, y) as elements (pixels). For conversion from the phase modulated image I_D{tilde over ( )} to the phase modulated image I_D, for example, a discrete inverse Fourier transform can be used (step S124).

The superposition unit 17 receives the original image I_C and the phase modulated image I_D as inputs, and obtains and outputs the added image I_R (image A), by superimposing a polarity-inversed image (phase modulation component a) of the phase modulated image I_D on the original image I_C. When the value of the disparity map Dmap is positive (crossed disparity) in a certain region, the phase of the phase modulated image I_D in the corresponding region is shifted from the phase of the original image I_C by 0.5π [rad] in the positive direction. Thus, the phase of the polarity-inversed image (phase modulation component a) of the phase modulated image I_D is obtained by shifting the phase of the original image I_C by 0.5π [rad] in the negative direction. On the other hand, when the value of the disparity map Dmap is negative in a certain region, the phase of the phase modulated image I_D is obtained by shifting the phase of the original image I_C by 0.5π [rad] in the negative direction. Thus, the phase of the polarity-inversed image (phase modulation component a) of the phase modulated image I_D in this region is obtained by shifting the phase of the original image I_C by 0.5π [rad] in the negative direction. The pixel value I_R (x, y) of each coordinate (x, y) of the added image I_R is obtained by adding the pixel value I_D (x, y) of each coordinate (x, y) of the polarity-inversed image of the phase modulated image I_D to the pixel value I_C (x, y) of each coordinate (x, y) of the original image I_C (I_R (x, y)=I_C (x, y) I_D (x, y)). Note that the polarity-inversed image of the phase modulated image I_D and the added image I_R correspond to “image containing phase modulation component a” obtained by giving the weight corresponding at least each “first region” to each “first region” of the image obtained by changing the phase of the original image I_C (step S17).

The superposition unit 18 receives the original image I c and the phase modulated image I_D as input, and superimposes the original image I_C and the phase modulated image I_D (phase modulation component b) and obtains and outputs an added image I_L (image B). The phase modulation component b is an inverse phase image of the phase modulation component a. When the value of the disparity map Dmap is positive (crossed disparity) in a certain region, the phase of the phase modulated image I_D in the corresponding region is shifted from the phase of the original image I_C by 0.5π [rad] in the positive direction. Thus, the phase of the phase modulation component b in this region is obtained by shifting the phase of the original image I_C by 0.5π [rad] in the positive direction. On the other hand, when the value of the disparity map Dmap is negative in a certain region, the phase of the phase modulated image I_D in the corresponding region is shifted from the phase of the original image I_C by 0.5π [rad] in the negative direction. Thus, the phase of the phase modulation component b in this region is obtained by shifting the phase of the original image I_C by 0.5π [rad] in the negative direction. The pixel value I_L (x, y) of each coordinate (x, y) of the added image I_L is obtained by adding the pixel value I_C (x, y) of each coordinate (x, y) of the original image I_C and the pixel value I_D (x, y) of each coordinate (x, y) of the phase modulated image I_D (I_L (x, y)=I_C (x, y)+I_D (x, y)). Note that the phase modulated image I_D and the added image I_L correspond to “image containing phase modulation component b” obtained by giving the weight corresponding at least each “second region” to each “second region” of the image obtained by changing the phase of the original image I_C (step S18).

Hereinafter, modification examples of the image generation unit 10 will be described.

Modification Example 1

Theoretically, the maximum value of the disparity between the added images I_R and I_L is a half wavelength of each wavelength. However, in order to obtain a shift of a half wavelength, the above-described weight A_{λ, μ}(x, y) needs to be infinite, which is not practical. Thus, the absolute values of the phase shift amounts of the added images I_R, I_L with respect to the original image I_C may be limited to π/4 [rad] or less. Under such restrictions, the absolute value of the weight A_{λ, μ}(x, y) becomes 1 or less, and the absolute value of the disparity between the added images I_R, I_L is less than π/(2ωλ |cos θ_μ|). In this case, the weight calculation unit 13 obtains and outputs a weight image A_{λ, μ} which is a two-dimensional array having each weight A_{λ, μ}(x, y) as an element, according to the following Equation (6) instead of Equation (3).

$\begin{array}{cc}\left[\mathrm{Equation}7\right]& \\ A_{\lambda ,\mu }\left(\varkappa ,y\right)=\tan \frac{{\hat{G}}_{D_{\lambda }}\left(\varkappa ,y\right){\omega }_{\lambda }\cos {\theta }_{\mu }}{2}& \left(6\right)\\ \mathrm{where}& \\ \left[\mathrm{Equation}8\right]& \\ {\hat{G}}_{D_{\lambda }}\left(\varkappa ,y\right)=\{\begin{array}{cc}\frac{\pi }{2{\omega }_{\lambda }\cos {\theta }_{\mu }}& \mathrm{if}G_{D_{\lambda }}\left(\varkappa ,y\right)>\frac{\pi }{2{\omega }_{\lambda }\cos {\theta }_{\mu }}\\ -\frac{\pi }{2{\omega }_{\lambda }\cos {\theta }_{\mu }}& \mathrm{if}G_{D_{\lambda }}\left(\varkappa ,y\right)<-\frac{\pi }{2{\omega }_{\lambda }\cos {\theta }_{\mu }}\\ G_{D_{\lambda }}\left(\varkappa ,y\right)& \mathrm{otherwise}\end{array}& \end{array}$

When this modification example is applied, comparing the weights of the spatial frequency bands included in the same orientation band, the weight corresponding to the spatial frequency band λ₁ (first spatial frequency band) having a certain spatial frequency ωλ1 as a peak is less than or equal to the weight corresponding to the spatial frequency band λ₂ (second spatial frequency band) having the peak at the spatial frequency ωλ2 higher than the spatial frequency band λ₁ (that is, ωλ2>ωλ1). Further, when the weights of the orientation bands included in the same spatial frequency band are compared, the weight corresponding to the orientation band μ₁ (first orientation band) having a certain orientation θ_μ1 as the peak is less than or equal to the weight corresponding to the orientation band μ₂ (second orientation band) having as the peak, orientation θ_μ2 which is closer to the vertical orientation than the orientation band μ₁ (0 or π) (that is, |θ_μ2−π/2|>|θ_μ1−π/2|). The rest is as described for the image generation unit 10.

Modification Example 2

The image generation unit 10 superimposes the polarity-inversed image of the phase modulated image I_D and the phase modulated image I_D on the original image I_C to obtain added images I_R and I_L having disparity. The amplitudes of the added images I_R and I_L obtained in this manner are larger than the amplitude of the original image I_C, and the added images I_R and I_L may exceed the predetermined lower limit b_L and/or upper limit b_U (b_L<b_U), that is, a predetermined dynamic range. As a simple countermeasure against this, a method of linearly compressing the total intensity of the added images I_R and I_L such that the added images I_R and I_L fall within the range between the lower limit b_L and the upper limit b_U is conceivable. However, with this simple measure, the contrast of the added images I_R and I_L is smaller than that of the original image I_C, and the impression of the image changes. In addition, a countermeasure for deleting a portion exceeding the lower limit value b_L and/or the upper limit value b_U of the added images I_R and I_L may be considered. In this case, there is a possibility that a doubly blurred image may be perceived when the added images I_R and I_L are not canceled out and the added images I_R and I_L are viewed at once by both eyes. Thus, in Modification Example 2, the phase modulated image I_D corresponding to a portion exceeding the lower limit b_L or the upper limit b_U of the added images I_R and I_L is clipped.

More specific description will be made. In Modification Example 2, the image generation unit 10 further includes a dynamic range adjustment unit 161 (see FIG. 4). The dynamic range adjustment unit 161 receives the phase modulated image I_D and the added images I_R and I_L obtained by the superposition units 17, 18 as inputs, updates the phase modulated image I_D into a phase modulated image I_D{circumflex over ( )}as follows, and outputs the image.

[Equation 9]

Î _D(x,y)=I_D(x,y)−M_U(x,y)−M_L(x,y)  (7)

Here, the phase modulated image I_D{circumflex over ( )} is a two-dimensional array including elements I_D{circumflex over ( )}(x, y), and M_U (x, y)=max (max (I_R (x, y)−b_U, 0), max (I_L (x, y)−b_U, 0)), M_L (x, y)=min (min (I_R (x, y)−b_L, 0), min (I_L (x, y)−b_L, 0)). max (α1, α2) represents the larger value of α1 and α2, and min (α1, α2) represents the smaller value of α1 and α2. Note that, “{circumflex over ( )}” in “I_D{circumflex over ( )}” should be given directly above “I” (see Equation (7)). However, “{circumflex over ( )}” may be given at the upper right of “I_D” due to restrictions on the description in the specification.

In this case, the superposition unit 17 further receives the original image I_C and the phase modulated image I_D{circumflex over ( )}, and obtains and outputs the added image I_R (image A) by superimposing a polarity-inversed image (phase modulation component a) of the phase modulated image I_D{circumflex over ( )} on the original image I_C. The pixel value I_R (x, y) of each coordinate (x, y) of the added image I_R is obtained by subtracting the pixel value I_D{circumflex over ( )}(x, y) of each coordinate (x, y) of the phase modulated image I_D{circumflex over ( )} from the pixel value I_C (x, y) of each coordinate (x, y) of the original image I_C (I_R (x, y)=I_C (x, y)−I_D{circumflex over ( )}(x, y)). In this case, the added image I_R obtained in step S17 described above is not output as the image A.

The superposition unit 18 further receives the original image I_C and the phase modulated image I_D{circumflex over ( )} as inputs, and obtains and outputs an added image I_L (image B) by superimposing the original image I_C and the phase modulated image I_D{circumflex over ( )} (phase modulation component b). The pixel value I_L (x, y) of each coordinate (x, y) of the added image I_L is obtained by adding the pixel value I_C (x, y) of each coordinate (x, y) of the original image I_C and the pixel value I_D (x, y) of each coordinate (x, y) of the phase modulated image I_D{circumflex over ( )}(I_L (x, y)=I_C (x, y)+I_D{circumflex over ( )}(x, y)). In this case, the added image I_L obtained in step S18 described above is not output as the image B.

Modification Example 3

When the original image I_C is a color image, the processes of steps S121, S112, S123, S14, S15, S122, S16, S124, S17, and S18 are executed on each color channel (R, G, and B channels), and the added image I_R (image A) and the added image I_L (image B) of all color channels may be output. Modification Examples 1 and 2 may be performed even when the original image I_C is a color image, and the process of Modification Example 2 in this case is performed for each color channel.

Modification Example 4

The image generation unit 10 sets an image obtained by shifting the phase of the original image I_C in the positive direction by 0.5π [rad] as a phase-shifted image S_{C′λ, μ} (Equation (2)). However, an image obtained by shifting the phase of the original image I_C in the positive direction by an amount close to 0.5π may be used as the phase shift image S_{C′λ, μ}. In this case, the phase of the region of the polarity-inversed image (phase modulation component a) of the phase modulated image I_D corresponding to the region where the value of the disparity map Dmap is positive (crossed disparity) is obtained by shifting the phase of the original image I_C by an amount close to 0.5π [rad] in the negative direction. On the other hand, the phase of the region of the polarity-inversed image (phase modulation component a) of the phase modulated image I_D corresponding to the region where the value of the disparity map Dmap is negative is obtained by shifting the phase of the original image I_C by an amount close to 0.5π [rad] in the positive direction. The phase of the region of the phase modulated image I_D (phase modulation component b) corresponding to the region where the value of the disparity map Dmap is positive (crossed disparity) is obtained by shifting the phase of the original image I_C by an amount close to 0.5π [rad] in the positive direction. On the other hand, the phase of the region of the phase modulated image I_D (phase modulation component b) corresponding to the region where the value of the disparity map Dmap is negative is obtained by shifting the phase of the original image I_C by an amount close to 0.5π [rad] in the negative direction. Further, instead of using the polarity-inversed image of the phase modulated image I_D as the phase modulation component a and the phase modulated image I_D as the phase modulation component b, the polarity-inversed image of the phase modulated image I_D may be the phase modulation component a, and an image near the opposite phase of the phase modulation component a may be the phase modulation component b.

Others

As described in (image generation unit 10), a phase modulated image having a phase difference of π/2 is generated by using Equation (2). By using a predetermined equation instead of Equation (2), a phase modulated image having any phase difference can be generated.

The image generation unit 10 has been described in detail including modification examples in the above. Hereinafter, a case where K=2, φ1=π/4, φ2=π/2 will be described as an example (see FIG. 5). The image generation unit 10 generates the pseudo viewpoint images I_π/4, I_−π/4 and the pseudo viewpoint images I_π/2, I_−π/2, and outputs the pseudo viewpoint images I_π/2, I_−π/2, I_π/4, and I_−π/4 and the viewpoint image I. When these images are projected onto a screen using five projectors (projection apparatuses) in the sequence of I_π/2, I_π/4, I, I_−π/4, I_−π/2, and presented to an viewer, as the viewer moves, the viewer observes, for example, in the sequence of I_π/2, I_π/4, I, I_−π/4, I_−π/2. At this time, the image quality increases substantially linearly up to the images I_π/2, I_π/4, and I. Then, with the image quality of the image I as a peak, the image quality decreases substantially linearly up to I, I_−π/4, I_−π/2. For example, at the viewpoint position 1, the intermediate viewpoint position between the viewpoint position 1 and the viewpoint position 2, and the viewpoint position 2, the image quality gradually increases, so that the image quality does not fluctuate from good→poor→good as in the linear blending in the art. This enables to suppress unpleasant feelings that a viewer experiences due to fluctuation in image quality.

According to the disclosure of the present embodiment, it is possible to generate any number of pseudo viewpoint images at a viewpoint position different from the viewpoint position of the viewpoint image I, from one viewpoint image I and the disparity map Dmap. Further, by arranging the pseudo viewpoint image and the viewpoint image I in a sequence of the phase difference and projecting with the projector, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Second Embodiment

Hereinafter, an image generation apparatus 2 will be described with reference to FIG. 6 and FIG. 7. FIG. 6 is a block diagram illustrating a configuration of the image generation apparatus 2. FIG. 7 is a flowchart illustrating the operation of the image generation apparatus 2. As illustrated in FIG. 6, the image generation apparatus 2 includes an image generation unit 20 and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 2.

The image generation apparatus 2 receives, as inputs, the viewpoint image I and an adjacent viewpoint image I′, generates 2K (K is an integer of 1 or more) pseudo viewpoint images I_φk, I_−φk from the viewpoint image I and an adjacent viewpoint image I′ (1≤k≤K), and outputs pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents the phase difference between the viewpoint image I and the disparity inducing edge D_φk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. The disparity inducing edge D_φk is an image obtained by modulating a phase such that a phase difference from the viewpoint image I becomes φk. Further, the pseudo viewpoint image I_φk, I_−φk are images both at the viewpoint position different from the viewpoint position of the viewpoint image I (see FIG. 5). In the following, it is presented that φ0=0, I_φ0=I, and the viewpoint image I may be referred to as a pseudo viewpoint image I_φ0.

Note that, assuming a plurality of viewpoint positions horizontally arranged at equal intervals toward an object to be captured, an image captured by the camera from each viewpoint position is a viewpoint image, and an image captured from an adjacent viewpoint position is referred to as “the adjacent viewpoint image”. That is, the image generation apparatus 2 receives the viewpoint image I and the viewpoint image I′ captured from the viewpoint position adjacent to the viewpoint position where the viewpoint image I is captured.

The operation of the image generation apparatus 2 will be described in accordance with FIG. 7.

In step S20, the image generation unit 20 receives, as an input, the viewpoint image I and the adjacent viewpoint image I′, generates a pseudo viewpoint image I_φk, I_−φk, from the viewpoint image I and the viewpoint image I′, for k=1, . . . , K, and outputs the pseudo viewpoint image I_φk, I_−φk (1≤k≤K) and the viewpoint image I. A procedure for generating the pseudo viewpoint images I_φk, I_−φk will be described. First, the image generation unit 20 generates a disparity inducing edge D_φk, which is a phase modulated image having a phase difference φk from the viewpoint image I, from the viewpoint image I and the viewpoint image I′. Next, the image generation unit 20 generates the pseudo viewpoint image I_φk (I_φk=I+D_φk) by adding the disparity inducing edge D_φk to the viewpoint image I, and generates the pseudo viewpoint image I_−φk (I_−φk=I−D_φk) by adding the polarity-inversed image of the disparity inducing edge D_φk to the viewpoint image I. The image generation unit 20 sequentially generates the pseudo viewpoint images I_φk, I_−φk (1≤k≤K).

Note that the image generation apparatus 2 may further include a projection unit (not illustrated) that projects the pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I output at S20. The projection unit arranges the pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I in an sequence of phase differences from I_φk, I_φ(K−1), . . . , I_φ1, I, I_−φ1, . . . , I_−φK, that is, in a sequence of the viewpoint position, and projects the arranged images onto the screen.

Further, for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, the pseudo viewpoint image I_φk and the pseudo viewpoint image I_φ(k+1) are viewpoint images at viewpoint positions adjacent to each other, and the synthesized luminance of at least two images of the pseudo viewpoint image I_φk and the pseudo viewpoint image I_φ(k+1) may be used to present the images at viewpoints located in an intermediate position between the viewpoint positions adjacent to each other.

Hereinafter, the configuration and operation of the image generation unit 20 will be described in detail with reference to FIG. 8. Here, a case in which the phase difference is π/2 will be described. In addition, in the description, a viewpoint image is referred to as an original image, a disparity inducing edge is referred to as a phase modulated image, and a pseudo viewpoint image is referred to as an added image.

Image Generation Unit 20

As illustrated in FIG. 8, the image generation unit 20 includes spatial frequency domain conversion units 122, 221, 222, spatial domain conversion units 124, 224, 225, decomposition units 211, 212, a phase operation unit 23, and a phase difference calculation unit 24, a weight calculation unit 25, a weighting unit 15, a reconstruction unit 16, and superposition units 17, 18.

The processing by the image generation unit 20 will be described.

An original image I_L′=I_C (first original image) and an original image I_R′ (second original image) are input to the image generation unit 20. The original image I_L′ and the original image I_R′ are, for example, three-dimensional image contents for giving images an apparent depth by a binocular image presentation technology. There is a phase difference recognized as binocular disparity between the original image I_L′ and the original image I_R′. In other words, there is a phase difference recognized as binocular disparity between the original image I_L′ and the original image I_R′. When the original image I_R′ is viewed by one eye (for example, the right eye) and the original image I_L′ is viewed by the other eye (for example, the left eye), a “stereoscopic image” is perceived. On the other hand, when both the original image I_R′ and the original image I_L′ are viewed with both eyes, a doubly blurred image is perceived. The original image I_R, is a two-dimensional array having I_R′(x, y) as an element, and the original image ILS is a two-dimensional array having I_L′(x, y) as elements (pixels). The original image I_R′ and the original image I_L′ may be a still image, an image of individual frame of a moving image, an image read from an external recording device, or an image read from the recording unit 90.

The original image I_L′ is input to the spatial frequency domain conversion unit 221, and the original image I_R′ is input to the spatial frequency domain conversion unit 222. The spatial frequency domain conversion unit 221 converts the original image I_L′ into the original image I_L′{tilde over ( )} in the spatial frequency domain, and outputs the original image I_L′{tilde over ( )}. The spatial frequency domain conversion unit 222 converts the original image I_R′ into the original image I_R′{tilde over ( )} in the spatial frequency domain, and outputs the original image I_R′{tilde over ( )}. For the conversion in the spatial frequency domain conversion units 221 and 222, for example, a discrete Fourier transform can be used. It should be noted that the superscript “{tilde over ( )}” of “I_L′{tilde over ( )}” “I_R′{tilde over ( )}” should be given directly above “I”. However, “{tilde over ( )}” may be given at the upper right of “I_L” and “I_R” due to restrictions on the description in the specification (steps S221, S222).

The original image I_R′{tilde over ( )} in the spatial frequency domain is input to the decomposition unit 211, and the original image I_L′{tilde over ( )} in the spatial frequency domain is input to the decomposition unit 212. The decomposition units 211 and 212 apply the complex steerable filter sequence Ψ to the original images I_R′{tilde over ( )} and I_L′{tilde over ( )} as described below, and obtain and output each complex steerable pyramid S_{L′λ, μ}{tilde over ( )} and S_{R′λ, μ}{tilde over ( )} corresponding to each spatial frequency band λ and each orientation band μ (steps S211 and S212).

$\begin{array}{cc}\left[\mathrm{Equation}10\right]& \\ {\tilde{S}}_{L_{\lambda ,\mu }^{\prime }}={\tilde{I}}_{L^{\prime }}{\Psi }_{\lambda ,\mu }{\tilde{S}}_{R_{\lambda ,\mu }^{\prime }}={\tilde{I}}_{R^{\prime }}{\Psi }_{\lambda ,\mu }& \end{array}$

The complex steerable pyramids S_{L′λ, μ}{tilde over ( )} and S_{R′λ, μ}{tilde over ( )} are input to the spatial domain conversion units 224 and 225, respectively. The spatial domain conversion units 224 and 225 respectively convert the complex steerable pyramids S_{L′λ, μ}{tilde over ( )} and S_{R′λ, μ}{tilde over ( )} into the complex steerable pyramid S_{L′λ, μ}{tilde over ( )} and S_{R′λ, μ}{tilde over ( )} in the spatial domain, and output the complex steerable pyramids S_{L′λ, μ} and S_{R′λ, μ}. For the conversion in the spatial domain conversion units 224 and 225, for example, a discrete inverse Fourier transform can be used. Each complex steerable pyramid S_{L′λ, μ} is a two-dimensional array having S_{L′λ, μ}(x, y) as elements (pixels), and each complex steerable pyramid S_{R′λ, μ} is a two-dimensional array having S_{R′λ, μ}(x, y) as elements (steps S224, S225).

The complex steerable pyramids S_{L′λ, μ} and S_{R′λ, μ} are input to the phase difference calculation unit 24. The phase difference calculation unit 24 obtains and outputs the phase difference δ_{λ, μ}(x, y) between S_{L′λ, μ}(x, y) and S_{R′λ, μ}(x, y), by using the method described in Non-Patent Literature 1 (DIDYK, P., SITTHI-AMORN, P., FREEMAN, W., DURAND, F., AND MATUSIK, W. 2013, “Joint View Expansion and Filtering for Automultiscopic 3d Displays, “ACM Trans. Graph. 32, 6737 (November), 221:1-221: 8). A set having the phase difference δ_{λ, μ}(x, y) as an element is referred to as a phase difference δ_{λ, μ}, As described in reference Non-Patent Literature 1, for a spatial frequency band λ having a phase difference exceeding π/2 [rad], a value twice the phase difference δ_{λ−1, μ}(x, y) at the spatial frequency band λ−1 one lower than the spatial frequency band λ is defined as a phase difference δ_{λ, μ}(x, y).

δ_λ,μ(x,y)2δ_λ−1,μ(x,y)  [Equation 11]

Here, the peak spatial frequency ωλ of the spatial frequency band λ is twice or nearly twice the peak spatial frequency ωλ−1 of the spatial frequency band (λ−1) (step S24).

The phase difference δ_{λ, μ} is input to the weight calculation unit 25. The weight calculation unit 25 obtains and outputs a weight A_{λ, μ}(x, y) as follows (step S25).

A _λ,μ(x,y)=tan {δ_λ,μ(x,y)}  [Equation 12]

The complex steerable pyramid S_{L′λ, μ} obtained by the spatial domain conversion unit 224 is input to the phase operation unit 23. The phase operation unit 23 extracts an imaginary part of the complex steerable pyramid S_{L′λ, μ}, sets S_{C′λ, μ}=S_{L′λ, μ}, and in accordance with Equation (2), obtains and outputs each phase shift image S_{C′λ, μ} corresponding to each spatial frequency band λ and each orientation band μ (step S224).

Thereafter, the processes of steps S15, S122, S16, and S124 described in (Image generation unit 10) are performed, and the spatial domain conversion unit 124 outputs the phase modulated image I_D. Further, the process of step S17 is performed, and the superposition unit 17 obtains and outputs the added image I_R (image A). Further, the superposition unit 18 performs the process of step S18 to obtain and output the added image I_L (image B).

Hereinafter, modification examples of the image generation unit 20 will be described.

Modification Example 1

As described in Modification Example 1 of the image generation unit 10, the absolute value of the phase shift amount of the added image I_R with respect to the original image I_L, =I_C may be limited to π/4 [rad] or less. In this case, the image generation unit 20 (see FIG. 8) further includes a phase difference limiting unit 241. The phase difference limiting unit 241 receives the phase difference δ_{λ, μ}(x, y) obtained by the phase difference calculation unit 24 as an input, and updates the phase difference δ_{λ, μ}(x, y) into δ{circumflex over ( )}_{λ, μ}(x, y) as follows.

$\begin{array}{cc}\left[\mathrm{Equation}13\right]& \\ {\tilde{\delta }}_{\lambda ,\mu }\left(\varkappa ,y\right)=\{\begin{array}{cc}\pi /4& \mathrm{if}{\mathrm{\tau \delta }}_{\lambda ,\mu }\left(\varkappa ,y\right)/2>\pi /4\\ -\pi /4& \mathrm{if}{\mathrm{\tau \delta }}_{\lambda ,\mu }\left(\varkappa ,y\right)/2<-\pi /4\\ {\mathrm{\tau \delta }}_{\lambda ,\mu }\left(\varkappa ,y\right)/2& \mathrm{otherwise}\end{array}& \end{array}$

Here, τ is a positive constant for scaling. By reducing the value of τ, the range where the phase difference is clipped to π/4 [rad] can be narrowed.

In this case, in step S25, δ{circumflex over ( )}_{λ, μ}(x, y) is input to the weight calculation unit 25 instead of the phase difference δ_{λ, μ}(x, y). In this case, the weight calculation unit 25 obtains and outputs a weight A_{λ, μ}(x, y) as follows.

A _λ,μ(x,y)=tan {{circumflex over (δ)}_λ,μ(x,y)}  [Equation 14]

Subsequent processes are as described above.

Modification Example 2

As in Modification Example 2 of the image generation unit 10, the image generation unit 20 may further include the dynamic range adjustment unit 161 (see FIG. 8). In this case, as described in Modification Example 2 of the image generation unit 10, the dynamic range adjustment unit 161 updates the phase modulated image I_D to the phase modulated image I_D{circumflex over ( )} and outputs the phase modulated image, the superposition unit 17 superimposes a polarity-inversed image (phase modulation component a) of the phase modulated image I_D{circumflex over ( )} to the original image I_L′=I_C to obtain and output an added image I_R (image A), and the superposition unit 18 further receives the original image I_L′=I_C and the phase modulated image I_D{circumflex over ( )} and superimposes the original image I_L′=I_C and the phase modulated image I_D{circumflex over ( )}(phase modulation component b) to obtain and output an added image I_L (image B).

Modification Example 3

When the original image I_L′=I_C and the original image I_R′ are color images, processes of steps S221, S222, S211, S212, S224, S225, S24, S25, S23, S15, S122, S16, S124, S17, S18 are performed for each color channel (R, G, and B channels), and the added image I_R (image A) and the added image I_L (image B) of all color channels may be output. The process of step S24 may be performed on only one of the channels. Modification Examples 1 and 2 may be performed even when the original image I_L′=I_C is a color image, and the process of Modification Example 2 in this case is performed for each color channel.

Modification Example 4

The image generation unit 20 sets an image obtained by shifting the phase of the original image I_L′=I_C in the positive direction by 0.5π [rad] as a phase-shifted image S_{C′λ, μ} (Equation (2)).

However, an image obtained by shifting the phase of the original image I_L′=I_C in the positive direction by an amount close to 0.5π may be used as the phase shift image S In this case, the phase of the polarity-inversed image (phase modulation component a) of the phase modulated image I_D corresponding to the region where the value of the disparity map is positive (crossed disparity) is obtained by shifting the phase of the original image I_L′=I_C by an amount close to 0.5π [rad] in the negative direction, and the phase of the phase modulated image I_D (phase modulation component b) corresponding to the region where the value of the disparity map is positive (crossed disparity) is obtained by shifting the phase of the original image I_L′=I_C by an amount close to 0.5π [rad] in the negative direction. Further, instead of using the polarity-inversed image of the phase modulated image I_D as the phase modulation component a and the phase modulated image I_D as the phase modulation component b, the polarity-inversed image of the phase modulated image I_D may be the phase modulation component a, and an image near the opposite phase of the phase modulation component a may be the phase modulation component b.

Others

As described in (image generation unit 20), a phase modulated image having a phase difference of π/2 is generated by using Equation (2). By using a predetermined equation instead of Equation (2), a phase modulated image having any phase difference can be generated.

The image generation unit 20 has been described in detail including modification examples in the above. Note that, in a case where K=2, φ1=π/4, and φ2=π/2, the relationship between the viewpoint movement and the fluctuation in image quality is the same as in the image generation unit 10 (see FIG. 5).

According to the disclosure of the present embodiment, it is possible to generate any number of pseudo viewpoint images at a viewpoint position different from the viewpoint position of the viewpoint image I, from the viewpoint image I and the adjacent viewpoint image I′. Further, by arranging the pseudo viewpoint image and the viewpoint image I in a sequence of the phase difference and projecting with the projector, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Third Embodiment

Hereinafter, an image generation apparatus 3 will be described with reference to FIG. 9 and FIG. 10. FIG. 9 is a block diagram illustrating a configuration of the image generation apparatus 3. FIG. 10 is a flowchart illustrating the operation of the image generation apparatus 3. As illustrated in FIG. 9, the image generation apparatus 3 includes an image generation unit 10, an output image generation unit 30, and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 3.

The image generation apparatus 3 receives, as an input, the viewpoint image I and the disparity map Dmap corresponding to the viewpoint image I, and generates and outputs 2K (K is an integer of 2 or greater) output image Out_m (1≤m≤2K−1) from the viewpoint image I and the disparity map Dmap. Here, the output image Out_m is an image projected on a screen using a projector (projection apparatus).

The operation of the image generation apparatus 3 will be described in accordance with FIG. 10.

In step S10, the image generation unit 10 receives, as an input, the viewpoint image I and a disparity map Dmap of the viewpoint image I, generates pseudo viewpoint image I_φk, I_−φk, from the viewpoint image I and the disparity map Dmap for k=1, . . . , K, and outputs the pseudo viewpoint images I_φk, I_−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents the phase difference between the viewpoint image I and the disparity inducing edge D_φk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. In the following, it is presented that φ0=0 and I_φ0=I, and the viewpoint image I may be referred to as a pseudo viewpoint image I_φ0.

In step S30, the output image generation unit 30 receives, as an input, the pseudo viewpoint image I_φk, I_φk (1≤k≤K) and the viewpoint image I, which are output at S10, and generates and outputs an output image Out_m (1≤m≤2K−1). The pseudo viewpoint images I_φk and I_−φk (1≤k≤K) and the viewpoint image I are arranged in the sequence of I_φK, I_φ(K−1), . . . , I_φ1, I_φ0(=I), I_−φ1, . . . , and I_φK to make I⁽¹⁾, I⁽²⁾, . . . , I^(K), I^(K+1), I^(K+2), . . . , and I^(2K+1). That is, the output image generation unit 30 receives the pseudo viewpoint image I^(m) (1≤m≤2K+1) as an input. Hereinafter, an output image generation unit 30 will be described with reference to FIG. 11 and FIG. 12. As illustrated in FIG. 11, the output image generation unit 30 includes an image-for-synthesis generation unit 31 and an output image synthesis unit 32. The operation of the output image generation unit 30 will be described in accordance with FIG. 12.

In step S31, the image-for-synthesis generation unit 31 receives the pseudo viewpoint image I^(m) (1≤m≤2K+1) output in S10 as an input, generates images for synthesis I^(m)_L and I^(m)_R, for m=1, . . . , 2K, from the pseudo viewpoint image I^(m) and the pseudo viewpoint image I^(m+1) and outputs the image for synthesis I^(m)_L and I^(m)_R (1≤m≤2K). For example, the image-for-synthesis generation unit 31 may generate the images for synthesis I^(m)_L, I^(m)_R by the same procedure as the image generation unit 20. In this case, the image-for-synthesis generation unit 31 is a component having the same configuration as the image generation unit 20 (see FIG. 8). Further, after generating the disparity map Dmap^(m) of the pseudo viewpoint image I^(m) from the pseudo viewpoint image I^(m) and the pseudo viewpoint image I^(m+1) the image-for-synthesis generation unit 31 may generate images for synthesis I^(m)_L, I^(m)_R, from the pseudo viewpoint image I^(m) and the disparity map Dmap^(m), by the same procedure as that of the image generation unit 10. In this case, the image-for-synthesis generation unit 31 is a component including the disparity-map generation unit 311 and the image generation unit 10, as illustrated in FIG. 13. That is, the disparity map generation unit 311 generates the disparity map Dmap^(m) of the pseudo viewpoint image I^(m) from the pseudo viewpoint image I^(m) and the pseudo viewpoint image I^(m+1), and outputs the pseudo viewpoint image I^(m) and the disparity map Dmap^(m). In any case, in step S31, the image-for-synthesis generation unit 31 generates a disparity inducing edge D^(m) whose the phase difference from the pseudo viewpoint image I^(m) is a predetermined value (for example, π/2), from the pseudo viewpoint image I^(m) and the pseudo viewpoint image I^(m+1), for m=1, . . . , 2K, generates an image for synthesis I^(m)_L by adding the disparity inducing edge D^(m) to the pseudo viewpoint image I^(m) and generates an image for synthesis I^(m)_R by adding the polarity-inversed image of the disparity inducing edge D^(m) to the pseudo viewpoint image I^(m).

In step S32, the output image synthesis unit 32 receives the images for synthesis I^(m)_R and I^(m)_L (1≤m≤2K) generated in S31 as an input, synthesizes the output image Out_m, from the images for synthesis I^(m)_R and I^(m+1)_L for m=1, . . . , 2K−1, and outputs the output image Out_m (1≤m≤2K−1). For example, an output image may obtained by adding the RGB values of the respective pixels of the image for synthesis I^(m)_R and the image for synthesis I^(m+1)_L and dividing the sum by 2 (averaged value) as a pixel value.

The image generation apparatus 3 may further include a projection unit (not illustrated) that projects the output images Out_m (1≤m≤2K−1) output in S30. The projection unit arranges the output images Out_m (1≤m≤2K−1) in the sequence of Out₁, Out₂, . . . , and Out_2K−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.

A description will be made below with reference to FIG. 14. The output image Out₁ is an image obtained by synthesizing the images for synthesis I⁽¹⁾_R and I⁽²⁾_L. Further, the output image Out₂ is an image obtained by synthesizing the images for synthesis I⁽²⁾_R and I⁽³⁾_L. At the viewpoint position A, the output image Out₁ is perceived. On the other hand, at the viewpoint position B, the output image Out₁ and the output image Out₂ are synthesized and visually recognized (simultaneously viewed). That is, at the viewpoint position B, the phase modulation component is canceled and the image for synthesis I⁽²⁾ is mainly perceived. The image perceived at the viewpoint position B includes, as components that may be degraded, an image for synthesis I⁽¹⁾_R used for synthesizing the output image Out₁ at the viewpoint position A and an image for synthesis I⁽³⁾_L used for synthesizing the output image Out₃ at the viewpoint position C, which are perceived. However, the influence of these components that may be degraded is less than half of the linear blending in the related art, so that the degradation in image quality may be mitigated. As a result, a viewer hardly experiences unpleasant feelings associated with fluctuation in image quality due to the viewer's viewpoint moving from A to C.

As can be understood from the above description, when the output image Out_m and the output image Out_m+1 are synthesized and visually recognized (visually recognized simultaneously), an image in which the disparity inducing edge D^(m+1) as the phase modulation component is canceled out (pseudo viewpoint image I^(m+1)) is mainly perceived. In other words, the output image generation unit 30 generates an output image Out_m (1≤m≤2K−1) from the pseudo viewpoint image I^(m) (1≤m≤2K+1), and the output image Out_m and the output image Out_m+1 (1≤m≤2K−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).

According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Fourth Embodiment

Hereinafter, an image generation apparatus 4 will be described with reference to FIG. 15 and FIG. 16. FIG. 15 is a block diagram illustrating a configuration of the image generation apparatus 4. FIG. 16 is a flowchart illustrating the operation of the image generation apparatus 4. As illustrated in FIG. 15, the image generation apparatus 4 includes an image generation unit 20, an output image generation unit 30, and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 4.

The image generation apparatus 4 receives the viewpoint image I and the adjacent viewpoint image I′ (that is, the viewpoint image I and the viewpoint image I′ captured from the viewpoint position adjacent to the viewpoint position where the viewpoint image I is captured) as inputs, and generates and outputs 2K (K is an integer of 2 or more) output image Out_m (1≤m≤2K−1) from the viewpoint image I and the viewpoint image I′. Here, the output image Out_m is an image projected on a screen using a projector (projection apparatus).

The operation of the image generation apparatus 4 will be described in accordance with FIG. 16.

In step S20, the image generation unit 20 receives, as an input, the viewpoint image I and the adjacent viewpoint image I′, generates a pseudo viewpoint image I_φk and I_−φk from the viewpoint image I and the viewpoint image I′, k=1, K, and outputs the pseudo viewpoint image I_φk and I_−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents the phase difference between the viewpoint image I and the disparity inducing edge D_φk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. In the following, it is presented that φ0=0 and I_φ0=I, and the viewpoint image I may be referred to as a pseudo viewpoint image I_φ0.

In step S30, the output image generation unit 30 receives the pseudo viewpoint image I_φk, I_−φk (1≤k≤K) and the viewpoint image I, which are output at S20, and generates and outputs an output image Out_m (1≤m≤2K−1).

The image generation apparatus 4 may further include a projection unit (not illustrated) that projects the output images Out_m (1≤m≤2K−1) output in S30. The projection unit arranges the output images Out_m (1≤m≤2K−1) in the sequence of Out₁, Out₂, . . . , and Out_2K−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.

The output image Out_m (1≤m≤2) is an image obtained by synthesizing the images for synthesis I^(m)_R and I^(m+1)_L, so that as in the image generation apparatus 3, a viewer hardly experiences unpleasant feelings associated with the fluctuation in image quality caused by the viewpoint of the viewer moving from A to C (see FIG. 14).

In other words, a sequence in which the pseudo viewpoint images I_φk and I_−φk (1≤k≤K) and the viewpoint image I are arranged in the sequence of I_φK, I_φ(K−1), . . . , I_φ1, I_φ0 (=I), I_−φ1, . . . , and I_−φK is set as I⁽¹⁾, I⁽²⁾, . . . , I^(k), I^(K+1), I^(K+2), . . . , I^(2K+1), and the output image generation unit 30 generates output images Out_m (1≤m≤2K−1) from the pseudo viewpoint images I^(m) (1≤m≤2K+1), and the output image Out_m and the output image Out_m+1 (1≤m≤2K−2) include a phase modulation components that are canceled out when synthesized and visually recognized (visually recognized at the same time).

According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Fifth Embodiment

Hereinafter, an image generation apparatus 5 will be described with reference to FIG. 17 and FIG. 18. FIG. 17 is a block diagram illustrating a configuration of the image generation apparatus 5. FIG. 18 is a flowchart illustrating the operation of the image generation apparatus 5. As illustrated in FIG. 17, the image generation apparatus 5 includes an output image generation unit 50 and a recording unit 90. The output image generation unit 50 includes an image-for-synthesis generation unit 51 and an output image synthesis unit 52. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 5.

The image generation apparatus 5 receives, as inputs, a set of M viewpoint images I^(m) and the disparity map Dmap^(m) corresponding to the viewpoint images I^(m) (1≤m≤M, M is an integer of 3 or greater), and generates and outputs M−1 output images Out_m (1≤m≤M−1). Here, the viewpoint images I⁽¹⁾, I⁽²⁾, . . . , I^(M) are a sequence of viewpoint images arranged in the sequence of the viewpoint positions. The output image Out_m is an image projected on a screen using a projector (projection apparatus). The interval between the adjacent viewpoint image I^(m) and viewpoint image I^(m+1) is the same as the interval between the adjacent output image Out_m and the output image Out_m+1, that is, the placement interval between the projectors that projects the output image Out_m.

The operation of the image generation apparatus 5 will be described in accordance with FIG. 18.

In step S50, the output image generation unit 50 receives the viewpoint image I^(m) and the disparity map Dmap^(m) (1≤m≤M) as an input, and generates and outputs an output image Out_m (1≤m≤M−1).

In step S51, the image-for-synthesis generation unit 51 receives the viewpoint image I^(m) and the disparity map Dmap^(m) (1≤m≤M) as inputs, generates images for synthesis I^(m)_L, I^(m)_R, from the viewpoint image I^(m) and the disparity map Dmap^(m), for m=1, . . . , M, and outputs the images for synthesis I^(m)_L, I^(m)_R (1≤m≤M). For example, the image-for-synthesis generation unit 51 may generate the images for synthesis I^(m)_L, I^(m)_R by the same procedure as the image generation unit 10. That is, the added images I_L, I_R are set to the images for synthesis I^(m)_L, I^(m)_R. In this case, the image-for-synthesis generation unit 51 is a component having the same configuration as the image generation unit 10 (see FIG. 4), and repeats a generation process on a set of M viewpoint images I^(m) and disparity maps Dmap on). In other words, in step S51, the image-for-synthesis generation unit 51 generates a disparity inducing edge D^(m) whose the phase difference from the viewpoint image I^(m) is a predetermined value (for example, π/2), from the viewpoint image I^(m) and the disparity map Dmap^(m), for m=1, . . . , M, generates an image for synthesis I^(m)_L by adding the disparity inducing edge D^(m) to the viewpoint image I^(m) and generates an image for synthesis I^(m)_R by adding the polarity-inversed image of the disparity inducing edge D^(m)) to the viewpoint image I^(m).

In step S52, the output image synthesis unit 52 receives the images for synthesis I^(m)_R and I^(m)_L (1≤m≤M) generated in S51 as inputs, synthesizes the output image Out_m, from the images for synthesis I^(m)_R and I^(m+1)_L for m=1, . . . , M−1, and outputs the output image Out_m (1≤m≤M−1). The procedure for synthesizing the output image Out_m may be the same as that of the output image synthesis unit 32, and the generation process is repeated for a set of M−1 images for synthesis I^(m)_R and I^(m+1)_L.

In other words, the output image generation unit 50 generates an output image Out_m (1≤m≤M−1) from the viewpoint image I^(m) and the disparity map Dmap^(m) (1≤m≤M), and the output image Out_m and the output image Out_m+1 (1≤m≤M−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).

The image generation apparatus 5 may further include a projection unit (not illustrated) that projects the output images Out_m (1≤m≤M−1) output in S50. The projection unit arranges the output images Out_m (1≤m≤M−1) in the sequence of Out₁, Out₂, . . . , and Out_M−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.

According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Sixth Embodiment

Hereinafter, an image generation apparatus 6 will be described with reference to FIG. 19 and FIG. 20. FIG. 19 is a block diagram illustrating a configuration of the image generation apparatus 6. FIG. 20 is a flowchart illustrating the operation of the image generation apparatus 6. As illustrated in FIG. 19, the image generation apparatus 6 includes an output image generation unit 60 and a recording unit 90. The output image generation unit 60 includes an image-for-synthesis generation unit 61 and an output image synthesis unit 52. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 6.

The image generation apparatus 6 receives the viewpoint image I^(m) (1≤m≤M+1, where M is an integer of 3 or more), generates and outputs M−1 output images Out_m (1≤m≤M−1). Here, the viewpoint images I⁽¹⁾, I⁽²⁾, . . . , I^(M) are a sequence of viewpoint images arranged in the sequence of the viewpoint positions. The output image Out_m is an image projected on a screen using a projector (projection apparatus). The interval between the adjacent viewpoint image I^(m) and viewpoint image I^(m+1) is the same as the interval between the adjacent output image Out_m and the output image Out_m+1, that is, the placement interval between the projectors that projects the output image Out_m.

The operation of the image generation apparatus 6 will be described in accordance with FIG. 20.

In step S60, the output image generation unit 60 receives the viewpoint image I^(m) (1≤m≤M+1) as an input, and generates and outputs an output image Out_m (1≤m≤M−1).

In step S61, the image-for-synthesis generation unit 61 receives the viewpoint image I^(m) (1≤m≤M+1) as an input, generates images for synthesis I^(m)_L, I^(m)_R, from the viewpoint image I^(m) and the adjacent viewpoint images I^(m+1), for m=1, . . . , M, and outputs the images for synthesis I^(m)_L, I^(m)_R (1≤m≤M). For example, the image-for-synthesis generation unit 61 may generate the images for synthesis I^(m)_L, I^(m)_R by the same procedure as the image generation unit 20. That is, the added images I_L, I_R are set to the images for synthesis I^(m)_L, I^(m)_R. In this case, the image-for-synthesis generation unit 61 is a component having the same configuration as the image generation unit 20 (see FIG. 8), and repeats a generation process on a set of M viewpoint images I^(m) and I^(m+1). As another method, for example, the image-for-synthesis generation unit 61 may generate the images for synthesis I^(m)_L, I^(m)_R by the same procedure as the image-for-synthesis generation unit 31. In this case, the image-for-synthesis generation unit 61 is a component having the same configuration as the image-for-synthesis generation unit 31 (see FIG. 13), and repeats a generation process on a set of M viewpoint images I^(m) and I^(m+1). In any case, in step S61, the image-for-synthesis generation unit 61 generates a disparity inducing edge D^(m) whose the phase difference from the viewpoint image I^(m) is a predetermined value (for example, π/2), from the viewpoint image I^(m) and the neighboring viewpoint image I^(m+1), for m=1, . . . , M, generates an image for synthesis I^(m)_L by adding the disparity inducing edge D^(m) to the viewpoint image I^(m), and generates an image for synthesis I^(m)_R by adding the polarity-inversed image of the disparity inducing edge D^(m) to the viewpoint image I^(m).

In step S52, the output image synthesis unit 52 receives the images for synthesis I^(m)_R and I^(m)_L (1≤m≤M) generated in S61 as an input, synthesizes the output image Out_m, from the images for synthesis I^(m)_R and I^(m+1)_L for m=1, . . . , M−1, and outputs the output image Out_m (1≤m≤M−1).

In other words, the output image generation unit 60 generates an output image Out_m (1≤m≤M−1) from the viewpoint image I^(m) (1≤m≤M+1), and the output image Out_m and the output image Out_m+1 (1≤m≤M−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).

The image generation apparatus 6 may further include a projection unit (not illustrated) that projects the output images Out_m (1≤m≤M−1) output in S60. The projection unit arranges the output images Out_m (1≤m≤M−1) in the sequence of Out₁, Out₂, . . . , and Out_M−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.

According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Seventh Embodiment

Hereinafter, an image generation apparatus 7 will be described with reference to FIG. 21 and FIG. 22. FIG. 21 is a block diagram illustrating a configuration of the image generation apparatus 7. FIG. 22 is a flowchart illustrating the operation of the image generation apparatus 7. As illustrated in FIG. 21, the image generation apparatus 7 includes an output image generation unit 70 and a recording unit 90. The output image generation unit 70 includes an image-for-synthesis generation unit 71 and an output image synthesis unit 52. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 7.

The image generation apparatus 7 receives the viewpoint image I^(j) (1≤j≤2J+1, where J is an integer of 3 or more), generates J−1 output images Out_j (1≤j≤J−1), and outputs the generated images. Here, it is a sequence of viewpoint images in which the viewpoint images I⁽¹⁾, I⁽²⁾, . . . , I^(2J+1) are arranged in the sequence of the viewpoint positions. The output image Out_j is an image projected on a screen using a projector (projection apparatus). The interval between two adjacent viewpoint images is the same as the interval between two adjacent output images, that is, ½ of the placement interval between projectors that project the output image Out_j. FIG. 23 is a diagram illustrating a positional relationship between the viewpoint images I⁽¹⁾ to I⁽¹¹⁾ and the output images Out₁ to Out₄ when J=5. That is, the projectors are placed only at the viewpoint positions 3, 5, 7, and 9. Note that D is a placement interval of the projector.

The operation of the image generation apparatus 7 will be described in accordance with FIG. 22.

In step S70, the output image generation unit 70 receives the viewpoint image I^(j) (1≤j≤2J+1) as an input, and generates and outputs an output image Out_j (1≤j≤J−1).

In step S71, the image-for-synthesis generation unit 71 receives the viewpoint image I^(j) (1≤j≤2J+1) as an input, generates images for synthesis I^(2j)_L and I^(2j)_R, from the viewpoint images viewpoint image I^(2j−1), I^(2j), I^(2j+1), for j=1, . . . , J, and outputs the images for synthesis I^(2j)_L and I^(2j)_R (1≤j≤J). As illustrated in FIG. 24, the configuration of the image-for-synthesis generation unit 71 is the same as the configuration of the image generation unit 20. The difference is that the image generation unit 20 has two input images, whereas the image-for-synthesis generation unit 71 has three input images. Accordingly, the procedure for generating the images for synthesis I^(2j)_L and I^(2j)_R is the same as the procedure for generating the added image by the image generation unit 20. In other words, in step S71, the image-for-synthesis generation unit 71 generates a disparity inducing edge D^(2j) whose the phase difference from the viewpoint image I^(2j) is a predetermined value, from adjacent viewpoint images I^(2j−1), I^(2j), and I^(2j+1), for j=1, . . . , J, generates an image for synthesis I^(2j)_L by adding the disparity inducing edge D^(2j) to the viewpoint image I^(2j), and generates an image for synthesis I^(2j)_R by adding the polarity-inversed image of the disparity inducing edge D^(m) to the viewpoint image I^(2j).

In step S52, the output image synthesis unit 52 receives the images for synthesis I^(2j)_R and I^(2j)_L (1≤j≤J) generated in S71 as an input, synthesizes the output image Out_j, from the images for synthesis I^(2j)_R and I^(2j+2)_L, for j=1, . . . , J−1, and outputs the output image Out_j (1≤j≤J−1).

In other words, the output image generation unit 70 generates an output image Out_j (1≤j≤J−1) from the viewpoint image I^(j) (1≤j≤2J+1), and the output image Out_j and the output image Out_j+1 (1≤j≤J−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).

The image generation apparatus 7 may further include a projection unit (not illustrated) that projects the output images Out_j (1≤j≤J−1) output in S70. The projection unit arranges the output images Out_j (1≤j≤J−1) in the sequence of Out₁, Out₂, . . . , and Out_J−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.

FIG. 25 illustrates a result of an experiment. The vertical axis represents the value of the image quality evaluation index structural similarity (SSIM). The horizontal axis represents the viewpoint position of the viewpoint image used in FIG. 23. In the linear blending in the related art, at the viewpoint positions 3, 5, 7, and 9, the viewpoint image is directly used as the output image, so that the SSIM at the viewpoint positions 3, 5, 7, and 9 increases, while it can be seen that the SSIM has significantly decreased in the viewpoint positions 4, 6, and 8. On the other hand, in the method of the present disclosure, the SSIM at the viewpoint positions 4, 6, and 8 where no projector is placed is higher than the viewpoint positions 3, 5, 7, and 9 where the projectors are placed. As a result, the fluctuation in image quality associated with the viewpoint movement is reduced, and the viewer is less likely to feel unpleasant feelings.

According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.

Others

Some of the above embodiments use the disparity map as in the first embodiment, for example. In the embodiments using the disparity map, a depth map may be used instead of the disparity map. For example, when the depth map is input, the processes described in each embodiment may be performed after converting the depth map into the disparity map.

Appendix

Examples of the apparatuses in the present disclosure include, as single hardware entities, an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, a communication device (for example, a communication cable) that can communicate outside the hardware entity, a Central Processing Unit (CPU) (a cache memory or a register may be provided), a RAM and a ROM as memories, an external storage device as a hard disk, and a bus connected to the input unit, the output unit, the communication unit, the CPU, the RAM, the ROM, and the external storage device such that data can be exchanged. If necessary, the hardware entity may be provided with a device (drive) that can read and write data from and to a recording medium such as a CD-ROM. A physical entity having such hardware resources includes a general-purpose computer.

The external storage device of the hardware entity include a program necessary for realizing the above-described functions, data necessary for processing of the program, and or the like (without being limited to the external storage device, for example, the program may be read and stored in a ROM that is a dedicated storage device). Data obtained by the processing of these programs is appropriately stored in a RAM, an external storage device, or the like.

In the hardware entities, each program stored in an external storage device (or ROM or the like) and data necessary for processing of each program are read into a memory as needed, and interpreted, executed and processed by the CPU as appropriate. As a result, the CPU realizes a predetermined function (each component represented as . . . unit, . . . means, or the like described above).

The present disclosure is not limited to the embodiments described above, and may be changed as appropriate without departing from the scope of the disclosure. Further, the processing described in the above embodiment may be performed not only in chronological order according to the order of description, but also in parallel or individually according to the processing capability of the apparatus that executes the processing or as necessary.

As described above, when the processing function of the hardware entity (the apparatus in the present disclosure) described in the above embodiment is implemented by a computer, the processing content of the function that the hardware entity should have is described by a program. By executing this program on a computer, the processing functions of the hardware entity are realized on the computer.

A program describing this processing content can be recorded on a computer-readable recording medium. The computer-readable recording medium may be any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Specifically, for example, a hard disk device, a flexible disk, a magnetic tape, or the like can be used as a magnetic recording device, a Digital Versatile Disc (DVD), a DVD-Random Access Memory (RAM), a Compact Disc Read Only Memory (CD-ROM), a Recordable (CD-R)/ReWritable (RW), or the like can be used as an optical disk, a Magneto-Optical disc (MO), or the like is used as a magneto-optical recording medium, and an Electronically Erasable and Programmable-Read Only Memory (EEP-ROM), or the like can be used as a semiconductor memory.

Further, the distribution of the program is performed, for example, by selling, transferring, lending, or the like, a portable recording medium such as a DVD or a CD-ROM on which the program is recorded. Further, the program may be stored in a storage device of a server computer, and the program may be distributed by transferring the program from the server computer to another computer via a network.

A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. Then, when executing the process, the computer reads the program stored in its own recording medium and executes the process according to the read program. Further, as another execution form of the program, the computer may directly read the program from the portable recording medium and execute processing according to the program, and further, may sequentially execute the processing according to the received program, each time the program is transferred Further, the above-described processing may be executed by a so-called an Application Service Provider (ASP) type service that realizes a processing function only by executing an instruction and acquiring a result without transferring a program from the server computer to the computer. It should be noted that the program in the present disclosure includes information which is provided for processing by an electronic computer and is similar to the program (data that is not a direct command to the computer but has characteristics that define the processing of the computer).

Further, in this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims

1.-15. (canceled)

16. A computer-implemented method for generating an image at a viewpoint, the method comprising: receiving a first viewpoint image and a plurality of disparity maps, wherein the disparity map corresponds to the first viewpoint image;generating, based on the first viewpoint image and the disparity map, a plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has a distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence;generating, based on the plurality of disparity inducing edges and the first viewpoint image, a first set of pseudo viewpoint images;generating, based on polarity-inverted images associated with the plurality of disparity inducing edges and the first viewpoint image, a second set of pseudo viewpoint images;generating, based at least on the first set of pseudo viewpoint images and the second set of pseudo viewpoint images, a set of output images, wherein the set of output images are in the predetermined sequence, and wherein the set of output images include a plurality of phase modulations for canceling out upon a synthesis and visual confirmation; andproviding the set of output images.

17. The computer-implemented method of claim 16, the method comprising: receiving a second viewpoint image, wherein a first viewpoint of the first viewpoint image is adjacent to a second viewpoint of the second viewpoint image; andgenerating, based on the first viewpoint image, the second viewpoint image, and the disparity map, the plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has the distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence.

18. The computer-implemented method of claim 16, the method further comprising: receiving a set of viewpoint images and a set of disparity maps, wherein the set of viewpoint images are in a sequence of a set of viewpoint positions, and wherein each of the set of disparity maps corresponds to one of the set of viewpoint images; andgenerating, based at least on the set of viewpoint images and the set of disparity maps, the set of output images.

19. The computer-implemented method of claim 16, the method further comprising: generating, based on a set of viewpoint images, the set of output images, wherein the set of viewpoint images relates to in a predetermined sequence of viewpoint positions.

20. The computer-implemented method of claim 18, wherein a first number of output images in the set of output images is based on a second number of the viewpoint images in the set of viewpoint images, wherein the first number is less than a half of the second number, and wherein the second number corresponds to a third number of viewpoint positions.

21. The computer-implemented method of claim 16, the method comprising: aggregating luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andproviding the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.

22. The computer-implemented method of claim 18, the method comprising: aggregating luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andproviding the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.

23. A system for generating an image at a viewpoint, the system comprises: a processor; anda memory storing computer-executable instructions that when executed by the processor cause the system to: receive a first viewpoint image and a plurality of disparity maps, wherein the disparity map corresponds to the first viewpoint image;generate, based on the first viewpoint image and the disparity map, a plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has a distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence;generate, based on the plurality of disparity inducing edges and the first viewpoint image, a first set of pseudo viewpoint images;generate, based on polarity-inverted images associated with the plurality of disparity inducing edges and the first viewpoint image, a second set of pseudo viewpoint images;generate, based at least on the first set of pseudo viewpoint images and the second set of pseudo viewpoint images, a set of output images, wherein the set of output images are in the predetermined sequence, and wherein the set of output images include a plurality of phase modulations for canceling out upon a synthesis and visual confirmation; andprovide the set of output images.

24. The system of claim 23, the computer-executable instructions when executed further causing the system to: receive a second viewpoint image, wherein a first viewpoint of the first viewpoint image is adjacent to a second viewpoint of the second viewpoint image; andgenerate, based on the first viewpoint image, the second viewpoint image, and the disparity map, the plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has the distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence.

25. The system of claim 23, the computer-executable instructions when executed further causing the system to: receive a set of viewpoint images and a set of disparity maps, wherein the set of viewpoint images are in a sequence of a set of viewpoint positions, and wherein each of the set of disparity maps corresponds to one of the set of viewpoint images; andgenerate, based at least on the set of viewpoint images and the set of disparity maps, the set of output images.

26. The system of claim 23, the computer-executable instructions when executed further causing the system to: generate, based on a set of viewpoint images, the set of output images, wherein the set of viewpoint images relates to in a predetermined sequence of viewpoint positions.

27. The system of claim 25, wherein a first number of output images in the set of output images is based on a second number of the viewpoint images in the set of viewpoint images, wherein the first number is less than a half of the second number, and wherein the second number corresponds to a third number of viewpoint positions.

28. The system of claim 23, the computer-executable instructions when executed further causing the system to: aggregate luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andprovide the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.

29. The system of claim 25, the computer-executable instructions when executed further causing the system to: aggregate luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andprovide the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.

30. A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to: receive a first viewpoint image and a plurality of disparity maps, wherein the disparity map corresponds to the first viewpoint image;generate, based on the first viewpoint image and the disparity map, a plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has a distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence;generate, based on the plurality of disparity inducing edges and the first viewpoint image, a first set of pseudo viewpoint images;generate, based on polarity-inverted images associated with the plurality of disparity inducing edges and the first viewpoint image, a second set of pseudo viewpoint images;generate, based at least on the first set of pseudo viewpoint images and the second set of pseudo viewpoint images, a set of output images, wherein the set of output images are in the predetermined sequence, and wherein the set of output images include a plurality of phase modulations for canceling out upon a synthesis and visual confirmation; andprovide the set of output images.

31. The computer-readable non-transitory recording medium of claim 30, the computer-executable instructions when executed further causing the system to: receive a second viewpoint image, wherein a first viewpoint of the first viewpoint image is adjacent to a second viewpoint of the second viewpoint image; andgenerate, based on the first viewpoint image, the second viewpoint image, and the disparity map, the plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has the distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence.

32. The computer-readable non-transitory recording medium of claim 30, the computer-executable instructions when executed further causing the system to: receive a set of viewpoint images and a set of disparity maps, wherein the set of viewpoint images are in a sequence of a set of viewpoint positions, and wherein each of the set of disparity maps corresponds to one of the set of viewpoint images; andgenerate, based at least on the set of viewpoint images and the set of disparity maps, the set of output images.

33. The computer-readable non-transitory recording medium of claim 30, the computer-executable instructions when executed further causing the system to: generate, based on a set of viewpoint images, the set of output images, wherein the set of viewpoint images relates to in a predetermined sequence of viewpoint positions.

34. The computer-readable non-transitory recording medium of claim 32, wherein a first number of output images in the set of output images is based on a second number of the viewpoint images in the set of viewpoint images, wherein the first number is less than a half of the second number, and wherein the second number corresponds to a third number of viewpoint positions, the computer-executable instructions when executed further causing the system to: aggregating luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andproviding the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.

35. The computer-readable non-transitory recording medium of claim 30, the computer-executable instructions when executed further causing the system to: aggregate luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andprovide the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.