Image generation apparatus, image generation method, and program

Information

  • Patent Grant
  • 11265533
  • Patent Number
    11,265,533
  • Date Filed
    Wednesday, April 3, 2019
    5 years ago
  • Date Issued
    Tuesday, March 1, 2022
    2 years ago
  • CPC
    • H04N13/282
    • H04N13/111
  • Field of Search
    • US
    • 348 046000
    • CPC
    • H04N13/111
  • International Classifications
    • H04N13/282
    • H04N13/111
Abstract
Provided is an image generation technology capable of suppressing unpleasant feelings caused by fluctuation in image quality caused by the viewer's viewpoint movement. The image generation technology includes an image generation unit configured to generate a pseudo viewpoint image Iφk and a pseudo viewpoint image I−φk, by using disparity inducing edge Dφk having a phase difference φk from a viewpoint image I, where φk(1≤k≤K) is set as a real number satisfying 0<φ1< . . . <φK≤π/2, an output image generation unit configured to generate an output image Outm(1≤m≤2K−1), from the pseudo viewpoint image I(m)(1≤m≤2K+1), the viewpoint image I is set as a pseudo viewpoint image Iφ0, where the pseudo viewpoint images Iφk, I−φk (0≤k≤K) are arranged in a sequence of Iφk, Iφ(K−1), . . . , Iφ1, Iφ0(=I), I−φ1, . . . , and I−φK is set as I(1), I(2), . . . , I(K), I(K+1), I(K+2), . . . , I(2K+1), and the output image Outm and the output image Outm+1(1≤m≤2K−2) include a phase modulation component that is canceled out when synthesized and visually recognized.
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. 371 Application of International Patent Application No. PCT/JP2019/014774, filed on 3 Apr. 2019, which application claims priority to and the benefit of JP Application No. 2018-078327, filed on 16 Apr. 2018, the disclosures of which are hereby incorporated herein by reference in their entireties.


TECHNICAL FIELD

The present disclosure relates to an image generation technology, and particularly to an image generation technology for a viewer to perceive an image from a viewpoint located at an intermediate position between adjacent viewpoints.


BACKGROUND ART

A technology described in Non-Patent Literature 1, for example, for synthesizing luminance of the images (viewpoint images) from adjacent viewpoints smoothly based on the position of an intermediate viewpoint between the adjacent viewpoints (intermediate viewpoint position), and enabling a viewer to perceive an image from an intermediate viewpoint, is called “linear blending”.


The linear blending is advantageously used to reduce the number of projectors required for image display, because there is no need to prepare the image from the intermediate viewpoint position. In other words, the cost associated with image display can be reduced.


CITATION LIST
Non Patent Literature

Non-Patent Literature 1: Andrew Jones, Jonas Unger, Koki Nagano, Jay Busch, Xueming Yu, Hsuan-Yueh Peng, Oleg Alexander, Mark Bolas, Paul Debevec, “An Automultiscopic Projector Array for Interactive Digital Humans”, SIGGRAPH' 15, ACM SIGGRAPH 2015 Emerging Technologies Article No. 6, 2015.


SUMMARY OF THE INVENTION
Technical Problem

However, with the current linear blending, image quality may be degraded due to image multiplexing caused by overlapping images at the intermediate viewpoint position. In addition, while images at adjacent viewpoints that correspond to the positions of the projectors show no degradation in quality, images at the intermediate viewpoint positions show degradation in quality. When a viewer moves its viewpoint, a fluctuation in image quality occurs, causing the viewer to experience unpleasant feelings.


A description will be made with reference to FIG. 1. At a viewpoint A, a viewer visually recognizes only single viewpoint image (image 1) and therefore experiences a good image quality. However, at a viewpoint B, which is the intermediate viewpoint position, the viewer visually recognizes an image that is synthesized (visible at the same time) from the two viewpoint images (image 1 and image 2) and thus experiences a poor image quality due to overlapping. As a result, when the viewer moves in a sequence of A→B→C, the image quality fluctuates from good→poor→good.


Thus, an object of the present disclosure is to provide an image generation technology capable of suppressing unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Means for Solving the Problem

An aspect of the present disclosure includes: an image generation unit configured to generate, from a viewpoint image I and a disparity map Dmap corresponding to the viewpoint image I, a disparity inducing edge Dφk having a phase difference φk from the viewpoint image I for k=1, . . . , K, generate a pseudo viewpoint image Iφk, by adding the disparity inducing edge Dφk to the viewpoint image I, and generate a pseudo viewpoint image I−φk by adding a polarity-inversed image of the disparity inducing edge Dφk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2; and


an output image generation unit configured to generate an output image Outm (1≤m≤2K−1) from the pseudo viewpoint image I(m) (1≤m≤2K+1), where the viewpoint image I is set as the pseudo viewpoint image Iφ0, and the pseudo viewpoint images Iφk and I−φk (0≤k≤K) being arranged in a sequence of IφK, Iφ(K−1), . . . , Iφ1, Iφ0 (=I), I−φ1, . . . , and I−φK are set as I(1), I(2), . . . , I(K), I(K+1), I(K+2), . . . , and I(2K+1);


wherein the output image Outm and output image Outm+1 (1≤m≤2K−2) include a phase modulation component that is canceled out when synthesized and visually recognized.


Another aspect of the present disclosure includes: an image generation unit configured to generate, from a viewpoint image I and a viewpoint image I′ captured from a viewpoint position adjacent to a viewpoint position at which the viewpoint image I is captured, a disparity inducing edge Dφk having a phase difference φk from the viewpoint image I for k=1, K, generate a pseudo viewpoint image Iφk, by adding the disparity inducing edge Dφk to the viewpoint image I, and generate a pseudo viewpoint image I−φk by adding a polarity-inversed image of the disparity inducing edge Dφk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2; and


an output image generation unit configured to generate an output image Outm (1≤m≤2K−1) from the pseudo viewpoint image I(m) (1≤m≤2K+1), where the viewpoint image I is set as a pseudo viewpoint image Iφ0, and the pseudo viewpoint images Iφk and I−φk (0≤k≤K) being arranged in a sequence of IφK, Iφ(K−1), Iφ1, Iφ0 (=I), I−φ1, . . . , and I−φK are defined as I(1), I(2), . . . , I(K), I(K+1), I(K+2) and I(2K+1);


wherein the output image Outm and output image Outm+1 (1≤m≤2K−2) include a phase modulation component that is canceled out when synthesized and visually recognized.


Another aspect of the present disclosure includes: an output image generation unit configured to generate an output image Outm (1≤m≤M−1) from a viewpoint image I(m) and a disparity map Dmap(m) (1≤m≤M), where I(1), I(2), . . . , I(M) are set as viewpoint images arranged in a sequence of viewpoint positions, and Dmap(m) (1≤m≤M) is set as a disparity map corresponding to the viewpoint image I(m),


wherein the output image Outm and output image Outm+1 (1≤m≤M−2) include a phase modulation component that is canceled out when synthesized and visually recognized.


Another aspect of the present disclosure includes: an output image generation unit configured to generate an output image Outm (1≤m≤M−1) from a viewpoint image I(m) (1≤m≤M+1), where I(1), I(2), . . . , I(M) are set as viewpoint images arranged in a sequence of viewpoint positions,


wherein the output image Outm and output image Outm+1 (1≤m≤M−2) include a phase modulation component that is canceled out when synthesized and visually recognized.


Another aspect of the present disclosure includes: an output image generation unit configured to generate an output image Outj (1≤j≤J−1) from a viewpoint image IO) (1≤j≤2J+1), where I(1), I(2), . . . . I(2J+1) are set as viewpoint images arranged in a sequence of viewpoint positions, wherein the output image Outj and output image Outj+1 (1≤j≤J−2) include a phase modulation component that is canceled out when synthesized and visually recognized.


Another aspect of the present disclosure includes: an image generation unit configured to, from a viewpoint image I and a disparity map Dmap corresponding to the viewpoint image I, generate a disparity inducing edge Dφk having a phase difference φk from the viewpoint image I for k=1, . . . . K, generate a pseudo viewpoint image Iφk, by adding the disparity inducing edge Dφk to the viewpoint image I, and generate a pseudo viewpoint image I−φk by adding a polarity-inversed image of the disparity inducing edge Dφk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2,


wherein the viewpoint image I is set as a pseudo viewpoint image Iφ0, and for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, a pseudo viewpoint image Iφk and a pseudo viewpoint image Iφ(k+1) are viewpoint images at viewpoint positions adjacent to each other, and are images used for presenting an image obtained by synthesizing luminance of at least two images of the pseudo viewpoint image Iφk and the pseudo viewpoint image Iφ(k+1) as an image at a viewpoint located at an intermediate position between the viewpoint positions adjacent to each other.


Another aspect of the present disclosure includes: an image generation unit configured to generate, from a viewpoint image I and a viewpoint image I′ captured from a viewpoint position adjacent to a viewpoint position at which the viewpoint image I is captured, a disparity inducing edge Dφk having a phase difference φk from the viewpoint image I for k=1, . . . , K, generate a pseudo viewpoint image Iφk, by adding the disparity inducing edge Dφk to the viewpoint image I, and generate a pseudo viewpoint image I−φk by adding a polarity-inversed image of the disparity inducing edge Dφk to the viewpoint image I, where K is an integer of one or more, and φk(1≤k≤K) is a real number satisfying 0<φ1< . . . <φK≤π/2,


wherein the viewpoint image I is set as a pseudo viewpoint image Iφ0, and, for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, the pseudo viewpoint image Iφk and the pseudo viewpoint image Iφ(k+1) are viewpoint images in viewpoint positions adjacent to each other, and are images used for presenting an image obtained by synthesizing luminance of at least two images of the pseudo viewpoint image Iφk and the pseudo viewpoint image Iφ(k+1) as an image at a viewpoint located at an intermediate position between the viewpoint positions adjacent to each other.


Effects of Invention

According to the present disclosure, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.





BRIEF DESCRIPTION OF DRAWINGS


FIG. 1 is a diagram illustrating a related art (linear blending).



FIG. 2 is a block diagram illustrating a configuration of an image generation apparatus 1.



FIG. 3 is a flowchart illustrating an operation of the image generation apparatus 1.



FIG. 4 is a block diagram illustrating a configuration of an image generation unit 10.



FIG. 5 is a diagram illustrating a state of fluctuation in image quality associated with the viewpoint movement.



FIG. 6 is a block diagram illustrating a configuration of an image generation apparatus 2.



FIG. 7 is a flowchart illustrating an operation of the image generation apparatus 2.



FIG. 8 is a block diagram illustrating a configuration of an image generation unit 20.



FIG. 9 is a block diagram illustrating a configuration of an image generation apparatus 3.



FIG. 10 is a flowchart illustrating an operation of the image generation apparatus 3.



FIG. 11 is a block diagram illustrating a configuration of an output image generation unit 30.



FIG. 12 is a flowchart illustrating an operation of the output image generation unit 30.



FIG. 13 is a block diagram illustrating a configuration of an image-for-synthesis generation unit 31.



FIG. 14 is a diagram illustrating the technology of the present application.



FIG. 15 is a block diagram illustrating a configuration of an image generation apparatus 4.



FIG. 16 is a flowchart illustrating an operation of the image generation apparatus 4.



FIG. 17 is a block diagram illustrating a configuration of an image generation apparatus 5.



FIG. 18 is a flowchart illustrating an operation of the image generation apparatus 5.



FIG. 19 is a block diagram illustrating a configuration of an image generation apparatus 6.



FIG. 20 is a flowchart illustrating an operation of the image generation apparatus 6.



FIG. 21 is a block diagram illustrating a configuration of an image generation apparatus 7.



FIG. 22 is a flowchart illustrating an operation of the image generation apparatus 7.



FIG. 23 is a diagram illustrating a relationship between a viewpoint image and an output image when J=5.



FIG. 24 is a block diagram illustrating a configuration of an image-for-synthesis generation unit 71.



FIG. 25 is a diagram illustrating a result of an experiment.





DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that components having the same function are given the same reference numerals, and redundant descriptions are omitted.


First Embodiment

Hereinafter, an image generation apparatus 1 will be described with reference to FIG. 2 and FIG. 3. FIG. 2 is a block diagram illustrating a configuration of the image generation apparatus 1. FIG. 3 is a flowchart illustrating operations of the image generation apparatus 1. As illustrated in FIG. 2, the image generation apparatus 1 includes an image generation unit 10 and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 1.


The image generation apparatus 1 receives, as an input, the viewpoint image I and a disparity map Dmap corresponding to the viewpoint image I, generates 2K pseudo viewpoint images (K is an integer of 1 or greater), Iφk, I−φk (1≤k≤K) from the viewpoint image I and the disparity map Dmap, and outputs the pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents a phase difference between the viewpoint image I and the disparity inducing edge Dφk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. The disparity inducing edge Dφk is an image obtained by modulating a phase to make a phase difference from the viewpoint image I to be φk. Further, the pseudo viewpoint image Iφk, I−φk are images both at the viewpoint positions different from the viewpoint position of the viewpoint image I (see FIG. 5). In the following, it is presented that φ0=0, Iφk=I, and the viewpoint image I may be referred to as a pseudo viewpoint image Iφ0.


Note that the disparity in the disparity map Dmap corresponding to the viewpoint image I may not be a disparity obtained at the viewpoint position where the viewpoint image I is captured. For example, the desired disparity may be used.


The operation of the image generation apparatus 1 will be described in accordance with FIG. 3.


In step S10, the image generation unit 10 receives, as an input, the viewpoint image I and a disparity map Dmap of the viewpoint image I, generates pseudo viewpoint image Iφk, I−φk, from the viewpoint image I and the disparity map Dmap for k=1, . . . , K, and outputs the pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I. A procedure for generating the pseudo viewpoint images Iφk, I−φk will be described. First, the image generation unit 10 generates a disparity inducing edge Dφk, which is a phase modulated image having a phase difference φk from the viewpoint image I, from the viewpoint image I and the disparity map Dmap. Next, the image generation unit 10 generates a pseudo viewpoint image Iφk by adding the disparity inducing edge Dφk to the viewpoint image I (Iφk=I+Dφk), and generates a pseudo viewpoint image I−φk by adding the polarity-inversed image of the disparity inducing edge Dφk to the viewpoint image I (I−φk=I−Dφk). The image generation unit 10 sequentially generates the pseudo viewpoint images Iφk, I−φk (1≤k≤K).


Note that the image generation apparatus 1 may further include a projection unit (not illustrated) that projects the pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I which are output at S10. The projection unit arranges the pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I in the sequence of phase differences from Iφk, Iφk(K−1), . . . , Iφ1, I, I−φ1, . . . , I−φK, that is, in the sequence of the viewpoint positions, and projects the arranged images onto a screen.


Further, for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, the pseudo viewpoint image Iφk and the pseudo viewpoint image Iφ(k+1) are viewpoint images at viewpoint positions adjacent to each other, and the synthesized luminance of at least two images of the pseudo viewpoint image Iφk and the pseudo viewpoint image Iφ(k+1) may be used to present the images at viewpoints located at an intermediate position between the viewpoint positions adjacent to each other.


Hereinafter, the configuration and operation of the image generation unit 10 will be described in detail with reference to FIG. 4. Here, a case in which the phase difference is π/2 will be described. In addition, in the description, a viewpoint image is referred to as an original image, a disparity inducing edge is referred to as a phase modulated image, and a pseudo viewpoint image is referred to as an added image.


Image Generation Unit 10


As illustrated in FIG. 4, the image generation unit 10 includes decomposition units 111, 112, spatial frequency domain conversion units 121, 122, spatial domain conversion units 123, 124, a weight calculation unit 13, a phase operation unit 14, a weighting unit 15, a reconstruction unit 16, and superposition units 17, 18.


The processing by the image generation unit 10 will be described.


The original image IC and the disparity map Dmap are input to the image generation unit 10. The original image IC is a two-dimensional array having IC (x, y) as elements (pixels). The original image IC may be a still image, an image of individual frame of a moving image, an image read from an external recording device, or an image read from the recording unit 90. The disparity map Dmap is a two-dimensional array (two-dimensional grayscale image) having d(x, y) as elements (pixels). Here, the size of Dmap is the same as the size of the original image IC, x represents the horizontal coordinate of the spatial domain, y represents the vertical coordinate of the spatial domain, and it is satisfied that xmin≤x≤xmax, ymin≤y≤ymax, xmin<xmax, ymin<ymax. d(x, y) is one of a positive value, a negative value, and zero. In Dmap, in a region where d(x, y)>0, a disparity (crossed disparity) causes the region to be seen nearer than the display surface or the screen surface, and in a region where d(x, y)<0, a disparity (uncrossed disparity) causes the region to be seen farther than the display surface or the screen surface. The larger the absolute value |d(x, y)|, the greater the disparity. Such a Dmap can provide a desired disparity to each image region.


The original image IC is input to the spatial frequency domain conversion unit 121. The spatial frequency domain conversion unit 121 converts the original image IC into the original image IC{tilde over ( )} in the spatial frequency domain, and outputs original image IC{tilde over ( )}. Here, the original image IC{tilde over ( )} in the spatial frequency domain is a two-dimensional array having IC{tilde over ( )}(ωx, ωy) as elements. ωx represents the spatial frequency in the horizontal direction, and ωy represents the spatial frequency in the vertical direction. For conversion from the original image IC to the original image IC{tilde over ( )}, for example, discrete Fourier transform can be used. It should be noted that the superscript “{tilde over ( )}” of “IC{tilde over ( )}” should be given directly above “I”. However, “{tilde over ( )}” may be given at the upper right of “IC” due to restrictions on the description in the specification (step S121).


The original image IC{tilde over ( )} in the spatial frequency domain is input to the decomposition unit 112. The decomposition unit 112 applies a complex steerable filter sequence Ψ to the original image IC{tilde over ( )} and obtains and outputs a complex steerable pyramid. Here, the steerable filter sequence Ψ includes a steerable filter Ψλ, μ, corresponding to the spatial frequency band λ and the orientation band μ. Here, λ is an integer index corresponding to a spatial frequency band having a predetermined width, and μ is an integer index corresponding to an orientation band having a predetermined width. The conditions, λmin≤λ≤λmax, μmin≤μ≤μmax, λminmax, μminmax, are satisfied. A smaller λ corresponds to a lower frequency band. As described below, for all combinations of λ and μ, the decomposition unit 112 multiplies the steerable filter Ψλ, μ to the original image IC{tilde over ( )} and obtains and outputs a complex steerable pyramid SCλ, μ{tilde over ( )} corresponding to each spatial frequency band λ and each orientation band μ.

[Equation 1]
{tilde over (S)}Cλ,μCΨλ,μ  (1)


It should be noted that the superscript “{tilde over ( )}” of “SCλ, μ{tilde over ( )}” should be given directly above “S” (see Equation (1)). However, “{tilde over ( )}” may be given at the upper right of “SCλ, μ” due to restrictions on the description in the specification. Further, the subscript “Cλ, μ” of “SCλ, μ{tilde over ( )}” should be given as in “Cλ, μ{tilde over ( )}” (see Equation (1)). However, it may be given as “Cλ, μ” due to restrictions on the description in the specification (step S112).


The complex steerable pyramid SCλ, μ{tilde over ( )} is input to the spatial domain conversion unit 123. The spatial domain conversion unit 123 converts the complex steerable pyramid SCλ, μ{tilde over ( )} into the complex steerable pyramid SCλ, μ in the spatial domain, and outputs a complex steerable pyramid SCλ, μ. For example, discrete inverse Fourier transform can be used for the conversion from the complex steerable pyramid SCλ, μ{tilde over ( )} to the complex steerable pyramid S Each complex steerable pyramid SCλ, μ is a two-dimensional array having SCλ, μ(x, y) as elements (pixels) (step S123).


The complex steerable pyramid SCλ, μ in the spatial domain is input to the phase operation unit 14. The phase operation unit 14 extracts an imaginary part Im [SCλ, μ (x, y)] of each complex steerable pyramid SCλ, μ, and obtains and outputs each phase shift image SC′λ, μ corresponding to each spatial frequency band λ and each orientation band μ as follows.









[

Equation





2

]













S

C

λ
,
μ






(

ϰ
,
y

)


=

{




Im


[


S

C

λ
,
μ





(

ϰ
,
y

)


]






if






θ
μ




π
/
2







-

Im


[


S

C

λ
,
μ





(

ϰ
,
y

)


]







if






θ
μ


>

π
/
2










(
2
)







Here, each phase shift image SC′λ, μ is a two-dimensional array having SC′λ, μ (x, y) as elements (pixels). The subscript “C′λ, μ” of “SC′λ, μ” should be given as “C′λ, μ” (see Equation (2)). However, it may be given as “C′λ, μ” due to restrictions on the description in the specification. θμ refers to the orientation (peak orientation) corresponding to the component having the strongest power among the orientations included in the orientation band μ. An example of θμ is the orientation of the center of the orientation band μ. θμ is the angle formed by the lattice in the spatial domain consisting of certain spatial frequency components with respect to the vertical orientation, and satisfies 0≤θμ≤π. The phase of the phase shift image SC′λ, μ, is obtained by shifting the phase of the sine wave component corresponding to the spatial frequency band λ and the orientation band μ of the original image IC by 0.5π [rad] in the positive direction. In Equation (2), the reason why the polarity is inversed when θμ>π/2 is because the phase is shifted by 0.5π [rad] in the same direction (in the positive direction) for both the range of 0≤θμ≤π/2 and the range of π≥θμ>π/2 (step S14).


The disparity map Dmap which has been input to the image generation unit 10 is input to the decomposition unit 111. The decomposition unit 111 applies a Gaussian filter sequence to the disparity map Dmap to obtain and output a Gaussian pyramid G (x, y) of the disparity map Dmap. The Gaussian pyramid G includes spatial frequencies corresponding to spatial frequency band λ and lower spatial frequency bands than the spatial frequency band λ. That is, G (x, y) represents the value of each coordinate (x, y) of the Gaussian pyramid of the disparity map Dmap corresponding to the spatial frequency band λ corresponding to ωλ and the spatial frequency band lower than the spatial frequency band λ. The subscript “Dλ,” of “G” should be given as “Dλ”. However, it may be given as “Dλ” due to restrictions on the description in the specification (step S111).


The Gaussian pyramid G (x, y) input to the weight calculation unit 13. The weight calculation unit 13 obtains and outputs a weight image Aλ, μ which is a two-dimensional array having each weight Aλ, μ (x, y) as an element as follows.









[

Equation





3

]













A

λ
,
μ




(

ϰ
,
y

)


=

tan





G

D
λ




(

ϰ
,
y

)




ω
λ





cos






θ
μ





2






(
3
)







Here, ωλ means a spatial frequency (peak spatial frequency) corresponding to a component having the strongest power among the spatial frequencies included in the spatial frequency band λ. An example of ωλ is the spatial frequency at the center of the spatial frequency band λ (step S13).


The phase shift image SC′λ, μ and weight image Aλ, μ are input to the weighting unit 15. As follows, the weighting unit 15 multiplies each Aλ, μ (x, y) of the weight image Aλ, μ to each element SC′λ, μ (x, y) of the phase shift image SC′λ, μ and obtains and outputs a weighted image S{circumflex over ( )}C′λ, μ which is a two-dimensional array having each S{circumflex over ( )}C′λ, μ (x, y) as an element.









[

Equation





4

]














S
^


C

λ
,
μ






(

ϰ
,
y

)


=



A

λ
,
μ




(

ϰ
,
y

)





S

C

λ
,
μ






(

ϰ
,
y

)







(
4
)







It should be noted that the superscript “{circumflex over ( )}” of “S{circumflex over ( )}” should be given directly above “S” (see Equation (4)). However, “{circumflex over ( )}” may be given at the upper right of “S” due to the restrictions on the description in the specification (step S15).


The weighted image S{circumflex over ( )}C′λ, μ is input to the spatial frequency domain conversion unit 122. The spatial frequency domain conversion unit 122 converts the weighted image S{circumflex over ( )}C′λ, μ into the weighted image S{circumflex over ( )}{tilde over ( )}C′λ, μ in the spatial frequency domain, and outputs the weighted image S{circumflex over ( )}{tilde over ( )}C′λ, μ. Note that, it may be described as “S{circumflex over ( )}{tilde over ( )}C′λ, μ” due to restrictions on the description in the specification, but “S{circumflex over ( )}{tilde over ( )}C′λ, μ” is synonymous with [Equation 5].









[

Equation





5

]













S
^

~


C

λ
,
μ















Note that, for example, a discrete Fourier transform can be used for the conversion from the weighted image S{circumflex over ( )}C′λ, μ to the weighted image S{circumflex over ( )}{tilde over ( )}C′λ, μ (step S122).


The weighted image S{circumflex over ( )}{tilde over ( )}C′λ, μ is input to the reconstruction unit 16. The reconstruction unit 16 applies the above-described steerable filter sequence Ψ to the weighted image S{circumflex over ( )}{tilde over ( )}C′λ, μ and obtains and outputs the phase modulated image ID{tilde over ( )} in the spatial frequency domain as follows (step S16).









[

Equation





6

]













I
~

D

=




λ
,
μ





Ψ

λ
,
μ






S
^

~


C

λ
,
μ










(
5
)







The phase modulated image ID{tilde over ( )} in the spatial frequency domain is input to the spatial domain conversion unit 124. The spatial domain conversion unit 124 converts the phase modulated image ID{tilde over ( )} in the spatial frequency domain into the phase modulated image ID in the spatial domain, and outputs the phase modulated image ID. Here, the phase modulated image ID is a two-dimensional array having ID (x, y) as elements (pixels). For conversion from the phase modulated image ID{tilde over ( )} to the phase modulated image ID, for example, a discrete inverse Fourier transform can be used (step S124).


The superposition unit 17 receives the original image IC and the phase modulated image ID as inputs, and obtains and outputs the added image IR (image A), by superimposing a polarity-inversed image (phase modulation component a) of the phase modulated image ID on the original image IC. When the value of the disparity map Dmap is positive (crossed disparity) in a certain region, the phase of the phase modulated image ID in the corresponding region is shifted from the phase of the original image IC by 0.5π [rad] in the positive direction. Thus, the phase of the polarity-inversed image (phase modulation component a) of the phase modulated image ID is obtained by shifting the phase of the original image IC by 0.5π [rad] in the negative direction. On the other hand, when the value of the disparity map Dmap is negative in a certain region, the phase of the phase modulated image ID is obtained by shifting the phase of the original image IC by 0.5π [rad] in the negative direction. Thus, the phase of the polarity-inversed image (phase modulation component a) of the phase modulated image ID in this region is obtained by shifting the phase of the original image IC by 0.5π [rad] in the negative direction. The pixel value IR (x, y) of each coordinate (x, y) of the added image IR is obtained by adding the pixel value ID (x, y) of each coordinate (x, y) of the polarity-inversed image of the phase modulated image ID to the pixel value IC (x, y) of each coordinate (x, y) of the original image IC (IR (x, y)=IC (x, y) ID (x, y)). Note that the polarity-inversed image of the phase modulated image ID and the added image IR correspond to “image containing phase modulation component a” obtained by giving the weight corresponding at least each “first region” to each “first region” of the image obtained by changing the phase of the original image IC (step S17).


The superposition unit 18 receives the original image I c and the phase modulated image ID as input, and superimposes the original image IC and the phase modulated image ID (phase modulation component b) and obtains and outputs an added image IL (image B). The phase modulation component b is an inverse phase image of the phase modulation component a. When the value of the disparity map Dmap is positive (crossed disparity) in a certain region, the phase of the phase modulated image ID in the corresponding region is shifted from the phase of the original image IC by 0.5π [rad] in the positive direction. Thus, the phase of the phase modulation component b in this region is obtained by shifting the phase of the original image IC by 0.5π [rad] in the positive direction. On the other hand, when the value of the disparity map Dmap is negative in a certain region, the phase of the phase modulated image ID in the corresponding region is shifted from the phase of the original image IC by 0.5π [rad] in the negative direction. Thus, the phase of the phase modulation component b in this region is obtained by shifting the phase of the original image IC by 0.5π [rad] in the negative direction. The pixel value IL (x, y) of each coordinate (x, y) of the added image IL is obtained by adding the pixel value IC (x, y) of each coordinate (x, y) of the original image IC and the pixel value ID (x, y) of each coordinate (x, y) of the phase modulated image ID (IL (x, y)=IC (x, y)+ID (x, y)). Note that the phase modulated image ID and the added image IL correspond to “image containing phase modulation component b” obtained by giving the weight corresponding at least each “second region” to each “second region” of the image obtained by changing the phase of the original image IC (step S18).


Hereinafter, modification examples of the image generation unit 10 will be described.


Modification Example 1

Theoretically, the maximum value of the disparity between the added images IR and IL is a half wavelength of each wavelength. However, in order to obtain a shift of a half wavelength, the above-described weight Aλ, μ (x, y) needs to be infinite, which is not practical. Thus, the absolute values of the phase shift amounts of the added images IR, IL with respect to the original image IC may be limited to π/4 [rad] or less. Under such restrictions, the absolute value of the weight Aλ, μ (x, y) becomes 1 or less, and the absolute value of the disparity between the added images IR, IL is less than π/(2ωλ |cos θμ|). In this case, the weight calculation unit 13 obtains and outputs a weight image Aλ, μ which is a two-dimensional array having each weight Aλ, μ (x, y) as an element, according to the following Equation (6) instead of Equation (3).









[

Equation





7

]













A

λ
,
μ




(

ϰ
,
y

)


=

tan






G
^


D
λ




(

ϰ
,
y

)




ω
λ





cos






θ
μ





2






(
6
)





where










[

Equation





8

]














G
^


D
λ




(

ϰ
,
y

)


=

{




π

2


ω
λ





cos






θ
μ










if







G

D
λ




(

ϰ
,
y

)



>

π

2


ω
λ





cos






θ
μ












-

π

2


ω
λ





cos






θ
μ











if







G

D
λ




(

ϰ
,
y

)



<

-

π

2


ω
λ





cos






θ
μ














G

D
λ




(

ϰ
,
y

)




otherwise
















When this modification example is applied, comparing the weights of the spatial frequency bands included in the same orientation band, the weight corresponding to the spatial frequency band λ1 (first spatial frequency band) having a certain spatial frequency ωλ1 as a peak is less than or equal to the weight corresponding to the spatial frequency band λ2 (second spatial frequency band) having the peak at the spatial frequency ωλ2 higher than the spatial frequency band λ1 (that is, ωλ2>ωλ1). Further, when the weights of the orientation bands included in the same spatial frequency band are compared, the weight corresponding to the orientation band μ1 (first orientation band) having a certain orientation θμ1 as the peak is less than or equal to the weight corresponding to the orientation band μ2 (second orientation band) having as the peak, orientation θμ2 which is closer to the vertical orientation than the orientation band μ1 (0 or π) (that is, |θμ2−π/2|>|θμ1−π/2|). The rest is as described for the image generation unit 10.


Modification Example 2

The image generation unit 10 superimposes the polarity-inversed image of the phase modulated image ID and the phase modulated image ID on the original image IC to obtain added images IR and IL having disparity. The amplitudes of the added images IR and IL obtained in this manner are larger than the amplitude of the original image IC, and the added images IR and IL may exceed the predetermined lower limit bL and/or upper limit bU (bL<bU), that is, a predetermined dynamic range. As a simple countermeasure against this, a method of linearly compressing the total intensity of the added images IR and IL such that the added images IR and IL fall within the range between the lower limit bL and the upper limit bU is conceivable. However, with this simple measure, the contrast of the added images IR and IL is smaller than that of the original image IC, and the impression of the image changes. In addition, a countermeasure for deleting a portion exceeding the lower limit value bL and/or the upper limit value bU of the added images IR and IL may be considered. In this case, there is a possibility that a doubly blurred image may be perceived when the added images IR and IL are not canceled out and the added images IR and IL are viewed at once by both eyes. Thus, in Modification Example 2, the phase modulated image ID corresponding to a portion exceeding the lower limit bL or the upper limit bU of the added images IR and IL is clipped.


More specific description will be made. In Modification Example 2, the image generation unit 10 further includes a dynamic range adjustment unit 161 (see FIG. 4). The dynamic range adjustment unit 161 receives the phase modulated image ID and the added images IR and IL obtained by the superposition units 17, 18 as inputs, updates the phase modulated image ID into a phase modulated image ID{circumflex over ( )} as follows, and outputs the image.

[Equation 9]
ÎD(x,y)=ID(x,y)−MU(x,y)−ML(x,y)  (7)


Here, the phase modulated image ID{circumflex over ( )} is a two-dimensional array including elements ID{circumflex over ( )}(x, y), and MU (x, y)=max (max (IR (x, y)−bU, 0), max (IL (x, y)−bU, 0)), ML (x, y)=min (min (IR (x, y)−bL, 0), min (IL (x, y)−bL, 0)). max (α1, α2) represents the larger value of α1 and α2, and min (α1, α2) represents the smaller value of α1 and α2. Note that, “{circumflex over ( )}” in “ID{circumflex over ( )}” should be given directly above “I” (see Equation (7)). However, “{circumflex over ( )}” may be given at the upper right of “ID” due to restrictions on the description in the specification.


In this case, the superposition unit 17 further receives the original image IC and the phase modulated image ID{circumflex over ( )}, and obtains and outputs the added image IR (image A) by superimposing a polarity-inversed image (phase modulation component a) of the phase modulated image ID{circumflex over ( )} on the original image IC. The pixel value IR (x, y) of each coordinate (x, y) of the added image IR is obtained by subtracting the pixel value ID{circumflex over ( )}(x, y) of each coordinate (x, y) of the phase modulated image ID{circumflex over ( )} from the pixel value IC (x, y) of each coordinate (x, y) of the original image IC (IR (x, y)=IC (x, y)−ID{circumflex over ( )}(x, y)). In this case, the added image IR obtained in step S17 described above is not output as the image A.


The superposition unit 18 further receives the original image IC and the phase modulated image ID{circumflex over ( )} as inputs, and obtains and outputs an added image IL (image B) by superimposing the original image IC and the phase modulated image ID{circumflex over ( )} (phase modulation component b). The pixel value IL (x, y) of each coordinate (x, y) of the added image IL is obtained by adding the pixel value IC (x, y) of each coordinate (x, y) of the original image IC and the pixel value ID (x, y) of each coordinate (x, y) of the phase modulated image ID{circumflex over ( )}(IL (x, y)=IC (x, y)+ID{circumflex over ( )}(x, y)). In this case, the added image IL obtained in step S18 described above is not output as the image B.


Modification Example 3

When the original image IC is a color image, the processes of steps S121, S112, S123, S14, S15, S122, S16, S124, S17, and S18 are executed on each color channel (R, G, and B channels), and the added image IR (image A) and the added image IL (image B) of all color channels may be output. Modification Examples 1 and 2 may be performed even when the original image IC is a color image, and the process of Modification Example 2 in this case is performed for each color channel.


Modification Example 4

The image generation unit 10 sets an image obtained by shifting the phase of the original image IC in the positive direction by 0.5π [rad] as a phase-shifted image SC′λ, μ (Equation (2)). However, an image obtained by shifting the phase of the original image IC in the positive direction by an amount close to 0.5π may be used as the phase shift image SC′λ, μ. In this case, the phase of the region of the polarity-inversed image (phase modulation component a) of the phase modulated image ID corresponding to the region where the value of the disparity map Dmap is positive (crossed disparity) is obtained by shifting the phase of the original image IC by an amount close to 0.5π [rad] in the negative direction. On the other hand, the phase of the region of the polarity-inversed image (phase modulation component a) of the phase modulated image ID corresponding to the region where the value of the disparity map Dmap is negative is obtained by shifting the phase of the original image IC by an amount close to 0.5π [rad] in the positive direction. The phase of the region of the phase modulated image ID (phase modulation component b) corresponding to the region where the value of the disparity map Dmap is positive (crossed disparity) is obtained by shifting the phase of the original image IC by an amount close to 0.5π [rad] in the positive direction. On the other hand, the phase of the region of the phase modulated image ID (phase modulation component b) corresponding to the region where the value of the disparity map Dmap is negative is obtained by shifting the phase of the original image IC by an amount close to 0.5π [rad] in the negative direction. Further, instead of using the polarity-inversed image of the phase modulated image ID as the phase modulation component a and the phase modulated image ID as the phase modulation component b, the polarity-inversed image of the phase modulated image ID may be the phase modulation component a, and an image near the opposite phase of the phase modulation component a may be the phase modulation component b.


Others


As described in (image generation unit 10), a phase modulated image having a phase difference of π/2 is generated by using Equation (2). By using a predetermined equation instead of Equation (2), a phase modulated image having any phase difference can be generated.


The image generation unit 10 has been described in detail including modification examples in the above. Hereinafter, a case where K=2, φ1=π/4, φ2=π/2 will be described as an example (see FIG. 5). The image generation unit 10 generates the pseudo viewpoint images Iπ/4, I−π/4 and the pseudo viewpoint images Iπ/2, I−π/2, and outputs the pseudo viewpoint images Iπ/2, I−π/2, Iπ/4, and I−π/4 and the viewpoint image I. When these images are projected onto a screen using five projectors (projection apparatuses) in the sequence of Iπ/2, Iπ/4, I, I−π/4, I−π/2, and presented to an viewer, as the viewer moves, the viewer observes, for example, in the sequence of Iπ/2, Iπ/4, I, I−π/4, I−π/2. At this time, the image quality increases substantially linearly up to the images Iπ/2, Iπ/4, and I. Then, with the image quality of the image I as a peak, the image quality decreases substantially linearly up to I, I−π/4, I−π/2. For example, at the viewpoint position 1, the intermediate viewpoint position between the viewpoint position 1 and the viewpoint position 2, and the viewpoint position 2, the image quality gradually increases, so that the image quality does not fluctuate from good→poor→good as in the linear blending in the art. This enables to suppress unpleasant feelings that a viewer experiences due to fluctuation in image quality.


According to the disclosure of the present embodiment, it is possible to generate any number of pseudo viewpoint images at a viewpoint position different from the viewpoint position of the viewpoint image I, from one viewpoint image I and the disparity map Dmap. Further, by arranging the pseudo viewpoint image and the viewpoint image I in a sequence of the phase difference and projecting with the projector, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Second Embodiment

Hereinafter, an image generation apparatus 2 will be described with reference to FIG. 6 and FIG. 7. FIG. 6 is a block diagram illustrating a configuration of the image generation apparatus 2. FIG. 7 is a flowchart illustrating the operation of the image generation apparatus 2. As illustrated in FIG. 6, the image generation apparatus 2 includes an image generation unit 20 and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 2.


The image generation apparatus 2 receives, as inputs, the viewpoint image I and an adjacent viewpoint image I′, generates 2K (K is an integer of 1 or more) pseudo viewpoint images Iφk, I−φk from the viewpoint image I and an adjacent viewpoint image I′ (1≤k≤K), and outputs pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents the phase difference between the viewpoint image I and the disparity inducing edge Dφk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. The disparity inducing edge Dφk is an image obtained by modulating a phase such that a phase difference from the viewpoint image I becomes φk. Further, the pseudo viewpoint image Iφk, I−φk are images both at the viewpoint position different from the viewpoint position of the viewpoint image I (see FIG. 5). In the following, it is presented that φ0=0, Iφ0=I, and the viewpoint image I may be referred to as a pseudo viewpoint image Iφ0.


Note that, assuming a plurality of viewpoint positions horizontally arranged at equal intervals toward an object to be captured, an image captured by the camera from each viewpoint position is a viewpoint image, and an image captured from an adjacent viewpoint position is referred to as “the adjacent viewpoint image”. That is, the image generation apparatus 2 receives the viewpoint image I and the viewpoint image I′ captured from the viewpoint position adjacent to the viewpoint position where the viewpoint image I is captured.


The operation of the image generation apparatus 2 will be described in accordance with FIG. 7.


In step S20, the image generation unit 20 receives, as an input, the viewpoint image I and the adjacent viewpoint image I′, generates a pseudo viewpoint image Iφk, I−φk, from the viewpoint image I and the viewpoint image I′, for k=1, . . . , K, and outputs the pseudo viewpoint image Iφk, I−φk (1≤k≤K) and the viewpoint image I. A procedure for generating the pseudo viewpoint images Iφk, I−φk will be described. First, the image generation unit 20 generates a disparity inducing edge Dφk, which is a phase modulated image having a phase difference φk from the viewpoint image I, from the viewpoint image I and the viewpoint image I′. Next, the image generation unit 20 generates the pseudo viewpoint image Iφk (Iφk=I+Dφk) by adding the disparity inducing edge Dφk to the viewpoint image I, and generates the pseudo viewpoint image I−φk (I−φk=I−Dφk) by adding the polarity-inversed image of the disparity inducing edge Dφk to the viewpoint image I. The image generation unit 20 sequentially generates the pseudo viewpoint images Iφk, I−φk (1≤k≤K).


Note that the image generation apparatus 2 may further include a projection unit (not illustrated) that projects the pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I output at S20. The projection unit arranges the pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I in an sequence of phase differences from custom characterIφk, Iφ(K−1), . . . , Iφ1, I, I−φ1, . . . , I−φK, that is, in a sequence of the viewpoint position, and projects the arranged images onto the screen.


Further, for k=−K, −(K−1), . . . , −1, 0, 1, . . . , K−1, the pseudo viewpoint image Iφk and the pseudo viewpoint image Iφ(k+1) are viewpoint images at viewpoint positions adjacent to each other, and the synthesized luminance of at least two images of the pseudo viewpoint image Iφk and the pseudo viewpoint image Iφ(k+1) may be used to present the images at viewpoints located in an intermediate position between the viewpoint positions adjacent to each other.


Hereinafter, the configuration and operation of the image generation unit 20 will be described in detail with reference to FIG. 8. Here, a case in which the phase difference is π/2 will be described. In addition, in the description, a viewpoint image is referred to as an original image, a disparity inducing edge is referred to as a phase modulated image, and a pseudo viewpoint image is referred to as an added image.


Image Generation Unit 20


As illustrated in FIG. 8, the image generation unit 20 includes spatial frequency domain conversion units 122, 221, 222, spatial domain conversion units 124, 224, 225, decomposition units 211, 212, a phase operation unit 23, and a phase difference calculation unit 24, a weight calculation unit 25, a weighting unit 15, a reconstruction unit 16, and superposition units 17, 18.


The processing by the image generation unit 20 will be described.


An original image IL′=IC (first original image) and an original image IR′ (second original image) are input to the image generation unit 20. The original image IL′ and the original image IR′ are, for example, three-dimensional image contents for giving images an apparent depth by a binocular image presentation technology. There is a phase difference recognized as binocular disparity between the original image IL′ and the original image IR′. In other words, there is a phase difference recognized as binocular disparity between the original image IL′ and the original image IR′. When the original image IR′ is viewed by one eye (for example, the right eye) and the original image IL′ is viewed by the other eye (for example, the left eye), a “stereoscopic image” is perceived. On the other hand, when both the original image IR′ and the original image IL′ are viewed with both eyes, a doubly blurred image is perceived. The original image IR, is a two-dimensional array having IR′ (x, y) as an element, and the original image ILS is a two-dimensional array having IL′ (x, y) as elements (pixels). The original image IR′ and the original image IL′ may be a still image, an image of individual frame of a moving image, an image read from an external recording device, or an image read from the recording unit 90.


The original image IL′ is input to the spatial frequency domain conversion unit 221, and the original image IR′ is input to the spatial frequency domain conversion unit 222. The spatial frequency domain conversion unit 221 converts the original image IL′ into the original image IL′{tilde over ( )} in the spatial frequency domain, and outputs the original image IL′{tilde over ( )}. The spatial frequency domain conversion unit 222 converts the original image IR′ into the original image IR′{tilde over ( )} in the spatial frequency domain, and outputs the original image IR′{tilde over ( )}. For the conversion in the spatial frequency domain conversion units 221 and 222, for example, a discrete Fourier transform can be used. It should be noted that the superscript “{tilde over ( )}” of “IL′{tilde over ( )}” “IR′{tilde over ( )}” should be given directly above “I”. However, “{tilde over ( )}” may be given at the upper right of “IL” and “IR” due to restrictions on the description in the specification (steps S221, S222).


The original image IR′{tilde over ( )} in the spatial frequency domain is input to the decomposition unit 211, and the original image IL′{tilde over ( )} in the spatial frequency domain is input to the decomposition unit 212. The decomposition units 211 and 212 apply the complex steerable filter sequence Ψ to the original images IR′{tilde over ( )} and IL′{tilde over ( )} as described below, and obtain and output each complex steerable pyramid SL′λ, μ{tilde over ( )} and SR′λ, μ{tilde over ( )} corresponding to each spatial frequency band λ and each orientation band μ (steps S211 and S212).









[

Equation





10

]














S
~


L

λ
,
μ




=



I
~


L





Ψ


λ
,
μ

















S
~


R

λ
,
μ




=



I
~


R





Ψ


λ
,
μ





















The complex steerable pyramids SL′λ, μ{tilde over ( )} and SR′λ, μ{tilde over ( )} are input to the spatial domain conversion units 224 and 225, respectively. The spatial domain conversion units 224 and 225 respectively convert the complex steerable pyramids SL′λ, μ{tilde over ( )} and SR′λ, μ{tilde over ( )} into the complex steerable pyramid SL′λ, μ{tilde over ( )} and SR′λ, μ{tilde over ( )} in the spatial domain, and output the complex steerable pyramids SL′λ, μ and SR′λ, μ. For the conversion in the spatial domain conversion units 224 and 225, for example, a discrete inverse Fourier transform can be used. Each complex steerable pyramid SL′λ, μ is a two-dimensional array having SL′λ, μ (x, y) as elements (pixels), and each complex steerable pyramid SR′λ, μ is a two-dimensional array having SR′λ, μ (x, y) as elements (steps S224, S225).


The complex steerable pyramids SL′λ, μ and SR′λ, μ are input to the phase difference calculation unit 24. The phase difference calculation unit 24 obtains and outputs the phase difference δλ, μ (x, y) between SL′λ, μ (x, y) and SR′λ, μ (x, y), by using the method described in Non-Patent Literature 1 (DIDYK, P., SITTHI-AMORN, P., FREEMAN, W., DURAND, F., AND MATUSIK, W. 2013, “Joint View Expansion and Filtering for Automultiscopic 3d Displays,” ACM Trans. Graph. 32, 6737 (November), 221:1-221: 8). A set having the phase difference δλ, μ (x, y) as an element is referred to as a phase difference δλ, μ, As described in reference Non-Patent Literature 1, for a spatial frequency band λ having a phase difference exceeding π/2 [rad], a value twice the phase difference δλ−1, μ (x, y) at the spatial frequency band λ−1 one lower than the spatial frequency band λ is defined as a phase difference δλ, μ (x, y).

δλ,μ(x,y)2δλ−1,μ(x,y)  [Equation 11]


Here, the peak spatial frequency ωλ of the spatial frequency band λ is twice or nearly twice the peak spatial frequency ωλ−1 of the spatial frequency band (λ−1) (step S24).


The phase difference δλ, μ is input to the weight calculation unit 25. The weight calculation unit 25 obtains and outputs a weight Aλ, μ (x, y) as follows (step S25).

Aλ,μ(x,y)=tan {δλ,μ(x,y)}  [Equation 12]


The complex steerable pyramid SL′λ, μ obtained by the spatial domain conversion unit 224 is input to the phase operation unit 23. The phase operation unit 23 extracts an imaginary part of the complex steerable pyramid SL′λ, μ, sets SC′λ, μ=SL′λ, μ, and in accordance with Equation (2), obtains and outputs each phase shift image SC′λ, μ corresponding to each spatial frequency band λ and each orientation band μ (step S224).


Thereafter, the processes of steps S15, S122, S16, and S124 described in (Image generation unit 10) are performed, and the spatial domain conversion unit 124 outputs the phase modulated image ID. Further, the process of step S17 is performed, and the superposition unit 17 obtains and outputs the added image IR (image A). Further, the superposition unit 18 performs the process of step S18 to obtain and output the added image IL (image B).


Hereinafter, modification examples of the image generation unit 20 will be described.


Modification Example 1

As described in Modification Example 1 of the image generation unit 10, the absolute value of the phase shift amount of the added image IR with respect to the original image IL, =IC may be limited to π/4 [rad] or less. In this case, the image generation unit 20 (see FIG. 8) further includes a phase difference limiting unit 241. The phase difference limiting unit 241 receives the phase difference δλ, μ (x, y) obtained by the phase difference calculation unit 24 as an input, and updates the phase difference δλ, μ (x, y) into δ{circumflex over ( )}λ, μ (x, y) as follows.









[

Equation





13

]














δ
~


λ
,
μ




(

ϰ
,
y

)


=

{




π
/
4





if








τδ

λ
,
μ




(

ϰ
,
y

)


/
2


>

π
/
4








-
π

/
4





if








τδ

λ
,
μ




(

ϰ
,
y

)


/
2


<


-
π

/
4









τδ

λ
,
μ




(

ϰ
,
y

)


/
2



otherwise
















Here, τ is a positive constant for scaling. By reducing the value of τ, the range where the phase difference is clipped to π/4 [rad] can be narrowed.


In this case, in step S25, δ{circumflex over ( )}λ, μ (x, y) is input to the weight calculation unit 25 instead of the phase difference δλ, μ (x, y). In this case, the weight calculation unit 25 obtains and outputs a weight Aλ, μ (x, y) as follows.

Aλ,μ(x,y)=tan {{circumflex over (δ)}λ,μ(x,y)}  [Equation 14]


Subsequent processes are as described above.


Modification Example 2

As in Modification Example 2 of the image generation unit 10, the image generation unit 20 may further include the dynamic range adjustment unit 161 (see FIG. 8). In this case, as described in Modification Example 2 of the image generation unit 10, the dynamic range adjustment unit 161 updates the phase modulated image ID to the phase modulated image ID{circumflex over ( )} and outputs the phase modulated image, the superposition unit 17 superimposes a polarity-inversed image (phase modulation component a) of the phase modulated image ID{circumflex over ( )} to the original image IL′=IC to obtain and output an added image IR (image A), and the superposition unit 18 further receives the original image IL′=IC and the phase modulated image ID{circumflex over ( )} and superimposes the original image IL′=IC and the phase modulated image ID{circumflex over ( )}(phase modulation component b) to obtain and output an added image IL (image B).


Modification Example 3

When the original image IL′=IC and the original image IR′ are color images, processes of steps S221, S222, S211, S212, S224, S225, S24, S25, S23, S15, S122, S16, S124, S17, S18 are performed for each color channel (R, G, and B channels), and the added image IR (image A) and the added image IL (image B) of all color channels may be output. The process of step S24 may be performed on only one of the channels. Modification Examples 1 and 2 may be performed even when the original image IL′=IC is a color image, and the process of Modification Example 2 in this case is performed for each color channel.


Modification Example 4

The image generation unit 20 sets an image obtained by shifting the phase of the original image IL′=IC in the positive direction by 0.5π [rad] as a phase-shifted image SC′λ, μ (Equation (2)).


However, an image obtained by shifting the phase of the original image IL′=IC in the positive direction by an amount close to 0.5π may be used as the phase shift image S In this case, the phase of the polarity-inversed image (phase modulation component a) of the phase modulated image ID corresponding to the region where the value of the disparity map is positive (crossed disparity) is obtained by shifting the phase of the original image IL′=IC by an amount close to 0.5π [rad] in the negative direction, and the phase of the phase modulated image ID (phase modulation component b) corresponding to the region where the value of the disparity map is positive (crossed disparity) is obtained by shifting the phase of the original image IL′=IC by an amount close to 0.5π [rad] in the negative direction. Further, instead of using the polarity-inversed image of the phase modulated image ID as the phase modulation component a and the phase modulated image ID as the phase modulation component b, the polarity-inversed image of the phase modulated image ID may be the phase modulation component a, and an image near the opposite phase of the phase modulation component a may be the phase modulation component b.


Others


As described in (image generation unit 20), a phase modulated image having a phase difference of π/2 is generated by using Equation (2). By using a predetermined equation instead of Equation (2), a phase modulated image having any phase difference can be generated.


The image generation unit 20 has been described in detail including modification examples in the above. Note that, in a case where K=2, φ1=π/4, and φ2=π/2, the relationship between the viewpoint movement and the fluctuation in image quality is the same as in the image generation unit 10 (see FIG. 5).


According to the disclosure of the present embodiment, it is possible to generate any number of pseudo viewpoint images at a viewpoint position different from the viewpoint position of the viewpoint image I, from the viewpoint image I and the adjacent viewpoint image I′. Further, by arranging the pseudo viewpoint image and the viewpoint image I in a sequence of the phase difference and projecting with the projector, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Third Embodiment

Hereinafter, an image generation apparatus 3 will be described with reference to FIG. 9 and FIG. 10. FIG. 9 is a block diagram illustrating a configuration of the image generation apparatus 3. FIG. 10 is a flowchart illustrating the operation of the image generation apparatus 3. As illustrated in FIG. 9, the image generation apparatus 3 includes an image generation unit 10, an output image generation unit 30, and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 3.


The image generation apparatus 3 receives, as an input, the viewpoint image I and the disparity map Dmap corresponding to the viewpoint image I, and generates and outputs 2K (K is an integer of 2 or greater) output image Outm (1≤m≤2K−1) from the viewpoint image I and the disparity map Dmap. Here, the output image Outm is an image projected on a screen using a projector (projection apparatus).


The operation of the image generation apparatus 3 will be described in accordance with FIG. 10.


In step S10, the image generation unit 10 receives, as an input, the viewpoint image I and a disparity map Dmap of the viewpoint image I, generates pseudo viewpoint image Iφk, I−φk, from the viewpoint image I and the disparity map Dmap for k=1, . . . , K, and outputs the pseudo viewpoint images Iφk, I−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents the phase difference between the viewpoint image I and the disparity inducing edge Dφk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. In the following, it is presented that φ0=0 and Iφ0=I, and the viewpoint image I may be referred to as a pseudo viewpoint image Iφ0.


In step S30, the output image generation unit 30 receives, as an input, the pseudo viewpoint image Iφk, Iφk (1≤k≤K) and the viewpoint image I, which are output at S10, and generates and outputs an output image Outm (1≤m≤2K−1). The pseudo viewpoint images Iφk and I−φk (1≤k≤K) and the viewpoint image I are arranged in the sequence of IφK, Iφ(K−1), . . . , Iφ1, Iφ0(=I), I−φ1, . . . , and IφK to make I(1), I(2), . . . , I(K), I(K+1), I(K+2), . . . , and I(2K+1). That is, the output image generation unit 30 receives the pseudo viewpoint image I(m) (1≤m≤2K+1) as an input. Hereinafter, an output image generation unit 30 will be described with reference to FIG. 11 and FIG. 12. As illustrated in FIG. 11, the output image generation unit 30 includes an image-for-synthesis generation unit 31 and an output image synthesis unit 32. The operation of the output image generation unit 30 will be described in accordance with FIG. 12.


In step S31, the image-for-synthesis generation unit 31 receives the pseudo viewpoint image I(m) (1≤m≤2K+1) output in S10 as an input, generates images for synthesis I(m)L and I(m)R, for m=1, . . . , 2K, from the pseudo viewpoint image I(m) and the pseudo viewpoint image I(m+1) and outputs the image for synthesis I(m)L and I(m)R (1≤m≤2K). For example, the image-for-synthesis generation unit 31 may generate the images for synthesis I(m)L, I(m)R by the same procedure as the image generation unit 20. In this case, the image-for-synthesis generation unit 31 is a component having the same configuration as the image generation unit 20 (see FIG. 8). Further, after generating the disparity map Dmap(m) of the pseudo viewpoint image I(m) from the pseudo viewpoint image I(m) and the pseudo viewpoint image I(m+1) the image-for-synthesis generation unit 31 may generate images for synthesis I(m)L, I(m)R, from the pseudo viewpoint image I(m) and the disparity map Dmap(m), by the same procedure as that of the image generation unit 10. In this case, the image-for-synthesis generation unit 31 is a component including the disparity-map generation unit 311 and the image generation unit 10, as illustrated in FIG. 13. That is, the disparity map generation unit 311 generates the disparity map Dmap(m) of the pseudo viewpoint image I(m) from the pseudo viewpoint image I(m) and the pseudo viewpoint image I(m+1), and outputs the pseudo viewpoint image I(m) and the disparity map Dmap(m). In any case, in step S31, the image-for-synthesis generation unit 31 generates a disparity inducing edge D(m) whose the phase difference from the pseudo viewpoint image I(m) is a predetermined value (for example, π/2), from the pseudo viewpoint image I(m) and the pseudo viewpoint image I(m+1), for m=1, . . . , 2K, generates an image for synthesis I(m)L by adding the disparity inducing edge D(m) to the pseudo viewpoint image I(m) and generates an image for synthesis I(m)R by adding the polarity-inversed image of the disparity inducing edge D(m) to the pseudo viewpoint image I(m).


In step S32, the output image synthesis unit 32 receives the images for synthesis I(m)R and I(m)L (1≤m≤2K) generated in S31 as an input, synthesizes the output image Outm, from the images for synthesis I(m)R and I(m+1)L for m=1, . . . , 2K−1, and outputs the output image Outm (1≤m≤2K−1). For example, an output image may obtained by adding the RGB values of the respective pixels of the image for synthesis I(m)R and the image for synthesis I(m+1)L and dividing the sum by 2 (averaged value) as a pixel value.


The image generation apparatus 3 may further include a projection unit (not illustrated) that projects the output images Outm (1≤m≤2K−1) output in S30. The projection unit arranges the output images Outm (1≤m≤2K−1) in the sequence of Out1, Out2, . . . , and Out2K−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.


A description will be made below with reference to FIG. 14. The output image Out1 is an image obtained by synthesizing the images for synthesis I(1)R and I(2)L. Further, the output image Out2 is an image obtained by synthesizing the images for synthesis I(2)R and I(3)L. At the viewpoint position A, the output image Out1 is perceived. On the other hand, at the viewpoint position B, the output image Out1 and the output image Out2 are synthesized and visually recognized (simultaneously viewed). That is, at the viewpoint position B, the phase modulation component is canceled and the image for synthesis I(2) is mainly perceived. The image perceived at the viewpoint position B includes, as components that may be degraded, an image for synthesis I(1)R used for synthesizing the output image Out1 at the viewpoint position A and an image for synthesis I(3)L used for synthesizing the output image Out3 at the viewpoint position C, which are perceived. However, the influence of these components that may be degraded is less than half of the linear blending in the related art, so that the degradation in image quality may be mitigated. As a result, a viewer hardly experiences unpleasant feelings associated with fluctuation in image quality due to the viewer's viewpoint moving from A to C.


As can be understood from the above description, when the output image Outm and the output image Outm+1 are synthesized and visually recognized (visually recognized simultaneously), an image in which the disparity inducing edge D(m+1) as the phase modulation component is canceled out (pseudo viewpoint image I(m+1)) is mainly perceived. In other words, the output image generation unit 30 generates an output image Outm (1≤m≤2K−1) from the pseudo viewpoint image I(m) (1≤m≤2K+1), and the output image Outm and the output image Outm+1 (1≤m≤2K−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).


According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Fourth Embodiment

Hereinafter, an image generation apparatus 4 will be described with reference to FIG. 15 and FIG. 16. FIG. 15 is a block diagram illustrating a configuration of the image generation apparatus 4. FIG. 16 is a flowchart illustrating the operation of the image generation apparatus 4. As illustrated in FIG. 15, the image generation apparatus 4 includes an image generation unit 20, an output image generation unit 30, and a recording unit 90. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 4.


The image generation apparatus 4 receives the viewpoint image I and the adjacent viewpoint image I′ (that is, the viewpoint image I and the viewpoint image I′ captured from the viewpoint position adjacent to the viewpoint position where the viewpoint image I is captured) as inputs, and generates and outputs 2K (K is an integer of 2 or more) output image Outm (1≤m≤2K−1) from the viewpoint image I and the viewpoint image I′. Here, the output image Outm is an image projected on a screen using a projector (projection apparatus).


The operation of the image generation apparatus 4 will be described in accordance with FIG. 16.


In step S20, the image generation unit 20 receives, as an input, the viewpoint image I and the adjacent viewpoint image I′, generates a pseudo viewpoint image Iφk and I−φk from the viewpoint image I and the viewpoint image I′, k=1, K, and outputs the pseudo viewpoint image Iφk and I−φk (1≤k≤K) and the viewpoint image I. Here, φk(1≤k≤K) represents the phase difference between the viewpoint image I and the disparity inducing edge Dφk, and is a real number satisfying 0<φ1< . . . <φK≤π/2. In the following, it is presented that φ0=0 and Iφ0=I, and the viewpoint image I may be referred to as a pseudo viewpoint image Iφ0.


In step S30, the output image generation unit 30 receives the pseudo viewpoint image Iφk, I−φk (1≤k≤K) and the viewpoint image I, which are output at S20, and generates and outputs an output image Outm (1≤m≤2K−1).


The image generation apparatus 4 may further include a projection unit (not illustrated) that projects the output images Outm (1≤m≤2K−1) output in S30. The projection unit arranges the output images Outm (1≤m≤2K−1) in the sequence of Out1, Out2, . . . , and Out2K−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.


The output image Outm (1≤m≤2) is an image obtained by synthesizing the images for synthesis I(m)R and I(m+1)L, so that as in the image generation apparatus 3, a viewer hardly experiences unpleasant feelings associated with the fluctuation in image quality caused by the viewpoint of the viewer moving from A to C (see FIG. 14).


In other words, a sequence in which the pseudo viewpoint images Iφk and I−φk (1≤k≤K) and the viewpoint image I are arranged in the sequence of IφK, Iφ(K−1), . . . , Iφ1, Iφ0 (=I), I−φ1, . . . , and I−φK is set as I(1), I(2), . . . , I(k), I(K+1), I(K+2), . . . , I(2K+1), and the output image generation unit 30 generates output images Outm (1≤m≤2K−1) from the pseudo viewpoint images I(m) (1≤m≤2K+1), and the output image Outm and the output image Outm+1 (1≤m≤2K−2) include a phase modulation components that are canceled out when synthesized and visually recognized (visually recognized at the same time).


According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Fifth Embodiment

Hereinafter, an image generation apparatus 5 will be described with reference to FIG. 17 and FIG. 18. FIG. 17 is a block diagram illustrating a configuration of the image generation apparatus 5. FIG. 18 is a flowchart illustrating the operation of the image generation apparatus 5. As illustrated in FIG. 17, the image generation apparatus 5 includes an output image generation unit 50 and a recording unit 90. The output image generation unit 50 includes an image-for-synthesis generation unit 51 and an output image synthesis unit 52. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 5.


The image generation apparatus 5 receives, as inputs, a set of M viewpoint images I(m) and the disparity map Dmap(m) corresponding to the viewpoint images I(m) (1≤m≤M, M is an integer of 3 or greater), and generates and outputs M−1 output images Outm (1≤m≤M−1). Here, the viewpoint images I(1), I(2), . . . , I(M) are a sequence of viewpoint images arranged in the sequence of the viewpoint positions. The output image Outm is an image projected on a screen using a projector (projection apparatus). The interval between the adjacent viewpoint image I(m) and viewpoint image I(m+1) is the same as the interval between the adjacent output image Outm and the output image Outm+1, that is, the placement interval between the projectors that projects the output image Outm.


The operation of the image generation apparatus 5 will be described in accordance with FIG. 18.


In step S50, the output image generation unit 50 receives the viewpoint image I(m) and the disparity map Dmap(m) (1≤m≤M) as an input, and generates and outputs an output image Outm (1≤m≤M−1).


In step S51, the image-for-synthesis generation unit 51 receives the viewpoint image I(m) and the disparity map Dmap(m) (1≤m≤M) as inputs, generates images for synthesis I(m)L, I(m)R, from the viewpoint image I(m) and the disparity map Dmap(m), for m=1, . . . , M, and outputs the images for synthesis I(m)L, I(m)R (1≤m≤M). For example, the image-for-synthesis generation unit 51 may generate the images for synthesis I(m)L, I(m)R by the same procedure as the image generation unit 10. That is, the added images IL, IR are set to the images for synthesis I(m)L, I(m)R. In this case, the image-for-synthesis generation unit 51 is a component having the same configuration as the image generation unit 10 (see FIG. 4), and repeats a generation process on a set of M viewpoint images I(m) and disparity maps Dmap on). In other words, in step S51, the image-for-synthesis generation unit 51 generates a disparity inducing edge D(m) whose the phase difference from the viewpoint image I(m) is a predetermined value (for example, π/2), from the viewpoint image I(m) and the disparity map Dmap(m), for m=1, . . . , M, generates an image for synthesis I(m)L by adding the disparity inducing edge D(m) to the viewpoint image I(m) and generates an image for synthesis I(m)R by adding the polarity-inversed image of the disparity inducing edge D(m)) to the viewpoint image I(m).


In step S52, the output image synthesis unit 52 receives the images for synthesis I(m)R and I(m)L (1≤m≤M) generated in S51 as inputs, synthesizes the output image Outm, from the images for synthesis I(m)R and I(m+1)L for m=1, . . . , M−1, and outputs the output image Outm (1≤m≤M−1). The procedure for synthesizing the output image Outm may be the same as that of the output image synthesis unit 32, and the generation process is repeated for a set of M−1 images for synthesis I(m)R and I(m+1)L.


In other words, the output image generation unit 50 generates an output image Outm (1≤m≤M−1) from the viewpoint image I(m) and the disparity map Dmap(m) (1≤m≤M), and the output image Outm and the output image Outm+1 (1≤m≤M−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).


The image generation apparatus 5 may further include a projection unit (not illustrated) that projects the output images Outm (1≤m≤M−1) output in S50. The projection unit arranges the output images Outm (1≤m≤M−1) in the sequence of Out1, Out2, . . . , and OutM−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.


According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Sixth Embodiment

Hereinafter, an image generation apparatus 6 will be described with reference to FIG. 19 and FIG. 20. FIG. 19 is a block diagram illustrating a configuration of the image generation apparatus 6. FIG. 20 is a flowchart illustrating the operation of the image generation apparatus 6. As illustrated in FIG. 19, the image generation apparatus 6 includes an output image generation unit 60 and a recording unit 90. The output image generation unit 60 includes an image-for-synthesis generation unit 61 and an output image synthesis unit 52. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 6.


The image generation apparatus 6 receives the viewpoint image I(m) (1≤m≤M+1, where M is an integer of 3 or more), generates and outputs M−1 output images Outm (1≤m≤M−1). Here, the viewpoint images I(1), I(2), . . . , I(M) are a sequence of viewpoint images arranged in the sequence of the viewpoint positions. The output image Outm is an image projected on a screen using a projector (projection apparatus). The interval between the adjacent viewpoint image I(m) and viewpoint image I(m+1) is the same as the interval between the adjacent output image Outm and the output image Outm+1, that is, the placement interval between the projectors that projects the output image Outm.


The operation of the image generation apparatus 6 will be described in accordance with FIG. 20.


In step S60, the output image generation unit 60 receives the viewpoint image I(m) (1≤m≤M+1) as an input, and generates and outputs an output image Outm (1≤m≤M−1).


In step S61, the image-for-synthesis generation unit 61 receives the viewpoint image I(m) (1≤m≤M+1) as an input, generates images for synthesis I(m)L, I(m)R, from the viewpoint image I(m) and the adjacent viewpoint images I(m+1), for m=1, . . . , M, and outputs the images for synthesis I(m)L, I(m)R (1≤m≤M). For example, the image-for-synthesis generation unit 61 may generate the images for synthesis I(m)L, I(m)R by the same procedure as the image generation unit 20. That is, the added images IL, IR are set to the images for synthesis I(m)L, I(m)R. In this case, the image-for-synthesis generation unit 61 is a component having the same configuration as the image generation unit 20 (see FIG. 8), and repeats a generation process on a set of M viewpoint images I(m) and I(m+1). As another method, for example, the image-for-synthesis generation unit 61 may generate the images for synthesis I(m)L, I(m)R by the same procedure as the image-for-synthesis generation unit 31. In this case, the image-for-synthesis generation unit 61 is a component having the same configuration as the image-for-synthesis generation unit 31 (see FIG. 13), and repeats a generation process on a set of M viewpoint images I(m) and I(m+1). In any case, in step S61, the image-for-synthesis generation unit 61 generates a disparity inducing edge D(m) whose the phase difference from the viewpoint image I(m) is a predetermined value (for example, π/2), from the viewpoint image I(m) and the neighboring viewpoint image I(m+1), for m=1, . . . , M, generates an image for synthesis I(m)L by adding the disparity inducing edge D(m) to the viewpoint image I(m), and generates an image for synthesis I(m)R by adding the polarity-inversed image of the disparity inducing edge D(m) to the viewpoint image I(m).


In step S52, the output image synthesis unit 52 receives the images for synthesis I(m)R and I(m)L (1≤m≤M) generated in S61 as an input, synthesizes the output image Outm, from the images for synthesis I(m)R and I(m+1)L for m=1, . . . , M−1, and outputs the output image Outm (1≤m≤M−1).


In other words, the output image generation unit 60 generates an output image Outm (1≤m≤M−1) from the viewpoint image I(m) (1≤m≤M+1), and the output image Outm and the output image Outm+1 (1≤m≤M−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).


The image generation apparatus 6 may further include a projection unit (not illustrated) that projects the output images Outm (1≤m≤M−1) output in S60. The projection unit arranges the output images Outm (1≤m≤M−1) in the sequence of Out1, Out2, . . . , and OutM−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.


According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Seventh Embodiment

Hereinafter, an image generation apparatus 7 will be described with reference to FIG. 21 and FIG. 22. FIG. 21 is a block diagram illustrating a configuration of the image generation apparatus 7. FIG. 22 is a flowchart illustrating the operation of the image generation apparatus 7. As illustrated in FIG. 21, the image generation apparatus 7 includes an output image generation unit 70 and a recording unit 90. The output image generation unit 70 includes an image-for-synthesis generation unit 71 and an output image synthesis unit 52. The recording unit 90 is a component configured to appropriately record information required for processing of the image generation apparatus 7.


The image generation apparatus 7 receives the viewpoint image I(j) (1≤j≤2J+1, where J is an integer of 3 or more), generates J−1 output images Outj (1≤j≤J−1), and outputs the generated images. Here, it is a sequence of viewpoint images in which the viewpoint images I(1), I(2), . . . , I(2J+1) are arranged in the sequence of the viewpoint positions. The output image Outj is an image projected on a screen using a projector (projection apparatus). The interval between two adjacent viewpoint images is the same as the interval between two adjacent output images, that is, ½ of the placement interval between projectors that project the output image Outj. FIG. 23 is a diagram illustrating a positional relationship between the viewpoint images I(1) to I(11) and the output images Out1 to Out4 when J=5. That is, the projectors are placed only at the viewpoint positions 3, 5, 7, and 9. Note that D is a placement interval of the projector.


The operation of the image generation apparatus 7 will be described in accordance with FIG. 22.


In step S70, the output image generation unit 70 receives the viewpoint image I(j) (1≤j≤2J+1) as an input, and generates and outputs an output image Outj (1≤j≤J−1).


In step S71, the image-for-synthesis generation unit 71 receives the viewpoint image I(j) (1≤j≤2J+1) as an input, generates images for synthesis I(2j)L and I(2j)R, from the viewpoint images viewpoint image I(2j−1), I(2j), I(2j+1), for j=1, . . . , J, and outputs the images for synthesis I(2j)L and I(2j)R (1≤j≤J). As illustrated in FIG. 24, the configuration of the image-for-synthesis generation unit 71 is the same as the configuration of the image generation unit 20. The difference is that the image generation unit 20 has two input images, whereas the image-for-synthesis generation unit 71 has three input images. Accordingly, the procedure for generating the images for synthesis I(2j)L and I(2j)R is the same as the procedure for generating the added image by the image generation unit 20. In other words, in step S71, the image-for-synthesis generation unit 71 generates a disparity inducing edge D(2j) whose the phase difference from the viewpoint image I(2j) is a predetermined value, from adjacent viewpoint images I(2j−1), I(2j), and I(2j+1), for j=1, . . . , J, generates an image for synthesis I(2j)L by adding the disparity inducing edge D(2j) to the viewpoint image I(2j), and generates an image for synthesis I(2j)R by adding the polarity-inversed image of the disparity inducing edge D(m) to the viewpoint image I(2j).


In step S52, the output image synthesis unit 52 receives the images for synthesis I(2j)R and I(2j)L (1≤j≤J) generated in S71 as an input, synthesizes the output image Outj, from the images for synthesis I(2j)R and I(2j+2)L, for j=1, . . . , J−1, and outputs the output image Outj (1≤j≤J−1).


In other words, the output image generation unit 70 generates an output image Outj (1≤j≤J−1) from the viewpoint image I(j) (1≤j≤2J+1), and the output image Outj and the output image Outj+1 (1≤j≤J−2) includes a phase modulation component that is canceled out when synthesized and visually recognized (simultaneously viewed).


The image generation apparatus 7 may further include a projection unit (not illustrated) that projects the output images Outj (1≤j≤J−1) output in S70. The projection unit arranges the output images Outj (1≤j≤J−1) in the sequence of Out1, Out2, . . . , and OutJ−1, and in other words, in the sequence of the viewpoint positions, and projects the output images on the screen.



FIG. 25 illustrates a result of an experiment. The vertical axis represents the value of the image quality evaluation index structural similarity (SSIM). The horizontal axis represents the viewpoint position of the viewpoint image used in FIG. 23. In the linear blending in the related art, at the viewpoint positions 3, 5, 7, and 9, the viewpoint image is directly used as the output image, so that the SSIM at the viewpoint positions 3, 5, 7, and 9 increases, while it can be seen that the SSIM has significantly decreased in the viewpoint positions 4, 6, and 8. On the other hand, in the method of the present disclosure, the SSIM at the viewpoint positions 4, 6, and 8 where no projector is placed is higher than the viewpoint positions 3, 5, 7, and 9 where the projectors are placed. As a result, the fluctuation in image quality associated with the viewpoint movement is reduced, and the viewer is less likely to feel unpleasant feelings.


According to the disclosure of the present embodiment, it is possible to suppress unpleasant feelings associated with fluctuation in image quality caused by the viewer's viewpoint movement.


Others


Some of the above embodiments use the disparity map as in the first embodiment, for example. In the embodiments using the disparity map, a depth map may be used instead of the disparity map. For example, when the depth map is input, the processes described in each embodiment may be performed after converting the depth map into the disparity map.


Appendix

Examples of the apparatuses in the present disclosure include, as single hardware entities, an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, a communication device (for example, a communication cable) that can communicate outside the hardware entity, a Central Processing Unit (CPU) (a cache memory or a register may be provided), a RAM and a ROM as memories, an external storage device as a hard disk, and a bus connected to the input unit, the output unit, the communication unit, the CPU, the RAM, the ROM, and the external storage device such that data can be exchanged. If necessary, the hardware entity may be provided with a device (drive) that can read and write data from and to a recording medium such as a CD-ROM. A physical entity having such hardware resources includes a general-purpose computer.


The external storage device of the hardware entity include a program necessary for realizing the above-described functions, data necessary for processing of the program, and or the like (without being limited to the external storage device, for example, the program may be read and stored in a ROM that is a dedicated storage device). Data obtained by the processing of these programs is appropriately stored in a RAM, an external storage device, or the like.


In the hardware entities, each program stored in an external storage device (or ROM or the like) and data necessary for processing of each program are read into a memory as needed, and interpreted, executed and processed by the CPU as appropriate. As a result, the CPU realizes a predetermined function (each component represented as . . . unit, . . . means, or the like described above).


The present disclosure is not limited to the embodiments described above, and may be changed as appropriate without departing from the scope of the disclosure. Further, the processing described in the above embodiment may be performed not only in chronological order according to the order of description, but also in parallel or individually according to the processing capability of the apparatus that executes the processing or as necessary.


As described above, when the processing function of the hardware entity (the apparatus in the present disclosure) described in the above embodiment is implemented by a computer, the processing content of the function that the hardware entity should have is described by a program. By executing this program on a computer, the processing functions of the hardware entity are realized on the computer.


A program describing this processing content can be recorded on a computer-readable recording medium. The computer-readable recording medium may be any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Specifically, for example, a hard disk device, a flexible disk, a magnetic tape, or the like can be used as a magnetic recording device, a Digital Versatile Disc (DVD), a DVD-Random Access Memory (RAM), a Compact Disc Read Only Memory (CD-ROM), a Recordable (CD-R)/ReWritable (RW), or the like can be used as an optical disk, a Magneto-Optical disc (MO), or the like is used as a magneto-optical recording medium, and an Electronically Erasable and Programmable-Read Only Memory (EEP-ROM), or the like can be used as a semiconductor memory.


Further, the distribution of the program is performed, for example, by selling, transferring, lending, or the like, a portable recording medium such as a DVD or a CD-ROM on which the program is recorded. Further, the program may be stored in a storage device of a server computer, and the program may be distributed by transferring the program from the server computer to another computer via a network.


A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. Then, when executing the process, the computer reads the program stored in its own recording medium and executes the process according to the read program. Further, as another execution form of the program, the computer may directly read the program from the portable recording medium and execute processing according to the program, and further, may sequentially execute the processing according to the received program, each time the program is transferred Further, the above-described processing may be executed by a so-called an Application Service Provider (ASP) type service that realizes a processing function only by executing an instruction and acquiring a result without transferring a program from the server computer to the computer. It should be noted that the program in the present disclosure includes information which is provided for processing by an electronic computer and is similar to the program (data that is not a direct command to the computer but has characteristics that define the processing of the computer).


Further, in this embodiment, a hardware entity is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

Claims
  • 1. A computer-implemented method for generating an image at a viewpoint, the method comprising: receiving a first viewpoint image and a plurality of disparity maps, wherein the disparity map corresponds to the first viewpoint image;generating, based on the first viewpoint image and the disparity map, a plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has a distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence;generating, based on the plurality of disparity inducing edges and the first viewpoint image, a first set of pseudo viewpoint images;generating, based on polarity-inverted images associated with the plurality of disparity inducing edges and the first viewpoint image, a second set of pseudo viewpoint images;generating, based at least on the first set of pseudo viewpoint images and the second set of pseudo viewpoint images, a set of output images, wherein the set of output images are in the predetermined sequence, and wherein the set of output images include a plurality of phase modulations for canceling out upon a synthesis and visual confirmation; andproviding the set of output images.
  • 2. The computer-implemented method of claim 1, the method comprising: receiving a second viewpoint image, wherein a first viewpoint of the first viewpoint image is adjacent to a second viewpoint of the second viewpoint image; andgenerating, based on the first viewpoint image, the second viewpoint image, and the disparity map, the plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has the distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence.
  • 3. The computer-implemented method of claim 1, the method further comprising: receiving a set of viewpoint images and a set of disparity maps, wherein the set of viewpoint images are in a sequence of a set of viewpoint positions, and wherein each of the set of disparity maps corresponds to one of the set of viewpoint images; andgenerating, based at least on the set of viewpoint images and the set of disparity maps, the set of output images.
  • 4. The computer-implemented method of claim 1, the method further comprising: generating, based on a set of viewpoint images, the set of output images, wherein the set of viewpoint images relates to in a predetermined sequence of viewpoint positions.
  • 5. The computer-implemented method of claim 3, wherein a first number of output images in the set of output images is based on a second number of the viewpoint images in the set of viewpoint images, wherein the first number is less than a half of the second number, and wherein the second number corresponds to a third number of viewpoint positions.
  • 6. The computer-implemented method of claim 1, the method comprising: aggregating luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andproviding the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.
  • 7. The computer-implemented method of claim 3, the method comprising: aggregating luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andproviding the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.
  • 8. A system for generating an image at a viewpoint, the system comprises: a processor; anda memory storing computer-executable instructions that when executed by the processor cause the system to: receive a first viewpoint image and a plurality of disparity maps, wherein the disparity map corresponds to the first viewpoint image;generate, based on the first viewpoint image and the disparity map, a plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has a distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence;generate, based on the plurality of disparity inducing edges and the first viewpoint image, a first set of pseudo viewpoint images;generate, based on polarity-inverted images associated with the plurality of disparity inducing edges and the first viewpoint image, a second set of pseudo viewpoint images;generate, based at least on the first set of pseudo viewpoint images and the second set of pseudo viewpoint images, a set of output images, wherein the set of output images are in the predetermined sequence, and wherein the set of output images include a plurality of phase modulations for canceling out upon a synthesis and visual confirmation; andprovide the set of output images.
  • 9. The system of claim 8, the computer-executable instructions when executed further causing the system to: receive a second viewpoint image, wherein a first viewpoint of the first viewpoint image is adjacent to a second viewpoint of the second viewpoint image; andgenerate, based on the first viewpoint image, the second viewpoint image, and the disparity map, the plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has the distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence.
  • 10. The system of claim 8, the computer-executable instructions when executed further causing the system to: receive a set of viewpoint images and a set of disparity maps, wherein the set of viewpoint images are in a sequence of a set of viewpoint positions, and wherein each of the set of disparity maps corresponds to one of the set of viewpoint images; andgenerate, based at least on the set of viewpoint images and the set of disparity maps, the set of output images.
  • 11. The system of claim 8, the computer-executable instructions when executed further causing the system to: generate, based on a set of viewpoint images, the set of output images, wherein the set of viewpoint images relates to in a predetermined sequence of viewpoint positions.
  • 12. The system of claim 10, wherein a first number of output images in the set of output images is based on a second number of the viewpoint images in the set of viewpoint images, wherein the first number is less than a half of the second number, and wherein the second number corresponds to a third number of viewpoint positions.
  • 13. The system of claim 8, the computer-executable instructions when executed further causing the system to: aggregate luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andprovide the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.
  • 14. The system of claim 10, the computer-executable instructions when executed further causing the system to: aggregate luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andprovide the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.
  • 15. A computer-readable non-transitory recording medium storing computer-executable instructions that when executed by a processor cause a computer system to: receive a first viewpoint image and a plurality of disparity maps, wherein the disparity map corresponds to the first viewpoint image;generate, based on the first viewpoint image and the disparity map, a plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has a distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence;generate, based on the plurality of disparity inducing edges and the first viewpoint image, a first set of pseudo viewpoint images;generate, based on polarity-inverted images associated with the plurality of disparity inducing edges and the first viewpoint image, a second set of pseudo viewpoint images;generate, based at least on the first set of pseudo viewpoint images and the second set of pseudo viewpoint images, a set of output images, wherein the set of output images are in the predetermined sequence, and wherein the set of output images include a plurality of phase modulations for canceling out upon a synthesis and visual confirmation; andprovide the set of output images.
  • 16. The computer-readable non-transitory recording medium of claim 15, the computer-executable instructions when executed further causing the system to: receive a second viewpoint image, wherein a first viewpoint of the first viewpoint image is adjacent to a second viewpoint of the second viewpoint image; andgenerate, based on the first viewpoint image, the second viewpoint image, and the disparity map, the plurality of disparity inducing edges, wherein each of the plurality of disparity inducing edges has the distinct predetermined phase difference based on the first viewpoint image and a predetermined sequence.
  • 17. The computer-readable non-transitory recording medium of claim 15, the computer-executable instructions when executed further causing the system to: receive a set of viewpoint images and a set of disparity maps, wherein the set of viewpoint images are in a sequence of a set of viewpoint positions, and wherein each of the set of disparity maps corresponds to one of the set of viewpoint images; andgenerate, based at least on the set of viewpoint images and the set of disparity maps, the set of output images.
  • 18. The computer-readable non-transitory recording medium of claim 15, the computer-executable instructions when executed further causing the system to: generate, based on a set of viewpoint images, the set of output images, wherein the set of viewpoint images relates to in a predetermined sequence of viewpoint positions.
  • 19. The computer-readable non-transitory recording medium of claim 17, wherein a first number of output images in the set of output images is based on a second number of the viewpoint images in the set of viewpoint images, wherein the first number is less than a half of the second number, and wherein the second number corresponds to a third number of viewpoint positions, the computer-executable instructions when executed further causing the system to:aggregating luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andproviding the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.
  • 20. The computer-readable non-transitory recording medium of claim 15, the computer-executable instructions when executed further causing the system to: aggregate luminance of at least two adjacent images of a combination of the first set of pseudo viewpoint image and the received viewpoint image, and wherein the at least two adjacent images corresponds to at least two adjacent viewpoints; andprovide the aggregated luminance of the at least two adjacent images as an image at a view located at an intermediate position of the at least two adjacent viewpoints.
Priority Claims (1)
Number Date Country Kind
JP2018-078327 Apr 2018 JP national
PCT Information
Filing Document Filing Date Country Kind
PCT/JP2019/014774 4/3/2019 WO 00
Publishing Document Publishing Date Country Kind
WO2019/202986 10/24/2019 WO A
US Referenced Citations (2)
Number Name Date Kind
8106949 Tan Jan 2012 B2
20130300828 Yamato Nov 2013 A1
Foreign Referenced Citations (1)
Number Date Country
201856983 Apr 2018 JP
Non-Patent Literature Citations (2)
Entry
Jones, Andrew, et al., “An Automultiscopic Projector Array for Interactive Digital Humans,” SIGGRAPH 2015 Emerging Technologies, Aug. 9-13, 2015, ACM 978-1-4503-3635-Feb. 15, 2008.
Takada, Hideaki, et al., “Smooth Motion Parallax with Fewer Projectors Realizing Viewpoint Movable Unaided 3D Video Screen,” Jul. 28, 2017, Video Information Media Society Technical Report, ITE Technical Report, vol. 41, No. 24, IDY2017-34.
Related Publications (1)
Number Date Country
20210112237 A1 Apr 2021 US