The present application claims priority to Korean Provisional Applications No. 10-2020-0040893, filed Apr. 3, 2020, and No. 10-2021-0036548, filed Mar. 22, 2021, the entire contents of which are incorporated herein for all purposes by this reference.
The present disclosure relates to an apparatus, method and system for generating a three-dimensional image using a coded phase mask.
A camera collects light reflected by an object and generates an image expressing the shape and color of the object from the collected light through a recording medium. Generally, a camera collects light reflected by an object by using a lens. Herein, various aberrations occur due to the curvature of a spherical lens. Such optical aberration inhibits a clear image from being generated and thus may be a main factor degrading the performance of a camera. Accordingly, as an additional technology for removing the optical aberration of a leans is needed, an optical system using a lens is configured somewhat complexly.
To solve various optical problems of such a lens-based camera, various technologies of generating images based on a pinhole camera structure have been proposed.
A coded aperture is an aperture that has a plurality of pinholes existing at arbitrary positions. Light reflected by an object is collected at each pinhole through a coded aperture and is recorded in a recording medium. When a decoding process is performed based on a plurality of recorded images (patterns), a single image of the object is generated.
The present disclosure provides an apparatus, method and system for generating three-dimensional image using a coded phase mask.
According to the present disclosure, an apparatus for generating a three-dimensional image, the apparatus may comprise a communicator for transmitting and receiving a signal and a processor for controlling the communicator, wherein the processor synthesizes complex data for an object based on an image, which is obtained by shooting an object, and generates a three-dimensional image of the object based on the complex data for the object and a point spread function (PSF) image for a point light source.
According to the present disclosure, a method for generating a three-dimensional image, the method may comprise synthesizing complex data for an object based on an image that is obtained by shooting an object and generating a three-dimensional image of the object based on the complex data for the object and a PSF image for a point light source.
According to the present disclosure, A three-dimensional image generation system, the system may comprise a coded phase mask (CPM) for modulating a phase of reflected light of an object, a polarization image sensor that records brightness information of the object as an image based on the reflected light of the object, of which the phase is modulated from the coded phase mask and a three-dimensional image generator that synthesizes complex data for the object based on an image of the object and generates a three-dimensional image of the object based on the complex data for the object and a PSF image for a point light source.
According to the present disclosure, it is possible to effectively generate a precise three-dimensional image.
In addition, it is possible to generate a three-dimensional image by using a coded phase mask at a reduced cost.
Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, which will be easily implemented by those skilled in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.
In the following description of the embodiments of the present disclosure, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present disclosure rather unclear. In addition, parts not related to the description of the present disclosure in the drawings are omitted, and like parts are denoted by similar reference numerals.
In the present disclosure, components that are distinguished from each other are intended to clearly illustrate each feature. However, it does not necessarily mean that the components are separate. That is, a plurality of components may be integrated into one hardware or software unit, or a single component may be distributed into a plurality of hardware or software units. Thus, unless otherwise noted, such integrated or distributed embodiments are also included within the scope of the present disclosure.
In the present disclosure, components described in the various embodiments are not necessarily essential components, and some may be optional components. Accordingly, embodiments consisting of a subset of the components described in one embodiment are also included within the scope of the present invention. Also, embodiments that include other components in addition to the components described in the various embodiments are also included in the scope of the present disclosure.
Meanwhile, in the present disclosure, a three-dimensional image may include a complex hologram.
Meanwhile, in the present disclosure, the terms “image generation”, “imaging”, and “image reconstruction” may be used interchangeably.
Meanwhile, in the present disclosure, the terms “point light source” and “pinhole” may be used interchangeably.
Meanwhile, in the present disclosure, the terms “complex hologram”, “complex hologram data”, and “complex data” may be used interchangeably.
Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings.
More specifically,
Unlike a coded aperture camera using a coded aperture of the conventional computational photography that demands a high-load iterative algorithm for reconstructing a three-dimensional image, the system of
When external light is collected through the object lens 201, light is input into the spatial light modulator 203 through the polarizer 202 that filters a light component of a particular polarity.
Next, an image including brightness information on a point light source may be obtained from the image sensor 204 by displaying a coded phase mask (CPM), which shows various phase angles from 0 to 360 degrees, on the spatial light modulator 203. Herein, the coded phase mask may be as shown in (a) of
When a PSF library is constructed, an object may be put in a space to which a point light source has been moved, and an image recording brightness information of the object for a corresponding point like (c) of
Here, Ik(
A sheet of complex data for an object may generate a three-dimensional image of the object by being convolution-operated with a PSF image (e.g., (b) of
Meanwhile, the image sensor may be a monochromatic polarization image sensor of
In one embodiment, the microlens array 401 may be attached on the polarizer array 402. For example, the polarizer array 402 may have 2×2 structure about a pixel and each polarizer is rotated by 0, 45, −45, 90 in degree for adjusting the geometric phase of the light wave as mentioned above. Meanwhile, a degree to which each polarizer rotates corresponds to one embodiment, to which the present disclosure is not limited.
Meanwhile, an image sensor of
In one embodiment, in the case of a color polarization image sensor, four polarization components may be expressed by three color channels (e.g., R, G, B) in one shot for an object (e.g., R, G, G, B). Herein, each color channel may be composed of, for example, four pixels, and each pixel may be based on wire-grid directions different from each other. In one embodiment, according to this, a total of four polarization components may be expressed in RGGB.
In one embodiment, the color polarization image sensor of
Brightness information image for an object, which is recorded by a color polarization image sensor, may be a raw image. In one embodiment, a color polarization image sensor may express four polarization components by three colors (e.g., R, G, B). Herein, a polarizer may have phase values that are different from each other. For example, it may have phase values of 0 degree, 45 degrees, 90 degrees, and 135 degrees.
A color polarization image sensor may perform complex hologram recombination by distinguishing polarization components according to phases. Based on this, each component may be collected in each color (demosicing).
More specifically, it is a view showing a random phase retarder that may be generated, when a coded phase mask like (a) of
In one embodiment, a three-dimensional image generation system using a coded phase mask may include an object to be shot 901, an object lens 902, a phase mask 903, a polarization image sensor 904, and a three-dimensional image generator 905.
In one embodiment, when describing the data processing process of
Also, the data processing process of
In one embodiment, a three-dimensional image generation system may perform the data processing process of
Before generating a three-dimensional image, a three-dimensional generator may perform a process 1001 of obtaining complex data for a point light source, for the first time. First, when a three-dimensional image generation system is configured (1000), a point light source (pinhole) is installed (1010) and is shot (1011), and complex data may be synthesized (1012) by separating a polarization image based on a captured image for the point light source. Separating a polarization image may include a process in which four sheets of images including modulated brightness information are generated based on an image for a point light source, which is obtained by a single shot, through the phase mask 903 of
After the PSF library is built up, an object to be shot is installed (1003) and is shot (1004). When the reflected light of an object is incident on the object lens 902, light information may be collected and may pass through the phase mask (903). At this time, the phase mask 903, which modulates a phase of the reflected light of the object, may be a coded phase mask (CPM) and may include a coded half-wave phase mask. Herein, the coded half-wave phase mask includes what is illustrated in
Next, the polarization image sensor 904 may record brightness information of the object as an image on the basis of the reflected light of the object, of which the phase is modulated by a coded phase mask. Based on an image that is generated when an object is shot (1004), each polarization image may be separated. Based on this, the three-dimensional image generator 905 may synthesize (1005) complex data for the object. More specifically, the polarization image sensor 904 may transform phase information of input light at a position corresponding to each pixel into intrinsic brightness information based on an allocation angle of a polarizer corresponding to the pixel and may record the brightness information in a pixel. Herein, the polarization image sensor 904 may include the monochromatic polarization image sensor of
Herein, when an object is shot at a specific depth zk, the three-dimensional image generator 905 may reconstruct (1007) a three-dimensional image of the object based on a PSF image for a point light source shot at a specific depth (e.g., zk), which is included in a PSF library, and synthesized complex data of the object. Herein, a PSF image for a point light source and synthesized complex data of an object may be convolutionally operated. In other words, the convolution of the PSF image for a point light source and the synthesized complex data of the object may be possible.
Meanwhile, the lens 902, the phase mask 903, the polarization image sensor 904, and the three-dimensional image generator 905 were distinguished above for clarity of explanation. The phase mask 903 and the polarization image sensor 904 may be configured as one apparatus, or the phase mask 903 and/or the polarization image sensor 904 may be included in the three-dimensional image generator 905. However, as this is merely one embodiment, the present disclosure is not limited thereto.
In one embodiment, the method for generating a three-dimensional image in
In another embodiment, the method for generating a three-dimensional image in
In one embodiment, based on an image that is obtained by shooting an object, complex data for the object may be synthesized (S1101). Herein, an image obtained by shooting an object may include, as described above with reference to another drawing, an image including phase modulation brightness information that is modulated through a phase mask after the object is shot. Herein, a phase mask includes a coded phase mask, and a coded phase mask may include a coded half-wave phase mask. Herein, the coded half-wave phase mask may be based on a geometric phase element that is fabricated by a scanning method. In one embodiment, there may be a plurality of images that are obtained by shooting an object. They may be obtained by a single shot using a polarization image sensor and may include phase-modulated brightness information for an object. Also, a polarization image sensor that is used may be the monochromatic polarization image sensor of
Based on synthesized complex data for an object and a PSF image for a point light source, a three-dimensional image of an object may be generated (S1102). A PSF image for a point light source may be a PSF image that is extracted from a PSF library for a point light source and corresponds to a position of an image obtained by shooting the object. A process of building up a PSF library may be the same as the process that was described with reference to
Meanwhile, as
In one embodiment, a three-dimensional image generator 1200 using a coded phase mask may include a communicator 1202 for transmitting and receiving a signal and a processor 1201 for controlling the communicator 1202.
In one embodiment, the three-dimensional image generator of
In one embodiment, the processor 1201 may synthesize complex data for an object based on an image, which is obtained by shooting an object, and may generate a three-dimensional image of the object based on the complex data for the object and a point spread function (PSF) image for a point light source. An image obtained by shooting an object may be obtained through a coded phase mask (CPM) by a polarization image sensor. A coded phase mask may be a coded half-wave phase mask, and a coded half-wave phase mask may be based on a geometric phase element that is fabricated by a scanning method. This may include what is described with reference to
Meanwhile, although not illustrated in
Also, there may be one or more processors 1201 of
In the existing system, a phase-only spatial light modulator is used to represent a coded phase mask, however, it is an expensive active element that requires a driving circuit and additional power and has a limit to enlarge the diameter of an aperture. However, according to the present disclosure, the system for generating a three-dimensional image is effectively configured by using a coded half-wave phase mask based on a geometric phase element.
In addition, according to the present disclosure, it is possible to obtain four images comprising phase-modulated brightness information by a single shot of an object instead of obtaining images by using coded phase masks of the polarization image sensor in time sequence. Therefore, efficient image obtaining may be possible.
Moreover, according to the present disclosure, a highly compact three-dimensional image generation system having no chromatic dispersion effect due to wavelength dependency of aberration and diffraction caused by a lens may be manufactured as a mass-producible device without additional power.
Accordingly, steps of a method or an algorithm described in relation to embodiments of the present disclosure may be directly implemented by hardware, which is executed by a processor, a software module, or a combination of these two. A software module may reside in a storage medium (that is, a memory and/or a storage) like RAM, flash memory, ROM, EPROM, EEPROM, register, hard disk, removable disk, and CD-ROM. An exemplary storage medium is coupled with a processor, and the processor may read information from a storage medium and may write information into a storage medium. In another method, a storage medium may be integrated with a processor. A processor and a storage medium may reside in an application-specific integrated circuit (ASIC). An ASIC may reside in a user terminal. In another method, a processor and a storage medium may reside in a user terminal as individual components.
In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor, or the like. For example, it is obvious that it can be implemented in the form of a program stored on a non-transitory computer readable medium that can be used at the end or edge, or a program stored on a non-transitory computer readable medium that can be used at the edge or the cloud. Do. In addition, it can be implemented by a combination of various hardware and software.
Although the exemplary methods of the present disclosure are represented by a series of acts for clarity of explanation, they are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement a method according to the present disclosure, the illustrative steps may include an additional step or exclude some steps while including the remaining steps. Alternatively, some steps may be excluded while additional steps are included.
The scope of the present disclosure is software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that allow an operation according to a method of various embodiments to be executed on a device or a computer, and such software or It includes a non-transitory computer-readable medium which stores instructions and the like and is executable on a device or a computer.
For example, a program for generating a three-dimensional image using a coded mask according to an embodiment of the present disclosure may be a program stored in a non-transitory computer-readable medium, which synthesizes complex data for an object based on an image, which is obtained by shooting an object, and generates a three-dimensional image based on the synthesized complex data and a PSF image for a point light source.
The present disclosure described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present disclosure for those of ordinary skill in the technical field to which the present disclosure belongs, so the scope of the present disclosure is described above. It is not limited by one embodiment and the accompanying drawings.
Number | Date | Country | Kind |
---|---|---|---|
10-2020-0040893 | Apr 2020 | KR | national |
10-2021-0036548 | Mar 2021 | KR | national |