The present disclosure relates to light field microscope-based image acquisition methods and apparatuses capable of acquiring changes in a sample as a 3D image having a high spatial resolution.
Since a conventional fluorescence microscope captures a 2D plane image of a sample, in order to check the distribution of fluorescence in a 3D space, a method of capturing several slice images in a z-axis direction and synthesizing the slice images to reconstruct them into a 3D image is used.
That is to say, in conventional art, a 3D image is acquired by a way of combining images captured at respective depths by precisely adjusting a z-axis focal length.
However, since the above way captures images several times while changing a z-axis position, a distortion due to a change in fluorescence of the sample may occur during the time of image capturing. In the case of a cell cluster (muscle cells) in which movement occurs, it is difficult to check 3D information on the movement, and thus, it is very difficult to acquire an entire image in the same time frame.
A light field microscope has been devised to solve the above problems.
The light field microscope has an optical structure in which a lens array is added to the conventional fluorescence microscope structure, and it is possible to convert an image acquired from the corresponding optical structure into a 3D image through a reconstruction method.
However, although the conventional light field microscope has the advantage of being able to acquire 3D image information with just one capture, a disadvantage exists in that a spatial resolution is lower than that of conventional microscopes because the spatial resolution depends on the number of lenses in the lens array.
In other words, in the case of the light field microscope, there is a problem in that resolution is greatly reduced due to the gap between a plurality of lenses constituting the lens array.
Therefore, although a way of reducing the size of the lenses constituting the lens array has been proposed, there is a problem in that the pixel area of an image sensor should also be reduced correspondingly. For reference, because the pixel area of an image sensor affects the sensitivity of an image to be acquired, in order to be applied to a fluorescence imaging application that requires high sensitivity, it is important to keep the size of pixels large.
Therefore, the present applicant proposes the present disclosure in order to solve the above problems, and prior art documents related thereto include Korean Patent No. 10-2253320 entitled “Method for Displaying 3 Dimension Image in Integral Imaging Microscope System, and Integral Imaging Microscope System Implementing the Same”.
The present disclosure is to solve the above problems, and an objective of the present disclosure is to provide a light field microscope-based image acquisition method and apparatus configured to be able to acquire an image having a high spatial resolution even without increasing the number of microlenses constituting a lens array by reducing the size of the microlenses.
In an embodiment, a light field microscope-based image acquisition apparatus may include: a lenslet synthesis processing unit configured to acquire a virtual lenslet image using a multi-lenslet image group; a sub-aperture image conversion unit configured to convert the virtual lenslet image generated in the lenslet synthesis processing unit into a sub-aperture image; and a 3D conversion unit configured to convert the sub-aperture image generated in the sub-aperture image conversion unit into a focal-stack image.
The multi-lenslet image group may be generated in such a way to acquire a plurality of image frame information through microlenses whose arrangement position is changed in each time frame within a set imaging sensor area.
The multi-lenslet image group may be acquired by rotating, for a set time, a disk on which a plurality of microlenses are arranged.
The lenslet synthesis processing unit may superimpose a plurality of image frames configuring the multi-lenslet image group to determine a disposition relationship of microlenses for each image.
The sub-aperture image conversion unit may find a pixel corresponding to the coordinates of a microlens in the virtual lenslet image, and may map the pixel to a coordinate position of the sub-aperture image to be converted.
The sub-aperture image conversion unit may generate a pixel value of the virtual lenslet image using
L
v(i,j,u,v)=ΣkwyLk(i,j,u,v) Equation 1:
where
Lv(i,j,u,v) is a pixel value corresponding to i, j, u and v coordinates, wk is a weight value according to a distance between a k-th lenslet image and the virtual lenslet image, and dk is a distance between the k-th lenslet image and the virtual lenslet image.
The 3D conversion unit may convert the sub-aperture image into the focal-stack image having 3D information of x, y and z axes, and an x-axis and y-axis position of the focal-stack image may include 2D image information at a corresponding z-axis coordinate point.
The 3D conversion unit may convert the sub-aperture image into the focal-stack image using
where Aα(s,t,u,v) is a converted sub-aperture value corresponding to s, t, u and v coordinates and a specific value of α, Fα(x,y) is a focal-stack coordinate value corresponding to the value of α, x, y are x and y coordinate values corresponding to a 3D focal-stack image, and N is the number of u and v coordinates.
The disk may be connected to an actuator or motor which provides rotational power or linear power.
A linear movement range of the disk may be 0 to 500 mm.
In an embodiment, a light field microscope-based image acquisition method may include: a first step of acquiring a multi-lenslet image group; a second step of acquiring a virtual lenslet image using the multi-lenslet image group acquired in the first step; a third step of converting the virtual lenslet image acquired in the second step into a sub-aperture image; and a fourth step of converting the sub-aperture image converted in the third step into a focal-stack image.
The light field microscope-based image acquisition method and apparatus according to the present disclosure provide a configuration capable of acquiring a multi-lenslet image group by way of rotating and linearly moving a disk on which microlenses are arranged and generating a 3D image using the multi-lenslet image group. Therefore, it is possible to acquire a 3D image having a higher spatial resolution than a conventional light field microscope.
In addition, the light field microscope-based image acquisition method and apparatus according to the present disclosure enable faster and more accurate acquisition of 3D image information of a measurement sample (muscle cells, etc.) in which movement occurs.
Also, the light field microscope-based image acquisition method and apparatus according to the present disclosure may acquire an image of a high spatial resolution by preventing a spatial resolution deterioration phenomenon due to the spacing of microlenses by changing the arrangement of microlenses by rotation of a disk.
Further, the light field microscope-based image acquisition method and apparatus according to the present disclosure may acquire an image of a high spatial resolution without increasing the number of arrangements in a disk or reducing the size of an image sensor by reducing the size of microlenses. Thus, the manufacturing cost of the apparatus may be reduced, and a cost required for image acquisition may be significantly reduced.
Advantages and features of the present disclosure and methods to achieve them will become apparent from the descriptions of embodiments herein below with reference to the accompanying drawings.
However, the present disclosure is not limited to the embodiments disclosed herein but may be implemented in various different ways. The embodiments are provided for making the present disclosure thorough and for fully conveying the scope of the present disclosure to those skilled in the art to which the present disclosure pertains. It is to be noted that the scope of the present disclosure is defined only by the claims.
Hereinafter, a light field microscope-based image acquisition method and apparatus according to embodiments of the present disclosure will be described in detail with reference to
As shown in
The light field microscope-based image acquisition apparatus 100 configured as described above may change the arrangement position of microlenses ML in each time frame in such a way to rotate or linearly move a disk D on which a plurality of microlenses ML are arranged, and may acquire the multi-lenslet image group 10 by acquiring a plurality of images according thereto.
For reference,
As shown in
Fluorescent light having passed through the dichroic mirror reaches a focal plane through a tube lens and an optical tube, and the disk D according to the embodiment of the present disclosure is disposed on the corresponding focal plane.
The plurality of microlenses ML which are defined on the disk D serve to align light converging on the focal plane in conformity with the position of each lens. Light having passed through the microlenses ML passes through a relay lens. The relay lens serves to adjust the difference in physical distance between the arrangement of the microlenses ML and an imaging sensor. Light having passed through the relay lens reaches the imaging sensor, and an image is acquired through the imaging sensor. It is possible to reconstruct the path information of light acquired through the corresponding image into a 3D image through a software algorithm.
A portion of the area of the disk D may be disposed between the optical tube and the relay lens. In addition, the central portion of the disk D may be connected to a drive unit M which is composed of a known actuator or motor for transferring rotational power or linear power.
Accordingly, the disk D is capable of linear reciprocating motion in the direction of a Z-axis shown in
Since the arrangement position of the microlenses ML is changed when the disk D is rotated, it is possible to acquire a different image at each time due to the changed arrangement position of the microlenses ML.
Accordingly, as shown in
In summary, a plurality of image frame information is acquired through the microlenses ML whose arrangement position is changed in each time frame within a set imaging sensor area, and the multi-lenslet image group 10 may be acquired on the basis of the information. A way of changing the arrangement position of the microlenses ML may be implemented by a structure in which the disk D with the plurality of microlenses ML arranged thereon is rotated and is linearly moved for a set time.
As shown in
As described above, by combining a plurality of image frame information acquired within a set time through the arrangement position of the microlenses ML that is changed in each time frame, it is possible to acquire the multi-lenslet image group 10.
For reference,
For example, as shown in
The lenslet synthesis processing unit 200 may superimpose the plurality of image frames constituting the multi-lenslet image group 10 to determine the disposition relationship of the microlenses ML for each image.
As shown in
The size of the virtual lenslet image to be outputted is linearly larger than the s and t resolution of the input image, and a magnification n is adjustable according to setting.
As shown in
The sub-aperture image conversion unit 300 may convert the virtual lenslet image into a sub-aperture image.
The sub-aperture image conversion unit 300 may find a pixel corresponding to the coordinates of a microlens ML in the virtual lenslet image and map the pixel to a coordinate position of the sub-aperture image to be converted.
As shown in
Since the position of each pixel in the light field image conforms to the s, t, u and v coordinates, a procedure of converting a lenslet image into an aligned image is required, and the sub-aperture image conversion unit 300 performs the procedure of converting the lenslet image into a sub-aperture image. The performing procedure is to find a pixel corresponding to an s, t, u, and v coordinate position in each lenslet image and map the pixel to a position of an s, t, v, and u image of the sub-aperture image. The converted sub-aperture image is composed of u×v number of block images having a size of s×t, and each block image has angular information corresponding to u and v coordinates.
As shown in
L
v(i,j,u,v)=ΣkwyLk(i,j,u,v) Equation 1:
Here,
and Σkwk=1. Lv(i,j,u,v) is a pixel value corresponding to i, j, u and v coordinates, wk is a weight value according to the distance between a k-th lenslet image and the virtual lenslet image, and dk is the distance between the k-th lenslet image and the virtual lenslet image.
As described above, the 3D conversion unit 400 may convert the sub-aperture image generated in the sub-aperture image conversion unit 300 into a focal-stack image.
As shown in
An x-axis and y-axis position of the focal-stack image includes 2D image information at a corresponding z-axis coordinate point, and the depth information of an image may be determined depending on the number of z-axis coordinates.
In a method in which the 3D conversion unit 400 converts the sub-aperture image into the focal-stack image, Equation 2 and Equation 3 below may be used.
Here, Aα(s,t,u,v) is a converted sub-aperture value corresponding to s, t, u and v coordinates and a specific value of α. The converted sub-aperture value is necessary to acquire the focal-stack image, and a focal-stack coordinate value F(x,y) corresponding to the value of a may be acquired. In addition, x, y are x and y coordinate values corresponding to the 3D focal-stack image. Furthermore, Nuv is the number of u and v coordinates, and a specific z coordinate conforms to the specific value of α. A case where α=1 corresponds to the central position of the focal-stack image, and α has a range of 0<α<1+i. By adjusting the range of α, it is possible to adjust the measurement range of the z-axis.
Hereinafter, with reference to
As shown in
a first step S100 of acquiring a multi-lenslet image group; a second step S200 of acquiring a virtual lenslet image using the multi-lenslet image group acquired in the first step S100; a third step S300 of converting the virtual lenslet image acquired in the second step S200 into a sub-aperture image; and a fourth step S400 of converting the sub-aperture image converted in the third step S300 into a focal-stack image.
In the first step S100, a plurality of image frame information may be acquired through the microlenses ML whose arrangement position is changed in each time frame within a set imaging sensor area, and the multi-lenslet image group may be acquired in such a way of rotating, for a set time, the disk D on which the plurality of microlenses ML are arranged.
In the second step S200, a plurality of image frames constituting the multi-lenslet image group may be superimposed to determine the disposition relationship of the microlenses ML for each image. The lenslet synthesis processing unit 200 may perform the second step S200.
In the third step S300, a pixel corresponding to the coordinates of the microlens ML in the virtual lenslet image may be found, and may be mapped to a coordinate position of the sub-aperture image to be converted. The sub-aperture image conversion unit 300 may perform the third step S300.
In the fourth step S400, the sub-aperture image is converted into the focal-stack image having 3D information of x, y and z axes, and an x-axis and y-axis position of the focal-stack image may include 2D image information at a corresponding z-axis coordinate point. The fourth step S400 may be performed by the 3D conversion unit 400.
The light field microscope-based image acquisition method according to the embodiment of the present disclosure generates a high-resolution 3D image on the basis of the configuration described above in the light field microscope-based image acquisition apparatus 100 according to the embodiment of the present disclosure. Therefore, the detailed description thereof will be omitted so as to avoid overlapped description.
While specific embodiments of the present disclosure have been described so far, various modifications are possible without departing from the scope of the present disclosure.
Therefore, the scope of the present disclosure should not be defined by being limited to the described embodiments, but should be defined by not only the appended claims but also equivalents thereto.
The present disclosure may be applied and sold in various industrial fields using safety equipment, component inspection, material inspection, measurement and analysis instruments.
Number | Date | Country | Kind |
---|---|---|---|
10-2022-0033343 | Mar 2022 | KR | national |
10-2022-0067667 | Jun 2022 | KR | national |
This is a continuation of International Application No. PCT/KR2022/008335 filed on Jun. 14, 2022, which claims priority from Korean Patent Application No. 10-2022-0033343 filed on Mar. 17, 2022, and Korean Patent Application No. 10-2022-0067667 filed on Jun. 2, 2022. The contents of these applications are incorporated herein by reference in their entireties.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/KR2022/008335 | Jun 2022 | US |
Child | 18467120 | US |