Patent Grant
11880965
This application claims priority under 35 U.S.C 119(a) to Korean Patent Application No. 10-2021-0021683, filed on Feb. 18, 2021, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.
The present disclosure relates to a smartphone and a method for obtaining image using a smartphone, more particularly to a smartphone for obtaining Fourier ptychography image and a method for obtaining Fourier ptychography image using a smartphone.
Research to obtain a large area of a biological sample with high resolution is continuously being made. As is well known, a high-resolution image of a sample is obtained using a microscope, and in a general microscope, magnification and resolution are in inverse proportion to each other. The resolution of the image for the object information is determined according to the spatial bandwidth proportional to the numerical aperture of the microscope objective lens. Although the bandwidth can be widened by using an objective lens of a high numerical aperture, the magnification is also increased, so the observation area that can be photographed at once is reduced.
Meanwhile, Fourier ptychography image restoration technology is a technology for improving resolution by composing an optical system with an objective lens of a low magnification and low numerical aperture that can capture large-area images, obtaining multiple images by changing the incident angle of the illumination of the sample, and applying a restoration algorithm to the obtained multiple images.
In recent years, the demand is increasing for a simple and highly portable image acquisition device that can be used for experiments and pathological diagnosis in a limited environment. Although the Fourier ptychography image restoration technology can simultaneously improve the observation area and resolution, it requires a separate illumination device to obtain multiple images, and a separate camera and processor for image processing are required for Fourier ptychography image restoration.
The present disclosure proposes a method capable of restoring a Fourier ptychography image at low cost without using a separate illumination device.
In addition, the present disclosure proposes a method for obtaining Fourier ptychography image using a smartphone capable of improving the quality of the Fourier ptychography image.
According to an aspect of the present disclosure, conceived to achieve the objectives above, there is provided a method for obtaining a Fourier ptychography image using a smartphone, in which a Fourier ptychography image of a sample is obtained by controlling the smartphone through an application installed on the smartphone, the method comprising the steps of: (a) sequentially providing illumination of different angles to the sample by sequentially displaying, according to a first pattern composed of point light sources at different positions, the point light sources of the first pattern on a display of the smartphone; (b) obtaining an image for each illumination angle of the sample using a camera of the smartphone whenever illumination of different angles is provided by the point light sources of the first pattern; and (c) restoring a first Fourier ptychography image using a plurality of images for each illumination angle obtained using the camera of the smartphone.
The point light sources of the first pattern include a first point light source displayed vertically aligned with the sample, and a plurality of point light sources are distributed in the first pattern around the first point light source.
The sizes of the plurality of point light sources distributed around the first point light source are set in proportion to a separation distance from the first point light source.
The first point light source has a circular shape, and shapes of a plurality of point light sources distributed around the first point light source are set based on a separation direction between the first point light source and the corresponding point light source.
Among the plurality of point light sources distributed around the first point light source, the point light source spaced apart from the first point light source in the horizontal direction has an elliptical shape having a major axis in the horizontal direction, and the point light source spaced apart from the first point light source in the vertical direction has an elliptical shape having a major axis in the vertical direction.
The method may further include the steps of: (d) repeating the steps (a) to (c) for a second pattern to an Nth pattern having a point light source position distribution different from the first pattern; (e) evaluating quality of the first to Nth Fourier ptychography images restored for each pattern; and (0 selecting a Fourier ptychography image of the best quality among the first to Nth Fourier ptychography images.
According to another aspect of the present disclosure, there is provided a smartphone for obtaining a Fourier ptychography image of a sample in combination with an optical system device, the smartphone comprising: a display control module configured to sequentially provide illumination of different angles to the sample by sequentially displaying, according to a first pattern of point light sources at different positions, the point light sources of the first pattern on a display; an image acquisition module for obtaining an image for each illumination angle of the sample by using a camera whenever illumination of different angles is provided by the point light sources of the first pattern; and a Fourier ptychography image restoration module for restoring a first Fourier ptychography image using a plurality of images for each illumination angle obtained using the camera.
According to the present disclosure, there is an advantage of being able to restore a Fourier ptychography image at low cost without using a separate illumination device, and to improve the quality of a Fourier ptychography image through a high degree of freedom.
In order to fully understand the present disclosure, operational advantages of the present disclosure, and objects achieved by implementing the present disclosure, reference should be made to the accompanying drawings illustrating preferred embodiments of the present disclosure and to the contents described in the accompanying drawings.
Hereinafter, the present disclosure will be described in detail by describing preferred embodiments of the present disclosure with reference to accompanying drawings. However, the present disclosure can be implemented in various different forms and is not limited to the embodiments described herein. For a clearer understanding of the present disclosure, parts that are not of great relevance to the present disclosure have been omitted from the drawings, and like reference numerals in the drawings are used to represent like elements throughout the specification.
Throughout the specification, reference to a part “including” or “comprising” an element does not preclude the existence of one or more other elements and can mean other elements are further included, unless there is specific mention to the contrary. Also, terms such as “unit”, “device”, “module”, “block”, and the like described in the specification refer to units for processing at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software.
Referring to
The optical system device 100 has an optical structure for providing an image of the sample to the smartphone 200.
The smartphone 200 is coupled to the optical system device 100, provides illumination to the optical system device 100, and obtains images of a sample from the optical system device 100. The smartphone 200 restores a Fourier ptychography image by using the obtained images of the sample.
The optical system device 100 includes a light path portion 110, a plurality of lenses 120, 130, 140 and 150, a plurality of mirrors 160 and 170, and a sample holder 180.
On the sample holder 180, a sample is placed. The sample holder 180 is made of a transparent material, and illumination for the sample is provided from the smartphone 200. The conventional illumination for obtaining a Fourier ptychography image was provided using a separate illumination system including a plurality of LEDs and a control device for controlling the luminescence of the plurality of LEDs. Since the illumination is provided through an illumination system including a control device and a plurality of LEDs, the conventional Fourier ptychography image acquisition requires high cost.
In the present disclosure, a display of the smartphone 200 is used as a light source for providing illumination and does not require a separate illumination system. Due to this structure, the existing high cost problem can be solved.
The smartphone 200 displays an image on a display by causing OLED to emit light, and the present disclosure is to utilize the OLED constituting the display of the smartphone as illumination for obtaining a Fourier ptychography image.
However, in order to obtain a Fourier ptychography image, illumination must be provided from a plurality of angles. Control of the display screen of the smartphone display for providing such illumination will be described with reference to a separate drawing.
A first end of the optical system device 100 is coupled to the smartphone 200, particularly to the camera of the smartphone 200. The light path portion 110 of the optical system device 100 provides an optical path such that an image of the sample to which illumination is provided is captured by the camera of the smartphone 200.
Referring to
At bending points of the light path portion 110, mirrors 160 and 170 are installed to reflect light by illumination so that the light can be transmitted along the light path portion. In the embodiment shown in
A plurality of lenses 120, 130, 140 and 150 are coupled to the light path portion 110, and the lenses function to adjust a focal length between a sample and a camera of the smartphone 200 and focus diffused light.
In the present disclosure, since an image is obtained using a camera pre-installed in a smartphone and a display of the smartphone is used as illumination, a Fourier ptychography image can be obtained in a simplified manner and at low cost.
In the smartphone 200, an application for controlling the smartphone 200 is installed to obtain a Fourier ptychography image. A detailed structure of the application installed on the smartphone and an operation of the smartphone 200 by the application will be described with reference to a separate drawing.
Referring to
In order to restore a Fourier ptychography image, it is necessary to have a plurality of images illuminated from different angles.
As shown in
When a preset number of images are obtained at different illumination angles, Fourier ptychography image restoration is performed using the obtained images. Since a plurality of images illuminated at different illumination angles are required, the illumination system for Fourier ptychography image restoration requires a plurality of LEDs arranged in different positions, and requires a control device for controlling light emission and off of such LEDs.
As described above, in the present disclosure, a display of a smartphone 200 performs the role of the illumination system shown in
Referring to
The display control module 400 controls the display of the smartphone such that only a selected region of the display of the smartphone emits light and LEDs of the remaining regions of the display are turned off to form a point light source.
The display control module 400 provides illumination at different angles to the sample while changing a position of the point light source based on a preset pattern. Since the illumination angles required to obtain a Fourier ptychography image are preset, and the height between the smartphone display and the sample is preset, it is possible to provide illumination at a required angle by adjusting the position where the point light source is formed on the display.
According to a preferred embodiment of the present disclosure, a pattern in which the positions of the point light sources are set is stored in advance, and the point light sources are sequentially displayed on the display at different positions according to the pattern.
Referring to
In order to provide illumination from various angles, a plurality of point light sources are arranged at regular intervals on the top, bottom, left, and right sides around the first point light source 500 formed in the center.
The display control module 400 in
The display control module 400 activates all the point light sources included in the pattern at a preset time interval according to a pre-determined order. A total of 25 point light sources are shown in
The image acquisition module 410 controls the camera of the smartphone 200 to obtain, whenever a point light source is formed according to the pattern, an image influenced by the point light source.
For example, the image acquisition module 410 controls the camera of the smartphone to obtain a first image, when the first point light source 500 shown in
The image acquisition module 410 controls the camera to obtain a plurality of images for each illumination angle, and the images for each illumination angle obtained in the camera through the image acquisition module 410 are provided to the Fourier ptychography image restoration module 420.
The Fourier ptychography image restoration module 420 restores a Fourier ptychography image by using the plurality of images for each illumination angle obtained through a camera of a smartphone.
In an optical system, a bandwidth of an image acquisition system is determined according to a numerical aperture of an objective lens, and a resolution of an obtained image is determined in proportion to the bandwidth. In addition, a central axis of the bandwidth changes according to an irradiation angle of the point light source. When an i-th image is taken, when the irradiation angle from an x-axis is αi and the irradiation angle from a y-axis is ßi, the central axis of the bandwidth with respect to the spatial frequency of an object moves in parallel in proportion to the values of sin(αi) and sin(βi). As the central axis of the bandwidth moves, each image contains information about the different spatial frequencies of the object. Fourier ptychography image restoration is to restore high-resolution object information of a wider synthesis bandwidth by applying an image quality enhancement algorithm called iterative phase retrieval using the information. In this case, the improvement of the synthesis bandwidth is determined according to the maximum irradiation angle, and the degree of restoration varies according to the scanning pattern.
Meanwhile, according to an embodiment of the present disclosure, a Fourier ptychography image may be restored using a plurality of patterns. For example, the Fourier ptychography image restoration module 420 restores a first Fourier ptychography image by obtaining images for each illumination angle according to the pattern shown in
Meanwhile, the pattern may have different shapes and colors of the point light sources as well as the positions of the point light sources. Various shapes of the point light source as well as the circular shape and elliptical shape described above may be applied to the point light source, and the point light source of each pattern may be set to a different color.
The image quality evaluation module 430 evaluates an image quality of each Fourier ptychography image when a plurality of Fourier ptychography images are restored using a plurality of patterns. The image quality evaluation module 430 may be performed using a previously known image quality evaluation algorithm, and since image quality evaluation is a general technique, a detailed description thereof will be omitted. The image quality evaluation module 430 may evaluate the quality of each Fourier ptychography image in a manner such as a rating or a score.
After a plurality of Fourier ptychography images are restored using a plurality of patterns and the quality of each Fourier ptychography image is evaluated, the image selection module 440 selects a Fourier ptychography image having the best image quality, and outputs the selected image as a final image.
When illumination of a specific angle is provided using an illumination system as in the prior art, there is a restriction on the degree of freedom for freely changing the pattern. However, since the present disclosure uses a display of a smartphone as an illumination for Fourier ptychography image restoration, there is no restriction on the degree of freedom for changing the pattern of the point light source, so it is possible to use a plurality of patterns. In addition, since a Fourier ptychography image providing the highest image quality can be selected among the plurality of patterns, the Fourier ptychography image restoration performance can be improved.
In the point light source pattern according to the first embodiment described with reference to
In
Referring to
Since the position of each point light source is different, the distance of the sample from the point light source is also different. According to the study of the inventors of the present disclosure, better Fourier ptychography image restoration is possible when illumination angles are different for accurate Fourier ptychography restoration but the same intensity of illumination is provided. To this end, in the pattern according to the second embodiment of the present disclosure, the size of each point light source is determined in consideration of the distance from the sample, and the size of the point light sources is set such that the fourth point light source 600 closest to the sample has the smallest size, and the size of the point light sources gradually increases as the distance from the fourth point light source 600 increases.
Such adjustment of the size of the point light source is an advantage of the present disclosure, which is difficult to be applied in a method using an existing physical illumination system, and by adjusting the size of the point light sources, uniform intensity of illumination can be provided despite the difference in distance, so that better Fourier ptychography image restoration becomes possible.
Referring to
In the pattern according to the third embodiment, not only the sizes of the point light sources are set differently, but also the shapes are set differently. At least some of the point light sources other than the fifth point light source 700 have an elliptical shape.
The shape and size of the point light sources are determined based on the separation direction and the separation distance from the fifth point light source 700.
As in the second embodiment, point light sources relatively far from the fourth point light source 600 have a relatively large size, and point light sources located relatively close to the fourth point light source 600 have a relatively small size.
Referring to
Meanwhile, a eighth point light source 730 spaced apart from the fifth point light source 700 in a second direction (vertical direction) orthogonal to the first direction (horizontal direction) has an elliptical shape having a major axis in the second direction.
The reason for setting the shape of the point light sources in consideration of the separation direction from the fifth point light source 700 is that when the shape of the point light source is set in an elliptical shape in consideration of the separation direction, it is suitable to provide illumination of uniform intensity for each point light source.
In the present disclosure, since point light sources are formed using a display of a smartphone, the size and shape can be freely adjusted, and through this degree of freedom, more uniform intensity of illumination can be provided for each angle, so that a better Fourier ptychography image can be restored compared to the case of using a physical illumination system.
First, a smartphone 200 on which an application is installed is coupled with an optical system device 100 (step 800). As shown in
After the smartphone 200 and the optical system device 100 are coupled, the application installed on the smartphone sequentially provides illumination by the point light sources based on the first preset pattern (step 802).
As in the first embodiment described above, illumination at different angles is provided while sequentially activating point light sources at different positions included in the pattern according to preset time intervals.
As in the second embodiment and the third embodiment, other point light sources may have different sizes and shapes compared to the first point light source aligned vertically with the sample, and it is preferable to set the size and shape of each point light source such that illumination of uniform intensity can be provided even when illumination is provided by point light sources at different positions.
Whenever illumination is sequentially provided by point light sources in step 802, the smartphone is controlled to obtain an image for each illumination angle through the camera of the smartphone (step 804).
After a plurality of images are obtained for each illumination angle, a first Fourier ptychography image is restored using the obtained images for each illumination angle (step 806).
After the first Fourier ptychography image is restored, the image quality of the restored first Fourier ptychography image is evaluated (step 808).
The steps 802 to 808 above are repeatedly performed for N patterns prepared in advance, N Fourier ptychography images are restored by restoring a Fourier ptychography image for each pattern, and the image quality is evaluated for each Fourier ptychography image.
After the image quality is evaluated for each Fourier ptychography image, a Fourier ptychography image with the best image quality is selected (step 810).
A method according to the present disclosure can be implemented as a computer program stored in a medium for execution on a computer. Here, the computer-readable medium can be an arbitrary medium available for access by a computer, where examples can include all types of computer storage media. Examples of a computer storage medium can include volatile and non-volatile, detachable and non-detachable media implemented based on an arbitrary method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data, and can include ROM (read-only memory), RAM (random access memory), CD-ROM's, DVD-ROM's, magnetic tapes, floppy disks, optical data storage devices, etc.
While the present disclosure is described with reference to embodiments illustrated in the drawings, these are provided as examples only, and the person having ordinary skill in the art would understand that many variations and other equivalent embodiments can be derived from the embodiments described herein.
Therefore, the true technical scope of the present disclosure is to be defined by the technical spirit set forth in the appended scope of claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2021-0021683 | Feb 2021 | KR | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20140133702 | Zheng | May 2014 | A1 |
| 20140267670 | Tipgunlakant | Sep 2014 | A1 |
| 20170178317 | Besley | Jun 2017 | A1 |
| 20170371141 | Besley | Dec 2017 | A1 |
| 20180203122 | Grauer | Jul 2018 | A1 |
| 20180231761 | Dai | Aug 2018 | A1 |
| 20180299376 | Cooper | Oct 2018 | A1 |
| 20220046169 | Divitt | Feb 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 10-2082747 | Feb 2020 | KR |
| Entry |
|---|
| Zheng et al. “Wide-Field, high-resolution Fourier ptychographic microscopy” Nat Photonics. Sep. 1, 2013. |
| Wang et al. “Enhanced image reconstruction of ptychographic microscopy with double-height illumination” Optics Express. Dec. 6, 2021. |
| Tomas Aidukas et al., “Miniature Fourier Ptychographic Microscope Using Mobile Phone Camera Sensors”, DOI:10.1364/MICROSCOPY.2018.MTu4A2,2018.04.17.). |
| Pavan Chandra Konda et al., Fourier Ptychography: current applications and future promises, vol. 28, No. 7/Mar. 30, 2020/Optics Express 9603-9630, Mar. 19, 2020). |
| Number | Date | Country | |
|---|---|---|---|
| 20230014453 A1 | Jan 2023 | US |
