Patent Application
20240273691
The disclosure relates to a digital hologram processing method in the field of digital holography technology, and in particular, relates to a digital holographic wrapped phase aberration compensation method based on deep learning.
When microstructural surface topography is measured using digital holography technology, the off-axis optical path structure, the spherical curvature mismatch between the object beam and the reference beam, and the optical aberration introduced by the system construction, etc. cause the phase information after holographic reconstruction to contain not only the phase of the sample to be measured, but also a large amount of tilt, quadratic, and high-order phase aberration. Not until these phase aberrations are accurately compensated will the three-dimensional profile of the sample to be measured be effectively restored. In the wrapped phase map obtained by the arctangent transformation, these phase aberrations appear as dense oblique curved fringes modulated by the phase of the sample to be measured. As the phase aberration introduced by the system increases, the wrapped phase fringes become denser.
When the structural edge of the sample to be measured is located on a dense fringe region, fringe dislocations may easily occur. Both fringe dislocations and coherent noise introduced by coherent light sources reduce the data reliability of wrapped phase map produce errors in the phase unwrapping process. In the conventional phase aberration numerical compensation methods, continuous phase compensation is performed after phase unwrapping. Inaccurate phase unwrapping directly restricts the effectiveness of phase aberration compensation.
The wrapped phase map be denoised and smoothed before phase unwrapping. However, for the dense fringe-wrapped phase map, inappropriate phase filtering may cause fringe damage, aggravate fringe misalignment or over-smooth the edge of the sample structure, and introduce new phase errors.
To solve the above technical problems, the disclosure provides a digital holographic wrapped phase aberration compensation method based on deep learning, in which a network model of wrapped phase map input and Zernike polynomial coefficients output is directly established, and most of aberration components in the wrapped phase map are automatically compensated before phase unwrapping, so that advantages such as fast calculation speed, high accuracy, and robustness are provided without any manual intervention, input of initial parameters, and limitation of sample types.
The disclosure includes the following technical solutions.
The method includes two stages including a network training stage and a holographic measurement stage, and the method is divided into the following steps:
The network training stage specifically includes the following.
In step one, the computer generates a large number of random Zernike polynomial coefficients A according to a predetermined numerical range, the Zernike polynomial coefficients A are used to fit several continuous two-dimensional surfaces as phase aberrations through Zernike polynomials to simulate the phase aberrations, and the phase aberrations are superimposed on the same type of predetermined microstructure phase as the sample to be measured to construct a microstructure phase aberration distribution φ, wherein a two-dimensional curved surface serves as a phase aberration, and a microstructure phase aberration distribution o is obtained after a phase aberration is superposed, and different two-dimensional curved surfaces correspond to different microstructure phase aberration distributions φ.
The predetermined microstructure phase is also obtained by computer simulation.
In step two, the microstructure phase aberration distribution φ is converted into a complex exponent, and a phase angle of the complex exponent is calculated to obtain a simulated wrapped phase map ϕ with a value in a range of [−π, π].
In step three, the neural network model is established, the neural network model is trained by treating the simulated wrapped phase map ϕ as an input of the neural network model and treating the corresponding Zernike polynomial coefficient A as a label of the neural network model, and the trained neural network model is obtained.
The steps of the holographic measurement stage includes the following.
In S1, a digital holographic optical setup is built to measure the sample and record a hologram of the sample to be measured, numerical reconstruction is performed based on the hologram to obtain complex amplitude U of the sample to be measured, a wrapped phase map of the complex amplitude U is calculated and inputted into the trained neural network model, and a Zernike polynomial coefficient Ac is outputted.
In S2, phase aberration φac is fitted by using the Zernike polynomial coefficient Ac, and a conjugate complex exponent exp(−jφa) of the fitted phase aberration φac is multiplied by the complex amplitude U compensate for most of the phase aberration in the complex amplitude U and to obtain a pre-compensated wrapped phase map with most of the phase aberration being compensated.
In S3, phase filtering and phase unwrapping are performed on the pre-compensated wrapped phase map to obtain a continuous phase distribution φc containing only a small part of residua phase aberration, and edge enhancement and local adaptive threshold segmentation are performed on the continuous phase distribution φc to obtain a binary mask that only represents a background region.
In S4, phase data of the background region in continuous phase distribution φc is extracted by using the binary mask, a Zernike polynomial equation set is constructed based on the phase data of the background region, a Zernike polynomial coefficient Ar of residual aberration is solved, and Zernike polynomial fitting is performed by using the Zernike polynomial coefficient Ar of the residual aberration to obtain a residual aberration phase distribution or.
In S5, the phase distribution φr is subtracted from the continuous phase distribution φc to recover and obtain a true phase of the sample to be measured, and wavelength conversion and outputting are performed on the true phase to obtain the three-dimensional profile distribution of the sample to be measured.
In the above processing of the disclosure, the wrapped phase aberration compensation based on deep learning is particularly performed before phase unwrapping. In this way, most of the phase aberration and fringe patterns modulated on the sample structure can be eliminated, and phase abnormality problems such as fringe damage and misalignment during the phase filtering process can be avoided. As such, coherent noise is suppressed while the edge information of the sample structure is protected, so that the quality of phase data is significantly improved, and automatic and accurate compensation of digital holographic phase aberration is achieved.
It is of great significance to directly perform aberration compensation on the wrapped phase map before phase unwrapping.
In the first step, the phase aberration constructed by the Zernike polynomial coefficients randomly-generated by the computer are superimposed on various microstructure phase models and the overall phase is truncated within the range of [−π, π], so as to simulate the actual digital holographic wrapped phase map and to form the training samples for being inputted to the neural network model. When training the network, there is no need to input actual digital holographic wrapped phase map and Zernike polynomial coefficients.
The neural network model in the disclosure is any neural network model with a residual structure for classification or a variant thereof. The network only needs to be trained once, and then the trained network can be used to perform unlimited regression analyzes on unknown samples.
The neural network model is resnet50, which can be implemented with frameworks such as Tensorflow or Pytorch.
In the S1, the digital holographic optical setup is any holographic optical setup such as coaxial structure, an off-axis structure, a microscopic imaging structure, a multi-wavelength structure, or a super-resolution structure and includes tilt aberration introduced by off-axis interference, secondary phase aberration introduced by a microscope objective lens, or high-order phase aberration introduced by optical path construction.
The sample to be measured is any microstructure used for holographic imaging and is a transmission type sample or a reflection type sample. The transmission sample is, for example, biological cells, a biological tissue structure, a microlens array, and the transmission type sample is, for example, a MEMS (Microelectromechanical Systems) micro-nano structure, an integrated circuit chip, a silicon wafer.
In the S1, the numerical reconstruction includes a first step and a second step performed in sequence, wherein the first step is a phase shifting step or a spatial filtering step, and the second step is a Fresnel diffraction method, a convolution method, an angular spectrum method, or a compressed sensing reconstruction step.
In the S3, the edge enhancement is achieved by superimposing gradients of continuous phases and a gradient binarization result, and the local adaptive threshold segmentation is to separately calculate a threshold of each pixel in the continuous phases and perform binarization processing.
Random Zernike polynomial coefficients and the corresponding wrapped phase maps are generated by a computer and are respectively treated as the learning labels and the network input to create a data set for neural network model training in the disclosure. A digital holographic optical setup is built to record the sample hologram, the wrapped phase map is inputted into the trained network after numerical reconstruction, and the Zernike polynomial coefficients are outputted to reconstruct the phase aberration distribution and to compensate the complex amplitude in the spatial domain. Phase filtering and unwrapping are performed on the pre-compensated wrapped phase map, and Zernike polynomial fitting based on background segmentation is performed on the unwrapped phase to compensate for residual aberration.
A network model with the wrapped phase map as input and the Zernike polynomial coefficients as output is directly established in advance, and most of the aberration components in the wrapped phase map are automatically compensated before phase unwrapping, so that the problem of difficulty in compensating the phase aberration component in the wrapped phase map before phase unwrapping is solved in the disclosure.
The technical solutions provided by the disclosure at least achieves the following beneficial technical effects.
The mapping relationship between the wrapped phase map and the Zernike polynomial coefficients is directly established by training the neural network, and the aberration is pre-compensated before the phase is unwrapped in the disclosure. The density of wrapped phase fringes is reduced, and problems such as fringe damage, misalignment, and excessive smoothing of sample structure edges during phase filtering are prevented from occurring, so that the reliability of wrapped phase data is effectively improved.
Compared to other methods that use deep learning to compensate for phase aberration, in the disclosure, a computer is only used to generate a large number of random Zernike polynomial coefficients and corresponding wrapped phase maps to create a data set for training the neural network. During network training, there is no need to build a digital holographic optical setup to record the actual microstructure hologram, numerically reconstruct the wrapped phase map, or go through a complex process of solving Zernike polynomial coefficients (including phase filtering, phase unwrapping, background segmentation, and Zernike polynomial fitting) to create a data set, so that the difficulty of making a network training data set is simplified.
Compared to the conventional phase aberration compensation methods, in the disclosure, there is no need for any manual intervention, input of initial parameters, and limitation of sample types, so the efficiency of phase aberration compensation calculations is improved. The disclosure not only can be used in conventional digital holographic systems, but can also be widely used in multi-wavelength digital holographic systems, synthetic aperture super-resolution holographic systems, and large field of view stitching digital holographic microscopy systems to simultaneously process multiple aberration compensation tasks.
In summary, the method of the disclosure only uses the simulation data set to train the network and compensates most of the aberration before phase unwrapping. In this way, the reliability of the wrapped phase data is improved, the accuracy of phase recovery is significantly improved, and advantages such as fast calculation speed and accurate aberration compensation are provided.
The disclosure is going to be further described in the following paragraphs together with the embodiments and drawings.
The embodiments of the disclosure are shown in the flow chart of
According to size of input data of a neural network (residual network ResNet with 20 convolutional layers), a length and a width of a two-dimensional plane are set on the X-axis and Y-axis respectively, the values are within the range of [−1, 1], and the number of sampling points is M×M. A set of randomly generated real numbers are used as Zernike polynomial coefficients A=[a0 a1 a2 . . . an]T, a0 a1 a2 . . . an represent the Zernike polynomial coefficients of the first n terms, and T represents matrix transpose.
A continuous two-dimensional surface is obtained through Zernike polynomial fitting to represent phase aberration. The phase aberration is superimposed with a simulated microstructure phase to simulate a microstructure phase aberration distribution ϕ.
A simulated wrapped phase map ϕ is thus obtained and is specifically expressed as:
where (x, y) is a coordinate point on the two-dimensional plane, 1≤x≤M, 1≤y≤M, i=√{square root over (−1)} represents an imaginary number, and angle represents an operation of obtaining a complex phase angle. ϕ(x, y) represents a wrapped phase with coordinates (x, y) on the wrapped phase map.
A training set with a large number of simulated wrapped phase maps ϕ and corresponding Zernike polynomial coefficients A as network inputs and labels is calculated and generated to train a ResNet convolutional neural network as shown in
A digital holographic coaxial optical path is built, and a positive first-order term OR* in a hologram of a sample to be measured is extracted through a phase shift method. Alternatively, a digital holographic off-axis optical path is built, and a positive first-order term OR* in the hologram is extracted through a spatial filtering algorithm. Next, a holographic reconstruction algorithm such as a Fresnel diffraction method, a convolution method, an angular spectrum method, or compressed sensing is adopted to reconstruct the complex amplitude U of the sample to be measured. A sample to be measured wrapped phase map angle(U) is inputted into the trained neural network model, and a set of Zernike polynomial coefficients Ac is automatically outputted to approximately represent the phase aberration.
Next, a phase aberration φac is fitted using the Zernike polynomial coefficient Ac. A conjugate complex exponent exp(−jφa) thereof is multiplied by the complex amplitude U of the sample to be measured to eliminate most of the phase aberration components in the complex amplitude. A pre-compensated wrapped phase map is calculated, that is, angle[U·exp(−jφac)].
The pre-compensated wrapped phase map then is subjected to phase filtering based on sine and cosine transform and phase unwrapping based on least squares iteration, and a continuous phase distribution φc containing only a small amount of residual aberration is obtained. Gradient of the continuous phase distribution φc is calculated, and the gradient is binarized and added to φc to obtain the continuous phase distribution φc with an enhanced edge feature. An adaptive threshold of each pixel in Φc is calculated, and local adaptive threshold segmentation is performed on Φc e to obtain a binary mask that only represents a background region.
Finally, phase data of the background region in the continuous phase distribution φc is extracted by using the binary mask. Only the background phase is fitted with a Zernike polynomial to obtain the Zernike polynomial coefficient Ar and the phase distribution φr of the residual aberration. Subtracting φr from φc recovers a true phase of the sample to be measured, and finally a three-dimensional profile distribution of the sample is outputted.
Through the above implementation, to address the problems found in the related art that the continuous phase aberration compensation method after unwrapping is limited by the coherent noise level and phase data reliability, in the disclosure, the aberration compensation is placed before the wrapped phase. Combined with the deep learning method in artificial intelligence, most of the aberration in the wrapped phase map may be compensated without first unwrapping. The density of wrapped phase fringes is reduced, problems such as fringe damage, misalignment, and excessive smoothing during the phase filtering process are prevented from occurring, and the reliability of phase data and the accuracy of holographic measurement are improved.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202211319611.3 | Oct 2022 | CN | national |
This application is a continuation of international application of PCT application serial no. PCT/CN2023/074510, filed on Feb. 6, 2023, which claims the priority benefit of China application no. 202211319611.3, filed on Oct. 26, 2022. The entirety of each of the above mentioned patent applications is hereby incorporated by reference herein and made a part of this specification.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2023/074510 | Feb 2023 | WO |
| Child | 18645399 | US |
