Patent Application
20240412333
The present invention relates to an observation system and an artifact correction method for the observation system.
In image reconstruction from sparse sampling data in an observation device such as a charged particle microscope device, a magnetic resonance imaging (MRI) device, and a computer tomographic device imaging (CT: computed tomography), artifacts as a secondary image pattern which does not exist in an original imaging target may occur in a reconstructed image (reconstructed image). A technique for reducing the artifacts has been proposed in PTL 1, for example.
According to a method disclosed in PTL 1, the artifacts of the reconstructed image are reduced by using a correction engine trained to learn artifact reduction of the reconstructed image. However, in a case of this method, the artifacts differ depending on an imaging method. Therefore, an effect of reducing the artifacts varies, thereby causing a problem in that it is difficult to uniformly reduce the artifacts.
PTL 1: JP2020-99667A
In order to solve this problem, the present invention aims to provide an observation system and an artifact correction method which can uniformly reduce artifacts varying due to a sampling coordinate group in a reconstructed image.
In order to solve the above-described problem, an observation system according to the present invention includes an imaging device and a processor subsystem. The processor subsystem sets a sparse sampling coordinate group with respect to a sample, acquires a pixel value group corresponding to the sampling coordinate group on the sample, and generates a second reconstructed image by assigning either the pixel value group or a first reconstructed image generated based on the pixel value group to a correction engine.
The correction engine is trained by using the following data (1) to (3) regarding the sample or a learning sample:
According to the present invention, it is possible to provide an observation system and an artifact correction method which can uniformly reduce artifacts varying due to a sampling coordinate group in a reconstructed image.
Hereinafter, embodiments will be described with reference to the drawings. In the accompanying drawings, functionally similar elements may be designated by the same number. The accompanying drawings represent embodiments and implementation examples in accordance with principles of the present disclosure. Meanwhile, the accompanying drawings are provided to understand the present disclosure, and should not be used to limit the present disclosure in any way. Descriptions in the present specification are merely typical examples, and are not intended to limit the scope of claims or application examples of the present disclosure in any way.
The present embodiment is described in sufficient detail for those skilled in the art to implement the present disclosure. Meanwhile, other implementations and other forms can be adopted, and it is necessary to understand that a configuration and a structure can be modified, and various elements can be replaced without departing from the scope and the concept of the technical idea of the present disclosure. Therefore, the following description should not be interpreted as being limited to this.
With reference to
As an example, the scanning electron microscope device 100 includes a scanning electron microscope 101, an input/output unit 121, a control unit 122, a processing unit 123, a storage unit 124, and an image processing unit 125. The scanning electron microscope 101 is an imaging unit that images a sample 106 and acquires the image. The input/output unit 121, the control unit 122, the processing unit 123, the storage unit 124, and the image processing unit 125 are realized by a computer 200 (processor subsystem), for example. For example, the computer 200 may be a general-purpose computer, and may include a CPU 151 (processor), a ROM 152, a RAM 153, a hard disk drive 154, an input device 155, and a display 156 (display unit). The computer 200 may be connected to the scanning electron microscope 101 via a network. The CPU 151 is one type of processors, and may be a GPU, a semiconductor device capable of other arithmetic processing, or a combination thereof. The computer 200 may have a plurality of processors or memories, and a configuration of the computer 200 in
The scanning electron microscope 101 generates an electron beam 103 (charged particle beam) from an electron gun 102, and focuses the electron beam 103 on a surface of the sample 106 by causing the electron beam 103 to pass through a condenser lens 104 and an objective lens 105. A secondary electron or a backscattered electron generated from the sample 106 is detected by a detector 108, and an image generated in accordance with a detection signal is saved in the storage unit 124. The storage unit 124 may include the RAM 153 and hard disk drive 154 (memory) of the computer 200. That is, the storage unit 124 can also be a memory. The scanning electron microscope 101 and the storage unit 124 form an image acquisition unit that acquires an image of the sample.
With regard to the detector 108, one scanning electron microscope 100 may include a plurality of detectors of the same type. In addition, the detector 108 may be a combination of different types of detectors (for example, a combination of a detector that detects an electron and a detector that detects an electromagnetic wave, a combination of detectors that detects only particles whose energy or spin direction falls within a specific range, or a combination of a secondary electron detector and a backscattered electron detector), or a plurality of detectors of the same type may be provided at different disposition positions. When the plurality of detectors are provided, a plurality of images can be acquired by performing one normal imaging operation.
The sample 106 can be placed on a stage 107. An image can be acquired at any position on the sample 106 by moving the stage 107 in a horizontal direction. In addition, the sample 106 can be scanned with the electron beam 103 by causing a beam deflector 109 to two-dimensionally change an orientation of the electron beam 103.
The input/output unit 121 inputs an instruction relating to an image capturing position or an image capturing condition from the outside by using the input device 155, and visualizes the obtained image. For example, the input/output unit 121 has a function of outputting and displaying the visualized image to and on the display 156. In the scanning electron microscope device 100, the display 156 may be omitted, and the input device 155 may be omitted. The input device 155 may be a network interface.
The control unit 122 is configured to perform various types of controlling on the scanning electron microscope 101 (for example, controlling a voltage applied to the electron gun 102, controlling focal point positions of the condenser lens 104 and the objective lens 105, controlling a position of the stage 107, and controlling a deflection degree of the beam deflector 109). The scanning electron microscope 101 in the following embodiment can perform sparse sampling, and the control unit 122 realizes the sparse sampling by controlling the electron gun 102, the condenser lens 104, the objective lens 105, and the beam deflector 109. The control unit 122 can be realized by a computer program stored in the CPU 151 and the ROM 152 of the computer 200. The computer program can be recorded on a portable recording medium. The portable storage medium is a non-volatile storage medium.
In addition, the control unit 122 is configured to control the input/output unit 121, the processing unit 123, the storage unit 124, and the image processing unit 125. The processing unit 123 performs various types of processing, for example, such as processing for generating a random number for setting a random sampling coordinate, and arithmetic processing for automatic focusing required for focusing a focal point of the electron beam 103 on a surface of the sample 106. The storage unit 124 stores (saves) information relating to the interest region, a sparse sampling coordinate group, a reconstructed image, a trained model for artifact reduction processing, data relating to an artifact occurrence degree of the reconstructed image, a learning data set, and various processing parameters. The image processing unit 125 is in charge of image processing on the acquired data. A computer and an operator performing learning of the correction engine and a computer and an operator performing artifact correction of the present embodiment by using the trained correction engine may be the same or may be different.
The image processing unit 125 is realized by the computer 200 and image processing software stored in the computer 200. For example, the image processing unit 125 includes a sampling coordinate setting unit 131, a reconstructed d image generation unit 132, an artifact reduction processing unit 133, an artifact occurrence degree evaluation unit 134, a sampling coordinate adjustment unit 135, and a correction engine learning unit 136. Each of the units 131 to 136 is realized in such a manner that the CPU 151 (processor) of the computer 200 reads and executes the image processing software stored in a memory (ROM 152 or the like).
The sampling coordinate setting unit 131 sets a sparse sampling coordinate group in the interest region of the sample 106. In the present specification, expression of “setting the sampling coordinate group” is used to mean not only generating a new sampling coordinate group but also updating a preset sampling coordinate group. The sampling coordinate group includes coordinate data of a plurality of sampling points. The reconstructed image generation unit 132 generates a reconstructed image (first reconstructed image), based on a pixel value group acquired by irradiating the sparse sampling coordinate group of the interest region with the electron beam. As an example, a value of the secondary electron corresponding to a pixel of an image signal obtained from the scanning electron microscope 100 (imaging device) can be digitized as a pixel value, and a set of the pixel values can be set as the pixel value group. Alternatively, the value of the secondary electron can be converted once into a certain value, and thereafter, the value can be further converted into the pixel values to obtain the pixel value group.
The artifact reduction processing unit 133 performs artifact reduction processing on the first reconstructed image, and generates a reconstructed image (second reconstructed image) having fewer artifacts compared to the first reconstructed image. In addition, the artifact reduction processing unit 133 trains the correction engine by using a plurality of sets of learning data sets (teaching data) in which the sparse sampling coordinate group, the reconstructed image having the artifact, and fully sampled captured images (images having a few artifact or no artifact) are used as one set. In this manner, the correction engine which performs artifact reduction processing of the reconstructed image is generated. Instead of (or in addition to) the fully sampled captured images, learning of the correction engine can be performed by using the reconstructed image after the artifact reduction processing. For example, when the fully sampled captured images cannot be obtained, learning can be performed by using the reconstructed image after the artifact reduction processing instead of the fully sampled captured images. In addition, learning of the correction engine can also be performed by using an artificially prepared ideal image such as a simulation image instead of the fully sampled captured images.
The artifact occurrence degree evaluation unit 134 evaluates the artifact occurrence degree in the reconstructed image, based on predetermined evaluation criteria. The sampling coordinate adjustment unit 135 adds the sampling coordinate group to the sampling coordinate group set by the sampling coordinate setting unit 131, based on the artifact occurrence degree. The correction engine learning unit 136 learns the correction engine by using (i) the sampling coordinate group, (ii) the pixel value group corresponding to the sampling coordinate group or the reconstructed image having the artifact which is reconstructed from the pixel value group by using the sampling coordinate group, and (iii) the image which includes no artifact or in which the artifacts are reduced. The correction engine learning unit 136 may acquire the sampling coordinate group and the reconstructed image, based on the sample 106 imaged by the scanning electron microscope 101, and may learn the correction engine, based on the data. Alternatively, learning of the correction engine can be performed by using the sampling coordinate group or the reconstructed image obtained based on a learning sample different from the sample 106, which is obtained by another imaging device or simulation device.
With reference to
When the interest region 203 is set, a sparse sampling coordinate group 205 is set in the interest region 203. For example, the sparse sampling coordinate group 205 is set in only one frame. A reference numeral 204 in
It is desirable that positions of the sampling points in the sparse sampling coordinate group 205 is set to have high randomness. On the other hand, when the positions of the sampling points cannot have the high randomness due to hardware constraints, the artifacts tend to occur in the reconstructed image.
An image 211 on a middle left side in
In addition, an image 213 on a lower left side in
When a sampling rate in the interest region 203 is sufficient for image reconstruction, the correction engine can sufficiently reduce the artifacts of the reconstructed image. On the other hand, when the sampling rate is not sufficient for image reconstruction, in some cases, it may be difficult to sufficiently reduce the artifacts by using the correction engine alone. In this case, in the present embodiment, the sampling coordinate group can be additionally set to image the sample 106, and a reconstructed image (third reconstructed image) can be acquired again. An image 214 on a lower right side in
An example of a procedure for performing the artifact correction processing in the first embodiment will be described with reference to a flowchart in
In Step S301, an operator sets the interest region 203 on the sample 106 via the input/output unit 121, and determines an imaging region of the sample 106. This step corresponds to setting the interest region 203 on the sample image 202 in the low magnification image 201 (
Subsequently, in Step S302, the sparse sampling coordinate group 205 is set in the interest region 203. A method for setting the sampling points of the sparse sampling coordinate group 205 is as follows, and several methods are conceivable.
(i) Image analysis is performed on the low magnification image 201. Based on an analysis result thereof, a range where a significant signal exists is extracted, and sampling points are set in the extracted range.
(ii) Sampling points are set, based on information taught in advance by the user.
In the present embodiment, a sampling point setting method other than the above-described method may be adopted. The sparse sampling coordinate group 205 may be set by being divided into a plurality of frames instead of one frame. In addition, the same pixel may be sampled multiple times. When the same pixel is not sampled multiple times, the sampling coordinate group may be expressed by a binary image (for example, a pixel to be sampled is set to white, and a pixel not to be sampled is set to black). On the other hand, when the same pixel is sampled multiple times, the sampling coordinate group can be expressed as a multivalued image.
In Step S303, the scanning electron microscope 101 irradiates a position on the sample 106 corresponding to the sampling point of the sampling coordinate group 205 with the electron beam from the electron gun 102, and obtains the sparse pixel value group corresponding thereto. Then, in Step S304, the reconstructed image is generated from the obtained sparse pixel value group. The reconstructed image corresponds to the reconstructed image 212 having the artifact.
Here, as a method for generating the reconstructed image, for example, a compressed sensing method may be used. According to one compressed sensing method, it is assumed that sampling data is sparse in an expression space using a dictionary image, and an image close to the captured image during full sampling can be reconstructed by linear summation of the dictionary image from the sampling data having the lesser number of data in full sampling. The dictionary image may be a dictionary image prepared based on a general discrete cosine transform, or may be a dictionary image specified by the user. In addition, another method is a method for using a neural network trained to output the reconstructed image by receiving the sparse pixel value group as an input.
When the reconstructed image is generated in this way, in Step S305 subsequent thereto, the artifact of the reconstructed image is corrected by using the correction engine trained to learn the artifact reduction processing. That is, as illustrated in
Here, a specific example of a learning sequence of the correction engine used in Step S305 will be described once with reference to a flowchart in
In Step S401, a variable i=1 is set, and Steps S402 to S406 are looped until the number of learning data candidates i=N. A learning data set is generated through the loop. When the number of samples to be imaged is defined N1, the number of interest regions to be set for each sample is defined N2, and a type of the sparse sampling coordinate group of the interest region is defined as N3, the number of learning data candidates N becomes N=N1×N2×N3.
In Step S402, Steps S301 to S304 described in
Still another method for evaluating the artifact occurrence degree is a method for using the evaluation engine trained to learn (trained) to output a distribution of the artifact occurrence degrees. The evaluation engine is a network trained to learn to output a distribution of the artifact occurrence degrees by receiving the sampling coordinate group and the reconstructed image having the artifact as inputs. For example, a convolutional neural network is used as the network of the evaluation engine. A specific example of a learning sequence of the evaluation engine will be described later.
When the artifact occurrence degree is evaluated, in Step S404 subsequent thereto, it is determined whether the artifact is present in the reconstructed image. When the artifact is absent (YES), the process proceeds to Step S407; and when the artifact is present (NO), the process proceeds to Step S405.
When the artifact is absent in the reconstructed image, the reconstructed image is excluded from candidates of the learning data (Steps S405 and S406 are omitted, and the process proceeds to Step S407). On the other hand, when the artifact is present in the reconstructed image, in Step S405, the interest region is fully sampled to acquire an image having no artifact. Then, in Step S406, the sparse sampling coordinate group, the reconstructed image having the artifact, and the image having no artifact are added to the learning data set as one set of data. When the pixel value groups are obtained from a plurality of detectors, the reconstructed image having a plurality of corresponding artifacts and an image having no artifact may be added as one set of data. In Step S407, in a case of i=N, the loop in Steps S402 to S406 is completed, and in a case of i<N, the process returns to Step S401, and the above-described procedure is repeated. When the N-number of learning data sets are acquired, the process proceeds to Step S408.
In Step S408, the acquired N-number of learning data sets are saved in the storage unit 124. In Step S409 subsequent thereto, variable j=1 is set, and Steps S410 to S413 are looped until the number of learning times j=M. Through the loop, learning of the correction engine which performs the artifact reduction processing is performed by the correction engine learning unit 136.
First, in Step S410, based on an artifact correction parameter of the correction engine, the reconstructed image in which the artifact is corrected is output from the sparse sampling coordinate group and the reconstructed image having the artifact. In a case of j=1, the artifact correction parameter of the correction engine may be initialized with a random number, or may be initialized with a parameter learned by using another learning data set.
In Step S411 subsequent thereto, a correction error is calculated by comparing the reconstructed image in which the artifact is corrected and the image having no artifact. For example, the correction error may be calculated as a mean square error. Then, in Step S412, it is determined whether the correction error is equal to or smaller than a predetermined threshold value. When the correction error is equal to or smaller than the threshold value (YES), the process proceeds to Step S415, and when the correction error is greater than the threshold value (NO), the process proceeds to Step S413.
When the correction error is equal to or smaller than the predetermined threshold value (YES), the process proceeds to Step S415, the calculated artifact correction parameter is saved, and the procedure in
Referring back to
In Step S308, the sparse sampling coordinate group is additionally set in the interest region 203. For example, when a sampling rate of the interest region 203 is set to 10% in Step S302, while the randomness of the sampling coordinate group is maintained, the sampling coordinate group is additionally set so that the sampling rate of the interest region 203 becomes 20%. Other specific examples relating to Step S308 will be described later.
In Step S309, the position of the sample 106 corresponding to the sampling point of the added sampling coordinate group in the interest region 203 is irradiated with the electron beam to obtain the corresponding sparse pixel value group. The process returns to Step S304, and Steps S304 to S307 are performed again. In this case, when it is determined in Step S307 that the artifact is sufficiently corrected, the process proceeds to Step S310. In Step S310, the reconstructed image is displayed on the GUI. That is, at least one reconstructed image 214 having substantially no artifact is displayed on a GUI.
As described above, in the first embodiment, the sampling coordinate group is included in the teaching data, when the learning data set of the correction engine is generated. In this manner, the correction engine can learn a relationship between the sampling coordinate group and the reconstructed image of the artifact. Therefore, it is possible to uniformly reduce the artifacts varying due to the sampling coordinate group in the reconstructed image. In addition, the artifact occurrence degree of reconstructed images is evaluated, and the sampling rate is adjusted to a minimum level required for correcting the artifact. In this manner, while sample damage is suppressed, the artifact of reconstructed images can be corrected.
The image 510 corresponds to the reconstructed image 213 illustrated in
With reference to a flowchart in
First, in Step S601, an instruction is given to additionally set the sampling coordinate group. In accordance with the command, an additional sampling coordinate group is set according to a provisionally set sampling rate and a distribution range 204 of the sampling points. For example, the sampling coordinate group having the sampling rate different from the sampling rate of the sampling coordinate group set in step 302 and having the same distribution range 204 of the sampling points may be provisionally set as the additional sampling coordinate group.
In Step S602, based on the evaluation of the artifact occurrence degree (Step S306), the region having the high artifact occurrence degree is extracted, and the image 520 obtained by mapping the artifact occurrence degree as illustrated in
In Step S603 subsequent thereto, a mask image that divides the region having the high artifact occurrence degree and other regions is generated. An example of a method for generating the mask image is a binarizing method in which the pixel value in a region having the higher artifact occurrence degree than a predetermined threshold value is set to 1, and other pixel values are set to 0. The pixel values near the region having the high artifact occurrence degree are also effective for artifact correction. Therefore, a region where the pixel value is 1 can be widened by performing dilation processing on a binary image.
Then, in Step S604, the generated mask image is used to exclude the sampling coordinate group in the region having the low artifact occurrence degree. In this manner, it is possible to additionally set the sparse sampling coordinate group 530 (refer to
The procedure for additionally setting the sampling coordinate group in Steps S307 to S309 described above may be omitted under a predetermined condition. In some cases, since the sampling coordinate group is set in Step S302, and a condition for generating the correction engine is properly set, the artifact reduction processing may be sufficiently performed to such an extent that subsequent correction of the artifact is not required. In this case, Steps S307 to S309 may be omitted. In addition, it is also possible to configure a device so that Steps S307 to S309 are not performed regardless of the condition.
First, in Step S701, the learning data set is generated and saved in the same manner as Steps S401 to S408. Then, in Step S702, a variable j=1 is set, and Steps S703 to S707 are looped until the number of learning times j=M. Through the loop, the artifact correction parameter of the correction engine is updated.
In Step S703, as in Step S410, the reconstructed image in which the artifact is corrected is output from the sparse sampling coordinate group and the reconstructed image having the artifact, based on the artifact correction parameter of the correction engine.
In Step S704 subsequent thereto, the artifact occurrence degree of the reconstructed image generated in Step S703 during learning of the artifact correction parameter is evaluated. A method for evaluating the artifact occurrence degree may be the same as that in Step S403.
Then, in Step S705, based on an evaluation result of the artifact occurrence degree, an error between the reconstructed image in which the artifact is corrected and the image having no artifact is calculated as a correction error. For example, as the correction error, it is possible to use an error in which a mean square error of the artifact occurrence degrees is added to a mean square error of the reconstructed image in which the artifact is corrected and the image having no artifact. It may be assumed that the artifact occurrence degree of the image having no artifact is zero.
As described above, in the first modification example, during learning of the correction engine, the parameter of the correction engine is updated, based on the evaluation result of the artifact occurrence degree. In this manner, the correction engine can accurately learn the artifact reduction processing of reconstructed images.
First, in Steps S801 to S804, the same operations as those in Steps S401 to S404 in
When the artifact is absent in the reconstructed image, the reconstructed image is excluded from candidates of the learning data (Steps S805 and S806 are omitted, and the process proceeds to Step S807). On the other hand, when the artifact is present in the reconstructed image, in Step S805, the distribution of the artifact occurrence degrees of the reconstructed image is output as a GUI on the display 156, and the GUI prompts a user to change the distribution. When there is an error in the distribution of the artifact occurrence degrees, the user teaches a correct distribution. The distribution of the artifact occurrence degrees may be expressed as a binary value, or may be expressed as a multiple value.
In Step S806 subsequent thereto, the learning data set in which the e sampling coordinate group, the reconstructed image having the artifact, and the distribution of artifact occurrence degrees are used as one set is added. In Step S807, in a case of i=N, the loop in Steps S802 to S806 is completed. In a case of i<N, the process returns to Step S801 and the above-described procedure is repeated. When the N-number of learning candidate data sets are processed, the process proceeds to Step S808.
When the learning data set is saved in Step S808, the variable j=1 is set in Step S809, and Steps S810 to S813 are looped until the number of learning times j=M. Through this loop, learning of the evaluation engine that outputs the distribution of the artifact occurrence degrees is performed.
In Step S810, the distribution of the artifact occurrence degrees is output from the sparse sampling coordinate group and the reconstructed image having the artifact, based on an artifact occurrence degree evaluation parameter of the evaluation engine. In a case of j=1, the artifact occurrence degree evaluation parameter of the evaluation engine may be initialized with a random number, or may be initialized with a parameter learned by using another learning data set.
In Step S811, a difference between the distribution of the artifact occurrence degrees output in Step S810 and the distribution of the artifact occurrence degrees of the learning data set is calculated as a distribution error. For example, the distribution error can be calculated as a mean square error.
In Step S812, it is determined whether the distribution error is equal to or smaller than a predetermined threshold value. When the distribution error is equal to or smaller than the threshold value (YES), the process proceeds to Step S815, and when the distribution error is greater than the threshold value (NO), the process proceeds to Step S813. In Step S813, the artifact occurrence degree evaluation parameter is updated so that the distribution error becomes smaller. As a method for updating the parameter, for example, an error backpropagation method can be used. In Step S814, when j is the number of learning times M, the loop in Steps S810 to S813 is completed. In Step S815, the artifact occurrence degree evaluation parameter of the evaluation engine is saved in the storage unit 124.
As described above, in the second modification example, the evaluation engine trained to output the distribution of the artifact occurrence degrees is used. In this manner, it is possible to accurately evaluate the distribution of the artifact occurrence degrees which are less likely to be modeled by performing rule base processing.
As an example, a GUI screen 900 in
Here, in the sparse sampling coordinate groups 905, 907, and 910, sampling coordinates 906, 908, and 911 are simply displayed. The sparse sampling coordinate groups 905, 907, and 910 respectively indicate the sampling coordinate group for initial setting, the sampling coordinate group for additional setting, and the sampling coordinate group obtained by combining the initial setting and the additional setting.
A parameter setting window 901 is configured to be capable of setting a sampling rate 902 of the sampling coordinate group during the initial setting, an increase rate 903 from the initial setting in the additionally set sampling coordinate group, and a threshold value 904 of the artifact occurrence degree.
A display example in
In the second Step S305 after the sampling coordinate group is additionally set, the reconstructed image 214 having substantially no artifact may be displayed on the display 156. In addition, in the second Step S306, an image 909 obtained by mapping the artifact occurrence degree may be displayed. Here, since the artifact occurrence degree of the reconstructed image 214 is smaller than the threshold value 904, there is no region having the high artifact occurrence degree (image 909). In this manner, the reconstructed image and the distribution of the artifact occurrence degrees are displayed in combination. Therefore, while an allowable artifact occurrence degree can be confirmed, it becomes possible to smoothly set the sampling coordinate group to a minimum required level.
The scanning electron microscope device 100 which is an observation device according to a second embodiment will be described with reference to
An example of artifact correction processing in the device according to the second embodiment will be described with reference to a flowchart in
Steps S1005 to S1009 are the same processing as Steps S306 to S310, and detailed description will be omitted. However, in the second embodiment, in Step S1004, the correction engine trained to learn the process for generating the reconstructed image having substantially no artifact from the sparse pixel value group is used to generate the reconstructed image obtained from the sparse pixel value group obtained from the sampling coordinate group of the interest region. This step corresponds to generating the reconstructed image 214 having substantially no artifact from the sparse pixel value group obtained from the sparse sampling coordinate 205 set in the interest region 203. The correction engine is a network trained to learn to output the reconstructed image having no artifact by receiving the sampling coordinate group and the sparse pixel value group as inputs. For example, a convolutional neural network is used as the network of the correction engine.
An example of the learning procedure of the correction engine in the second embodiment will be described with reference to a flowchart in
In Step S1101, the variable i=1 is set, and Steps S1102 to S1104 are repeatedly performed (looped) until the number of learning data i=K. A learning data set is generated through the loop. When the number of samples to be imaged is defined as N1, the number of interest regions set for each sample is defined as N2, and a type of the sparse sampling coordinate group of the interest region is defined as N3, the number of learning data K is K=N1×N2×N3.
In Step S1102, the sparse sampling coordinate group and the sparse pixel value group are acquired by using the same procedure as that in Steps S301 to S303. In Step S1103, the interest region 203 is fully sampled to acquire an image having no artifact.
In Step S1104, the sparse sampling coordinate group, the sparse pixel value group, and the image having no artifact are added to the learning data set as one set of data. In Step S1105, in a case of i=K, the loop in Steps S1102 to S1104 is completed. In a case of i<K, the process returns to Step S1101, and the above-described procedure is repeated. In Step S1106, the learning data set is saved in the storage unit 124.
In Step S1107, the variable j=1 is set, and Steps S1108 to S1112 are repeatedly performed (looped) until the number of learning times i=M. Through the loop, learning of the correction engine is performed to generate the reconstructed image having no artifact. In Step S1108, the reconstructed image having substantially no artifact is output from the sparse sampling coordinate group and the sparse pixel value group, based on the reconstruction parameter of the correction engine. In a case of j=1, the reconstruction parameter of the correction engine may be initialized with a random number, or may be initialized with a parameter learned by using another learning data set.
In Step S1109, the artifact occurrence degree of the reconstructed image is evaluated during learning of the reconstruction parameter. A method for evaluating the artifact occurrence degree may be the same as that in Step S403.
In Step S1110, a difference between the reconstructed image and the image having no artifact is calculated as a reconstruction error, based on the artifact occurrence degree determined in Step S1109. As the reconstruction error, for example, it is possible to use an error in which a mean square error of the artifact occurrence degrees is added to a mean square error of the reconstructed image and the image having no artifact. It may be assumed that the artifact occurrence degree of the image having no artifact is zero.
In Step S1111, it is determined whether the reconstruction error is equal to or smaller than a predetermined threshold value. When the reconstruction error is equal to or smaller than the threshold value (YES), the process proceeds to Step S1114, and when the reconstruction error is greater than the threshold value (NO), the process proceeds to Step S1112. When the reconstruction error is equal to or smaller than the predetermined threshold value, the reconstruction parameter calculated in Step S1114 is saved, and the procedure in
In Step S1112, the artifact correction parameter is updated so that the reconstruction error becomes smaller. As a method for updating the parameter, for example, an error backpropagation method can be used. In Step S1113, in a case of j=M, the loop in Steps S1108 to S1112 is completed, and the process proceeds to Step S1114.
In the second embodiment, the reconstructed image having substantially no artifact is generated from the sparse pixel value group without using the reconstructed image having the artifact. Therefore, fewer processing steps are performed compared to the first embodiment, and usability can be improved.
With reference to
As an example, when the sample 106 is a semiconductor, the design data may be a file in which edge information of a design shape of a semiconductor circuit pattern is written as coordinate data. Specifically, it is design data written in a format such as GDSII and OASIS. Since the design data is utilized, pattern layout information can be obtained without actually imaging the sample 106.
An example of a procedure for performing the artifact correction processing in the third embodiment will be described with reference to a flowchart in
Next, in Step S1202, as in Step S301 (
For example, in the third embodiment, the reconstructed image is generated in such a manner that the sample is imaged in advance by using various sampling coordinate groups, and analysis results of the reconstructed images are stored in the storage unit 124 as a database. When the design data is read in Step S1201, an optimal sampling coordinate group is selected and set in accordance with the analysis results relating to the sample which is the same as or similar to the design data.
In setting the sampling coordinate group, based on the design data (Step S1203), for example, instead of setting a uniform sampling coordinate in an entire region, the sampling coordinate group may be set for the interest region by locally changing sampling density. In addition, even when data which coincides with the set interest region is not retrieved from the database, data similar to the interest region may be retrieved from data in the database, and the sampling coordinate group corresponding to the similar data may be set for the interest region.
The sampling coordinate group may be set for the sample relating to the design data by using an estimation engine trained to learn the processing, based on the database. For example, the estimation engine is a network trained to learn to output the sparse sampling coordinate group by receiving the design data as an input. On the GUI of the display 156, for example, read design data, design data referenced from the database, and candidates of the sampling coordinate group can be displayed to prompt a user to select the sampling coordinate group.
After the sampling coordinate group is set in Step S1203, subsequently in Step S1204, as in Step S303 in FIG. 3, the sampling coordinate group 205 set in the interest region 203 is imaged, and the corresponding sparse pixel value group is obtained. Then, in Step S1205, as in Step S304, the reconstructed image is generated from the obtained sparse pixel value group. In Step S1206 subsequent thereto, the artifact of the reconstructed image is corrected by using the correction engine trained to learn the artifact reduction processing, learning sequence of the correction engine will be described later.
In Step S1207, the artifact occurrence degree is evaluated, based on the design data. In Step S1208 subsequent thereto, it is determined whether the artifact in Step S1206 is sufficiently corrected. In a case of YES, the process proceeds to Step S1211; and in a case of NO, the process proceeds to Step S1209.
In Step S1209, the sparse sampling coordinate group is additionally set in the interest region. The specific method is the same as that in Step S308 in
In the third embodiment, since the sampling coordinate group is set, based on the design data, it is possible to set the optimum sampling coordinate group for the sample. Therefore, robustness of the image reconstruction processing and the artifact correction processing for the sample can be improved. In addition, since the artifact occurrence degree is evaluated, based on the design data, the design data can be used as an evaluation reference. Therefore, it is possible to accurately evaluate the artifact occurrence degree.
A specific example of the learning sequence of the correction engine used in Step S1206 will be described with reference to a flowchart in
Therefore, repeated description will be omitted below, and only portions different from those in the above-described embodiments will be described.
In Step S1303, the artifact occurrence degree is evaluated, based on the design data. As a method for evaluating the artifact occurrence degree, based on design data, for example, the following method can be adopted. A Sobel filter is used to generate edge images of the reconstructed image and the design data, and a region having a large difference between the edge images is evaluated as the region having the high artifact occurrence degree.
As another method, the following method can be adopted. A difference image between an image converted into image quality close to the captured image by using the design data and an image conversion engine, and the reconstructed image is generated, and a region having a great pixel value of the difference image is evaluated as the region having the high artifact occurrence degree. For example, the image conversion engine is a network trained to learn to output the captured image of the interest region by receiving the design data of the interest region as an input.
In Step S1306, the design data, the sparse sampling coordinate group, the reconstructed image having the artifact, and the image having no artifact are added to the learning data set as one set of data.
In Step S1311, the artifact occurrence degree of the reconstructed image in which the artifact is corrected during learning of the artifact correction parameter is evaluated, based on design data. The method for evaluating the artifact occurrence degree is the same as that in Step S1303.
An example of a specific procedure for aligning the sampling coordinate group, based on the design data in Step S1203 of the third embodiment will be described with reference to a flowchart in
In Step S1402 subsequent thereto, a position on the sample 106 corresponding to the sampling point of the sampling coordinate group for alignment is irradiated with the electron beam to acquire the sparse pixel value group. Then, in Step S1403, the reconstructed image is generated from the sparse pixel value group. The method for generating the reconstructed image may be the same as that in Step S304.
In Step S1404, a positional deviation amount of the sparse sampling coordinate group is calculated, based on the design data and the reconstructed image for alignment which is obtained in Step S1403. As a method for calculating the positional deviation amount, for example, the following method is adopted. A Sobel filter is used to generate edge images of the reconstructed image and the design data, and the positional deviation amount is calculated from a peak position in a normalized cross-correlation map of the edge images. Then, in Steps S1405 and S1406, the sampling coordinate group is corrected, based on the calculated positional deviation amount, and the corrected sampling coordinate group is set. In this way, in the interest region, the sampling coordinate group for generating the reconstructed image is aligned by using the sampling coordinate group for alignment. In this manner, the pixel value group can be obtained from coordinates optimized, based on the design data. Therefore, the image can be accurately reconstructed.
The present invention is not limited to the above-described embodiments, and includes various modification examples. For example, the above-described embodiments have been described in detail to facilitate understanding of the present invention, and the present invention is not necessarily limited to those which include all of the above-described configurations. In addition, a part of the configuration in one embodiment can be replaced with a configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. In addition, a part of the configuration of each embodiment can be added, deleted from, and replaced with other configurations. In addition, each configuration, function, processing unit, and processing means which are described above may be partially or entirely realized through hardware by designing an integrated circuit, for example.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2021/040161 | 10/29/2021 | WO |
