Patent Application
20250125116
This application claims priority from European application EP23203083.3, filed Oct. 11, 2023. The entire disclosure of application EP23203083.3 is incorporated herein by reference.
The disclosure relates to correcting aberrations in charged particle beam microscopy, specifically but not exclusively electron microscopy.
Transmission electron microscopy (TEM) is a technique in which a beam of electrons is transmitted through a thin sample to form an image. The image is formed by the interaction of the electrons with atoms of the sample as the electrons are transmitted through the sample. The image can be captured by a camera. TEM is capable of producing high resolution images due to the small wavelength of the transmitted electrons. Sufficiently high-resolution images can be used to investigate and analyse the structural details of the sample on an atomic scale. However, optical aberrations in the microscope can limit the resolution of the images.
The subject matter disclosed herein pertains to improved methods and systems for obtaining a sample reconstruction. In particular, the subject matter disclosed herein pertains to methods and systems for imaging in charged particle beam microscope systems (e.g., operating in a transmission imaging mode or a scanning imaging mode) while obtaining a model relating to aberrations in the system, which is used to reduce or at least partially correct for the aberrations. Examples of optical aberrations that can be corrected include, but are not limited to, defocus, two-fold astigmatism, three-fold astigmatism, four-fold astigmatism, five-fold astigmatism, sixfold astigmatism, axial coma, fifth order axial coma, spherical aberration, sixth order spherical aberration, star aberration, sixth order star aberration, three-lobe aberration, rosette aberration, linear distortion, pincushion distortion, barrel distortion, spiral distortion, and elliptical distortion. Specific embodiments of charged particle beam microscope systems relate to electron microscopy systems. The disclosure therefore includes aspects and embodiments as set out in which the charged particles are electrons.
Sample reconstruction may refer to methods to extract information from a plurality of acquired images and carry out a reconstruction. An example of such a reconstruction is a (3D) structure reconstruction, a 3D-model, an exit wave reconstruction, a 3D atomic resolution reconstruction, or a surface image (as in the case of the scanning imaging mode). The images may be acquired through any imaging workflow. Some such workflows acquire many images at different sample positions, such as, but not limited to, single particle analysis or electron tomography. It will be understood that the sample position refers to one or both of the axial position (e.g., the x/y/z coordinates) of the sample and the rotation/tilt of the sample.
In one or more implementations, a method of acquiring a plurality of charged particle microscope images for sample reconstruction comprises acquiring a plurality of charged particle microscope images for a plurality of areas of a sample using a charged particle beam microscope system, e.g., an electron microscope system. The method comprises acquiring a respective first set of aberration predictor values for each acquisition of the plurality of charged particle microscope images, for example a temperature of the optical system of the charged particle microscope, a position and/or configuration of the optical system of the charged particle microscope, and so forth. Periodically an aberration measurement and a corresponding second set of aberration predictor values are acquired while acquiring the plurality of charged particle microscope images. The method further comprises obtaining an aberration model using the second sets of aberration predictor values and aberration measurements, e.g., by training or updating the model with the second sets of aberration predictor values and aberration measurements, or by reading the model from a memory. The model takes a set of aberration predictor values as an input and outputs predicted aberration data. The model is applied to the first sets of aberration predictor values to obtain respective predicted aberration data, which is used to reduce or at least partially correct for aberration. The aberration data may be used to (at least partially) correct for aberration in the charged particle microscope image data, and/or may be used to reduce aberration by tuning the charged particle beam microscope system during image acquisition (thereby reducing aberration in the subsequently acquired images). In any case, a sample reconstruction is obtained using the acquired charged particle microscope images, the first sets of aberration predictor values, and the aberration model.
The aberration predictor values may comprise any system parameters on which aberrations are known to be dependent, or parameters from which an aberration determination may otherwise be predicted. For example, aberration predictor values may comprise a two-dimensional position reading of the electron beam, a time stamp, a temperature, a sensor measurement from a surface of the imaging system, the sample position reading (e.g., x/y/z position and/or sample tilt), or any other environmental or operational variables which may impact the imaging system or the optical state of the system.
The optical state may comprise the parameters defining the optical elements of a system, which may drift or otherwise fluctuate, e.g., due to the variables discussed above. Such parameters defining the optical elements of a system may include, for example, a two- or three-dimensional position of the beam, a de (focus) or magnification of the beam, a state of magnetic or electrostatic deflectors, a state of one or more magnetic or electrostatic lens(es) The optical state of the system can change an unknown amount during a data acquisition session. However, the optical state can be tuned by adjusting the settings of magnetic or electrostatic deflectors, and the settings of the magnetic or electrostatic lens(es), e.g., by varying supply currents or voltages. Furthermore, the images acquired from the microscope column can be corrected, accounting for the changing optical state of the system. By obtaining an aberration model, the workflow of an imaging or data acquisition session is rendered more robust against less than perfect circumstances, such as changes of the parameters defined above. In particular, the parameter changes and their consequent aberrations may be more closely ‘followed’ over the data acquisition session without requiring additional aberration measurements, resulting in, for example, higher resolutions and/or more accurate 3D-models of the sample.
The aberration model may be any of various suitable classes of model. In some examples, the aberration model may be a parametric model with key system parameters as input. In some examples, the model may be a physical model, a statistical model, a linear model, a polynomial model, a decision tree, a random forest, a neural network, or a machine learning model.
The aberration model can also be described with reference to its inputs and outputs. In particular, a set of aberration predictor values is input into the aberration model. More specifically, the aberration model may take any system parameters upon which aberrations are known to be dependent as an input, the system parameters being acquired for each of the plurality of charged particle microscope images. Predicted aberration data is then output for the aberration model. It will be appreciated that the predicted aberration data may define the predicted aberration(s), or may define a correction or system adjustment for potential aberration(s). In particular, the predicted aberration data may include parameters specifying the type and numerical properties of one or more aberrations that may be present. The properties may comprise at least one of a magnitude or a direction of one or more aberrations, such as coma, or a set of values indicating the aberration properties of a set of identified aberrations, for example coma and spherical aberration, or any other form of data which may characterize one or more aberrations so that the characterized aberrations can be reduced or corrected for, e.g., by applying a correction transformation to the respective data acquisition and/or applying a settings adjustment to the microscope system based on the predicted aberration data in a subsequent step. In particular, the output predicted aberration data of the model comprises parameters specifying an image correction or a settings adjustment for the microscope system, and thus the output of the aberration model may be used directly to specify an operation that reduces or at least partially corrects for predicted distortive effects of aberrations. The correction or system adjustment can be applied as appropriate (e.g., a correction to the respective charged particle microscope image, or an adjustment to the microscope system) to reduce aberration prior to obtaining a sample reconstruction.
In one or more examples, the respective predicted aberration data comprises one or more aberration corrections for applying to the charged particle microscope image. In such examples, the respective predicted aberration data is used to correct for aberration(s) in the charged particle microscope image of each acquisition, and the corrected charged particle microscope images are used in sample reconstruction.
In some examples, the respective predicted aberration data may characterize one or more aberrations, such as, for example, defocus, astigmatism, coma, spherical, star, three-lobe, rosette, linear distortion, pincushion distortion, barrel distortion, spiral distortion, elliptical distortion or other aberrations. In such examples, correcting the charged particle microscope image may comprise computing an aberration correction for the charged particle microscope image from the respective predicted aberration data and correcting the charged particle microscope image using the computed aberration correction.
In some examples, the respective predicted aberration data may characterize one or more aberration corrections which can be used to correct for aberrations in the charged particle microscope images for sample reconstruction. In such examples, the predicted aberration data may comprise parameters specifying a transformation that can be used to correct for aberration in an image, for example including a scaling, rotation and/or skewing operation, or a Fourier filter. For example, the transformation can be a distortion field that corrects for the aberration. It will be appreciated that determining a correction (of any form) for a known aberration can be achieved using numerical methods and/or algorithms known in the art, such as those reviewed by Amit Singer et al [URL: https://arxiv.org/abs/2003.13828], which is incorporated herein by reference. Thus, in this example, using the output(s) of the aberration model, the distortive effects of aberrations can be corrected following data-acquisition in a post-processing step. By correcting for aberrations in post-processing, there is a reduced need to acutely monitor the imaging systems to correct for aberrations during data acquisition. Nevertheless, the image corrections may also be determined on-the-fly on the microscope during image acquisition. In particular, one or more model fitting or image correction steps as described herein may be carried out while data acquisition is ongoing, e.g., in order to limit processing latency.
In one or more examples, once predicted aberration data is obtained, it is determined whether the respective predicted aberration data meets a condition indicating that the charged particle microscope system needs to be tuned or indicating that aberrations are present which need to be reduced. In some examples, if the condition, which may be a predetermined threshold of a value of an aberration or of an adjustment, is met, the charged particle microscope system is tuned to reduce aberration.
In some examples, the predicted aberration data may comprise parameters specifying adjustments to the settings of the optical system, such as an adjustment of the electrical current applied to one or more components of the optical system, to tune components of the optical system and mitigate one or more potential aberrations during image acquisition. In other examples, the predicted aberration data characterises one or more aberrations such as, for example, defocus, astigmatism, coma, spherical, star, three-lobe, rosette, linear distortion, pincushion distortion, barrel distortion, spiral distortion, elliptical distortion or other aberrations. In such examples, the charged particle beam microscope system can be tuned by computing one or more adjustments for the charged particle beam microscope system from the predicted aberration data and applying the adjustment(s) to the necessary components of charged particle beam microscope system. It will be appreciated that, in the case of large inferred aberration values, it may be better to apply adjustments to the optical elements of the microscope to reduce aberration on-the-fly during acquisition. In the case of smaller aberrations, corrections may be more easily applied to the images, e.g., to correct for aberration in post-processing. These techniques may thus be used in combination, for example, if the condition indicating that the microscope system needs to be tuned is not met, the predicted aberration data may still indicate that a (small) aberration is present which can be corrected for with a correction to the corresponding acquired image/data acquisition.
In some examples, the method comprises, while acquiring images, periodically inferring aberration data and determining if the adjustment condition is met, and, subsequent to tuning the charged particle beam microscope system, continuing to acquire images. For example, the periodic aberration data inference can be every acquisition, after every n acquisitions, or according to a rule (e.g., the periodic acquisition may be triggered by sensors on the microscope system). After the periodic acquisition, the images may continue to be acquired but now using the adjusted microscope system, i.e., with reduced aberration following the adjustment.
In one or more examples, the predicted aberration data may characterize or correct for a coma aberration, or for a plurality of aberrations, including coma, spherical, or otherwise.
In some examples, the aberration model may comprise a plurality of models for various aberrations. For example, the aberration model may comprise a plurality of separate models to characterize each of a plurality of aberrations, e.g., one model per aberration. In other examples, the aberration model may be a single model with one or more outputs for the respective one or more aberrations. In this way, models for different aberrations may be computed separately and applied one by one, e.g., with sequential aberration correction and/or sample reconstruction, or a joint multi-output model for a plurality of aberrations may computed and applied for a broad aberration mitigation and/or sample reconstruction.
In some examples, the aberration predictor values comprise any or more of the following: one or more temperature values, the temperature values corresponding to a respective one or more temperature sensors; a time stamp of the acquisition; an optical state of the charged particle beam microscope system; a measurable physical state of the charged particle beam microscope system; a sensor measurement from a surface of the imaging system; a sample position reading; and/or a position reading of optics of the charged particle microscope. By training a model on periodically acquired aberration predictor values comprising the above, the imaging process is rendered more robust to temperature or other system changes over time, thereby allowing for a higher resolution sample reconstruction to be achieved consistently. Furthermore, there is a reduced need for acute monitoring and correcting of the imaging system during data acquisition.
In one or more examples, periodically acquiring an aberration measurement can comprise obtaining an aberration measurement for each charged particle microscope image. For example, the aberration measurement acquisition may be carried out following each exposure. In this way, the first part of the exposure is used for the acquisition of the charged particle microscope image, which is beneficial for samples which may be radiation sensitive.
Obtaining an aberration measurement for each charged particle microscope image may comprise obtaining one or more aberration measurements with a low precision (e.g., with significant noise on the estimate of the aberration value). By fitting and applying a model when taking low precision measurements with each exposure, the noise in the measurements can be effectively averaged out, and hence the low precision measurements can be ‘denoised’.
In one or more examples, an aberration measurement may be obtained periodically during image acquisition, for example between 1 to 200 image acquisitions per aberration measurement, such as an aberration measurement every 20 image acquisitions.
The aberration measurement used for training the model may comprise one or more of the following: one or more images, acquired while changing one or more microscope parameters; one or more images which can be processed to obtain one or more aberration parameters; and/or a parameter characterizing aberration. For example, one or more images may be acquired while changing beam tilt. In such examples, the one or more images (or the shifts between them) may be indicative of the presence of one or more aberration and can therefore be processed to obtain one or more aberration parameters.
Parameters characterising aberration are known in the art and may comprise, for example, Thon ring analysis. Such aberration-characterising parameters and methods are discussed for example in Barthel J, Thust A “Aberration measurement in HRTEM: implementation and diagnostic use of numerical procedures for the highly precise recognition of diffractogram patterns”, Ultramicroscopy, Volume 111, Issue 1, 2010, Pages 27-46, which is incorporated herein by reference.
A parameter characterizing aberration may also comprise image shifts across the acquired plurality of images. In a specific example, a test pattern is applied to beam-tilt coils in the microscope column and the resulting beam tilt leads to aberration-induced image shifts across the acquired plurality of images, for example as discussed in EP 4235731 A1. In this specific example, while operating the charged particle beam microscope system in a transmission imaging mode and directing a charged particle beam to a sample, a time series of beam tilts is applied in a pattern to the charged particle beam to produce a sequence of images. At least some of the images in the sequence of images are captured while the charged particle beam is transitioning between one beam tilt in the time series of beam tilts and a sequentially adjacent beam tilt in the time series of beam tilts. The sequence of images may be analysed according to the methods disclosed herein to obtain a sample reconstruction. In this specific example, the image of a test pattern refers to a pattern which is configured such that the beam tilts induce image changes (e.g., image shifts) in the sequence of images that can be correlated to optical aberrations in the microscope system. For example, image changes can be determined from the sequence of images and used to estimate one or more optical aberration values for the microscope system, and/or the one or more optical aberration values can be determined by a trained neural network that receives the sequence of images or information derived from the sequence of images (e.g., power spectra of the images) as input data. In this specific example of an aberration measurement, the beam tilt pattern is a periodic pattern, such as a Lissajous figure. The use of periodic patterns can make it possible to determine an unknown time delay and unknown deviation in frequency of the applied beam tilt pattern in relation to the measured image changes over time. The periodic patterns can also enable a system that is robust against other effects that can induce image changes (e.g., specimen drift and mechanical vibrations), allowing signal processing to be used to separate the different signal sources.
In some examples, the sample reconstruction comprises a Single Particle Analysis (SPA) reconstruction and/or a three-dimensional reconstruction. The sample reconstruction may comprise or be based on a charged particle microscope acquisition in which many images are acquired at different sample positions. For example, the sample reconstruction may be for electron tomography reconstruction.
In some implementations, a charged particle beam microscope system may comprise one or more memories which store computer-readable instructions configured to cause one or more processors to perform operations comprising methods as described above.
In some implementations, a computer-readable medium comprises instructions to be executed by one or more data processing apparatuses, which, when executed cause the respective one or more apparatuses to perform operations comprising methods as described above.
In one or more implementations, a method of training an aberration model for reducing or at least partially correcting aberration in a sample reconstruction comprises accessing previously acquired pairs of sets of aberration predictor values and corresponding aberration measurements and using the pairs to train the model. The obtained aberration model takes a set of aberration predictor values as an input and outputs predicted aberration data. In some examples, a reconstruction or improved resolution of an imaged sample may be obtained using the aberration model to reduce or at least partially correct for aberration in charged particle microscope images. Training the model comprises providing the acquired set of predictor values as inputs to the aberration model, obtaining outputs of the aberration model in response to the set of predictor values, and comparing the outputs of the model with the corresponding known aberration measurements for each input set of predictor values. The input aberration predictor values comprise: one or more temperature values (each temperature value corresponding to respective one or more sensors or to different acquisition times); a time of obtaining the aberration measurement; a sample position; an optical state of the charged particle beam microscope system; a measurable physical state of the charged particle beam microscope system; and/or a position of optics of the charged particle beam microscope system. The parameters of the model are then updated and/or tuned to reduce the error between the output values and the known aberration data. Training may comprise using the aberration measurements to train the model, such that the predicted aberration data characterises the aberration, or, alternatively, training may comprise determining parameters defining a correction at least partially correcting for aberration in an image or an adjustment to the charged particle beam microscope system reducing aberrations corresponding to the aberration measurement, and using the determined parameters to train the model, such that the predicted aberration data characterises the correction and/or the adjustment. Various methods of training the model are known in the art and suitable for the methods disclosed herein. For example, the model may be a deep learning model, such as an artificial neural network (ANN). ANN can be trained using gradient descent, stochastic gradient descent or other optimization algorithms like ADAM, as is well known in the field of ANN. The model may be any other parametric model, for example a physical model of aberration, a polynomial model, etc, with the parameters updated to reduce the error using any suitable optimization routine, including those mentioned above.
In one or more implementations, a method of acquiring a plurality of charged particle microscope images for sample reconstruction comprises acquiring a plurality of charged particle microscope images for a plurality of areas of a sample. The method comprises acquiring a respective first set of aberration predictor values for each of the plurality of charged particle microscope images, for example a temperature of the optical system of the charged particle beam microscope system, a position and/or configuration of the optical system of the charged particle beam microscope system, and so forth. A sample reconstruction is obtained using the acquired charged particle microscope images. In these implementations, a model is pre-trained and/or a pre-trained model is used for image reconstruction and/or charged particle microscope system adjustment, without necessarily obtaining or updating the model using the images acquired for the reconstruction. In this way, obtaining the aberration model may merely comprise reading a trained aberration model from a memory. The examples described above are equally applicable to these implementations in an analogous manner.
The subject matter is described with implementations and examples. In some cases, as will be recognized by one skilled in the art, the disclosed implementations and examples may be practiced without one or more of the disclosed specific details, or may be practiced with other methods, structures, and materials not specifically disclosed herein. All the implementations and examples described herein and shown in the drawings may be combined without any restrictions to form any number of combinations, unless the context clearly dictates otherwise, such as if the proposed combination involves elements that are incompatible or mutually exclusive. The sequential order of the acts in any process described herein may be rearranged, unless the context clearly dictates otherwise, such as if one act requires the result of another act as input.
In the interest of conciseness, and for the sake of continuity in the description, same or similar reference characters may be used for same or similar elements in different figures, and description of an element in one figure will be deemed to carry over when the element appears in other figures with the same or similar reference character. In some cases, the term “corresponding to” may be used to describe correspondence between elements of different figures. In an example usage, when an element in a first figure is described as corresponding to another element in a second figure, the element in the first figure is deemed to have the characteristics of the other element in the second figure, and vice versa, unless stated otherwise.
The word “comprise” and derivatives thereof, such as “comprises” and “comprising”, are to be construed in an open, inclusive sense, that is, as “including, but not limited to”. The singular forms “a”, “an”, “at least one”, and “the” include plural referents, unless the context dictates otherwise. The term “and/or”, when used between the last two elements of a list of elements, means any one or more of the listed elements. The term “or” is generally employed in its broadest sense, that is, as meaning “and/or”, unless the context clearly dictates otherwise. When used to describe a range of dimensions, the phrase “between X and Y” represents a range that includes X and Y. As used herein, an “apparatus” may refer to any individual device, collection of devices, part of a device, or collections of parts of devices.
In the illustrated example, the charged particle beam microscope system 104 includes a charged particle beam microscope column 120 (hereafter, microscope column 120) enclosed in a vacuum chamber 106. The microscope column 120 defines an optical axis 116 of the charged particle beam microscope system 104.
A sample 124 to be investigated can be disposed along the optical axis 116. The sample 124 can be supported by a sample holder 140, which in some examples can have capabilities to translate, rotate, and/or tilt the sample 124 relative to the optical axis 116. The sample holder 140 can allow different areas of the sample 124 to be positioned relative to the optical axis 116 for investigation.
At the top of the microscope column 120 is a charged particle source 108 (e.g., a thermal electron source, Schottky-emission source, field emission source, etc.) that emits a charged particle beam 112 (e.g., an electron beam). The charged particle beam 112 is directed along the optical axis 116 such that it can impinge on the sample 124 and interact with the sample 124. The charged particle beam 112 passes through the sample 124 to produce an image.
A camera 136 at the bottom of the microscope column 120 can be operated to capture an image of the sample 124. The camera 136 can include, for example, a CCD (charged-coupled device) imaging sensor, a CMOS (complementary metal-oxide-semiconductor) imaging sensor, or, more generally, an array of photodetectors. In one example, the camera 136 can be operable in a “movie” mode to capture a sequence of images of the sample.
The microscope column 120 can include various charged particle beam lens systems (hereafter, lens systems) that direct the charged particle beam 112 along the optical axis 116 and configure the charged particle beam 112 to provide a constant illumination covering a desired field of view on the sample 124. In some examples, the various lens systems can also project the image produced by interaction of the charged particle beam 112 with the sample 124 into a field of view of the camera 136.
In some examples, the various lens systems can include a condenser lens system 122, an objective lens system 126, and a projector lens system 130, which can have any suitable configuration to modify the charged particle beam 112 to provide a desired field of view. The condenser lens system 122 can be configured to condense the charged particle beam 112 into a nearly parallel beam. The objective lens system 126 can be configured to focus the image formed by transmitting the charged particle beam 112 through the sample 124. The projector lens system 130 can be configured to project the image coming from the objective lens system 126 to the field of view of the camera 136.
In some examples, the condenser lens system 122 can include condenser lenses 122a, 122b and other components, such as a condenser stigmator 122c (i.e., a device to correct axial astigmatism), and a condenser aperture 122d.
In some examples, the objective lens system 126 can include an objective lens 126a and other components, such as beam deflectors 126b (which can be used to centre the beam), an objective stigmator 126c (i.e., a device to correct axial astigmatism), and objective aperture 126d. In the illustrated example, the sample 124 is positioned above the objective lens 126a (in other examples, the sample 124 can be inserted into the objective lens 126a).
In some examples, the projector lens system 130 can include, for example, an intermediate lens 130a (e.g., to magnify the image) and projector lenses 130b, 130c. The projector lens system 130 can include other components, such as a stigmator, image deflector, and aperture.
The components of the lens systems can be electro-optical components that are tuneable by varying the electrical current applied to the components. For example, the various lenses of the lens systems 122, 126, 130 can be electromagnetic lenses. An electromagnetic lens modifies charged particle beams by a magnetic field generated by a solenoid. By changing the electric current to the solenoid, the magnetic field can be changed, which would correspondingly change the focal length and magnification of the lens. The other components of the lens systems 122, 126, 130, such as the stigmators and beam deflectors, can be electromagnetic devices, which can be tuned in the same manner as described for the electromagnetic lenses. The ability to tune the components of the lens systems 122, 126, 130 by varying electrical current supplied to the components can allow aberration corrections to be applied to the microscope column 120 on-the-fly (e.g., while the charged particle beam microscope system 104 is operating in a transmission imaging mode).
The lens systems 122, 126, 130 can include other optical components not shown in
In some specific examples relating to measuring aberrations with an image of a test pattern, the system 100 may include a pattern generator 144 that can generate various patterns. The pattern generator 144 can be, for example, a programmable function generator. In some examples, the pattern generator 144 can be configured to generate periodic patterns. In one example, the periodic pattern can be a Lissajous figure. Lissajous figure is given by the following parametric equations:
The Lissajous figure can be as simple as a circle (e.g., the parametric equations (1) and (2) give a circle when a/b=1, A=B, and d=p/2 radians) or can be more complex (e.g., the ratio a/b is not equal to 1).
In such examples comprising test patterns, the system 100 may include beam deflectors 134 inserted in the microscope column 120. The beam deflectors 134 can be controlled according to the pattern to tilt the charged particle beam 112 in the X and Y directions over time. The beam deflectors 134 can apply tilt motion to the charged particle beam 112 prior to the charged particle beam 112 reaching the sample 124 (e.g., prior to condensing the charged particle beam 112). The beam deflectors 134 can be electro-optic deflectors (e.g., electrostatic or magnetic beam deflectors), which would allow the tilt angles applied to the charged particle beam 112 to be changed by varying the electrical current to the beam deflectors.
The beam deflectors 134, operated according to a pattern, apply a time series of beam tilts to the charged particle beam 112. Each beam tilt produces an image. Thus, the time series of beam tilts results in a sequence of images. Image shifts between successive (or adjacent) images in the sequence of images can indicate optical aberrations in the microscope system.
The system 100 can include a frame grabber 148 in communication with the camera 136. The frame grabber 148 can, for example, form an interface between the camera 136 and a computing system 154 (e.g., a server). The frame grabber 148 is an electronic device that can capture individual image frames from a stream of image data produced by the camera 136.
In such examples relating to image shift methods of acquiring an aberration measurement, the system 100 can include an image shift tracker 152 in communication with the frame grabber 148. The image shift tracker 152 can receive image frames from the frame grabber 148 and measure (or compute) the image shift between successive image frames over time. The image shift tracker 152 can use any suitable imaging processing technique to measure images shift. One example technique is cross-correlation. In one example, the image shift tracker 152 can be executed by a processor (e.g., GPU, CPU, or FPGA) of the computing device 154.
The computing system 154 also acquires or otherwise receives aberration predictor values corresponding to the image frames received from the frame grabber 148. The aberration predictor values are preferably received with each of the plurality of images acquired by the frame grabber and may refer to any parameters on which aberrations may be dependent or parameters from which an aberration determination may otherwise be predicted. For example, aberration predictor values may refer to a two-dimensional position reading of an image beam shift, a time stamp, a temperature, a sensor measurement from a surface of the imaging system, or any other environmental or operational variables which may impact the imaging system or the optical state. Any of these key parameters are inputted to the computing system 154 alongside their respective received image frames.
The system 100 can include an aberration calculator 156 that determines values for the optical aberrations in a charged particle beam microscope system. The optical aberration may be determined, for example, from Thon ring analysis or from the measured image shifts from a series of images acquired while varying the beam tilt using a pattern; from one or more parameters characterizing aberrations; or from one or more images which can be processed to obtain aberration parameters. Examples of optical aberrations that can be found in charged particle beam microscopes and measured are defocus, two-fold astigmatism, three-fold astigmatism, four-fold astigmatism, five-fold astigmatism, sixfold astigmatism, axial coma, fifth order axial coma, spherical aberration, sixth order spherical aberration, star aberration, sixth order star aberration, three-lobe aberration, rosette aberration, linear distortion, pincushion distortion, barrel distortion, spiral distortion, and elliptical distortion. The aberration calculator 156 can be configured to determine values for any combination of these optical aberrations as well as other optical aberrations not specifically mentioned.
In one example, the aberration calculator 156 can receive a data acquisition or charged particle microscope image from the computing system 154 and measure the optical aberration in the microscope column 120 based thereon. In one example, the aberration calculator 156 can estimate the optical aberrations from an image shift tracker 152 which fits image shifts to an optical aberration function that provides a relationship between parameters of an image shift, e.g., between test pattern image shifts, beam tilts, and optical aberrations in the microscope column 120. In some examples, the optical aberration function can be extended with one or more auxiliary parameters that are also fitted.
In one example, the aberration calculator 156 can receive the image data directly from the camera 136 or receive image frames from the frame grabber 148. The aberration calculator can process the image data to measure aberrations using any of the multitude of techniques known in the art, for example, J. Barthel, A. Thust, Aberration measurement in HRTEM: Implementation and diagnostic use of numerical procedures for the highly precise recognition of diffractogram patterns, Ultramicroscopy, Volume 111, Issue 1, 2010, Pages 27-46.
The aberration calculator 156 fits an aberration model for predicting aberration data, according to the methods described herein. With reference to
The system 100 can include an aberration corrector 168 that is communicatively coupled to the aberration calculator 156. The aberration corrector 168 takes the predicted aberration data, corrects the received images and/or adjusts the instrumentation within optical system, and obtains a sample reconstruction from the (corrected) images. For example, the predicted aberration data may specify the correction (e.g., image data correction and/or system adjustment) and the aberration corrector 168 applies that correction and/or system adjustment. Alternatively, the predicted aberration data may indicate or characterise one or more predicted aberrations, the aberration corrector 168 receives the predicted aberration data from the aberration calculator 156 and determines a correction or adjustment for the relevant one or more aberrations therefrom. Once the correction or adjustment is determined, the aberration corrector 168 applies the correction to the respective image data or to the optical system (i.e., to the microscope column). For example, the aberration corrector 168 can apply adjustments to the components of the microscope column 120 (e.g., the components of the lens systems) based on the predicted aberration data determined by the aberration calculator 156 to reduce aberrations. In particular, where the components are electro-optical components, the aberration corrector 168 can change the electrical current applied to the components in order to change how the components modify the charged particle beam 112. In some examples, the aberration corrector 168 can apply the adjustments to the components of the microscope column 120 based on predetermined conditions (e.g., if the optical aberrations exceed predetermined thresholds or if the confidence of an estimated aberration value exceeds a predetermined threshold (e.g., if the absolute value of an estimated aberration value is less than a confidence interval for that optical aberration)).
In some examples, the aberration calculator 156 and aberration corrector 168 can be programs or scripts running on the computing system 154. The aberration corrector 168 can be in communication with the aberration calculator 156. In some examples, the computing system 154 can be coupled to a display 164 (e.g., a console of the charged particle beam microscope system 104). In some examples, the computing system 154 can present current aberration values and trends in the aberrations on the display 164.
The aberration calculator 156 and aberration corrector 168 can be implemented in software and/or hardware. Software components of the aberration calculator 156 and aberration corrector 168 can be stored in one or more computer-readable storage media or computer-readable storage devices and executed by one or more processors (e.g., processors in the computing system 154). Any of the computer-readable media herein can be non-transitory (e.g., volatile memory such as DRAM or SRAM, non-volatile memory such as magnetic storage, optical storage or the like) and/or tangible. It will be appreciated that the system 100 can include one or more computing systems. For example, the different programs or scripts of the methods disclosed herein (e.g., the aberration calculator 156 and corrector 168 of computing system 154) may be shared across a plurality of computing systems (or servers) communicatively coupled to one another to individually carry out the different programs or scripts.
At 210, the method includes operating a charged particle beam microscope system in an imaging mode (e.g., transmission imaging). The operation 210 can include generating a charged particle beam (e.g., an electron beam) using a charged particle source at the top of the microscope column and directing the charged particle beam to a sample using optical components in the microscope column. The charged particle beam interacts with the sample to produce an image. The operation 210 can include projecting the image into a field of view of a camera at the bottom of the microscope column.
At 220, with the charged particle beam microscope system operating in the transmission imaging mode as described in operation 210, the method can include applying a time series of beam tilts in a pattern to the charged particle beam coming out of the charged particle source. The operation 220 can include generating the pattern. The operation 220 can include applying the time series of beam tilts by traversing the pattern. At each point on the traversal path, the x, y coordinates of the point determine the amount of tilt to apply to the charged particle beam in the X and Y directions. The tilt amounts can be used to control the beam deflectors inserted in microscope column to tilt the beam. In some examples, the pattern can be a periodic pattern. In some examples, the periodic pattern can be a Lissajous figure (or Lissajous curve).
At 225, while applying beam tilts to the charged particle beam as described in operation 220, the method can include operating a camera in a movie mode to capture images of the sample. For every beam tilt, the camera can obtain a full image of the sample. Thus, the camera can capture a sequence of images corresponding to applying the time series of beam tilts to the charged particle beam. The time between applying the beam tilt and the actual tilting of the beam can vary such that there is an unknown time delay between the beam tilting pattern and the sequence of images captured. In some examples, the optical aberration function can incorporate the unknown time delay as an auxiliary variable to be fitted. Other auxiliary variables can be incorporated in the optical aberration function as well, e.g., beam tilt pattern or sample drift.
At 230, the method can include acquiring aberration measurements for the optical aberrations in the microscope column from the images captured in operation 225. In one example, the optical aberrations can be estimated from an optical aberration function that relates image shifts to beam tilts and optical aberrations of the microscope column. In another example, the optical aberrations can be obtained by optimizing a function that provides a relationship between the sequence of images and optical aberrations in the microscope column. In another example, the optical aberrations can be obtained as an inference of a trained neural network. The inference can include the prediction of the neural network as well as other information such as confidence of the prediction.
Equation (3) below is an example of an optical aberration function. For illustrative purposes, Equation (3) can obtain values for defocus, astigmatism (astigX, astigY), coma (comaX, comaY), threefold astigmatism (threefoldX, threefoldY), spherical aberration (spherical), star aberration (starX, starY), and fourfold astigmatism (fourfoldX, fourfoldY).
The vector on the right-hand side of the equation contains values for optical aberrations for a given state of the optical system of the microscope system at a certain moment in time. When the tilt of the beam is adjusted with (tx, ty) radians, then the induced image shift will be given by (shiftX, shiftY), which can be measured from images captured while applying the beam tilts (e.g., in operation 240). The measured shiftX, shiftY and known tx, ty can be fitted into the function of Equation (3) to obtain the estimated values for the optical aberrations. The parameters tx, ty are time-dependent, which allows unknown time delay between the applied beam tilt pattern and the induced image shift pattern to be factored into the function for fitting. Variations of the optical aberration function can incorporate other auxiliary parameters, such as beam tilt pattern and/or sample drift.
The operations 220-230 can be repeated each time an aberration measurement is due during a planned acquisition or each time predetermined criteria for an aberration measurement are met during a planned acquisition. The aberration measurements over time can be displayed on a screen and monitored.
At step 302, the method comprises assessing if a new aberration acquisition is needed, i.e., if criteria for a new aberration measurement is met. The criteria for a new aberration measurement may comprise, for example: when another measurement is scheduled (such as a defocus measurement), when the last measurement is a specified amount of time ago, when more information on aberrations in another area of the two-dimensional optical axis is acquired, or if a change in one or more environmental or operational variables are detected, e.g., if a temperature change is detected, if a change in sample position (e.g., a change in relative height of the sample) is detected, or otherwise. It will be understood that step 302 may also refer only to a singular criterion. The new aberration acquisition may comprise an image of a test pattern with distortion indicative of aberration (e.g., using the example method illustrated in
If it is assessed that a new aberration acquisition is not needed, the method proceeds to step 303. At step 303, the method comprises acquiring a data acquisition for use in reconstructing an image or model of the particle and acquiring one or more aberration predictor values. For example, the data acquisition can comprise a low-dose high-resolution image acquisition at a selected area of the sample and a predictor value can comprise a temperature measurement. The one or more predictor values are stored in association with the data acquisition (for example in the metadata of the data acquisition) in data acquisition storage 307.
If a new aberration acquisition is due, the method progresses to step 304, which comprises acquiring an aberration acquisition for measuring one or more aberrations and acquiring one or more associated predictor value(s) for the aberration(s). Both acquisitions are stored in aberration acquisition storage 306. Optical aberrations can be measured using a relatively small amount of data (e.g., defocus), or a relatively large amount of data may be required for appropriate measurement (e.g., coma and astigmatism). The one or more predictor values are stored in association with the aberration acquisition (for example in the metadata of the aberration acquisition) in data acquisition storage 306. It will be appreciated that the metadata of the data acquisition may also comprise other values that are not necessarily relevant to the aberrations but may be otherwise relevant for good sample reconstruction, such as a high-tension value, a pixel size, or other system state parameters.
The aberration predictor values, associated and stored with the acquisitions of steps 303 and 304 respectively, refer to any parameters on which aberrations might depend or parameters from which an aberration may be predicted. For example, aberration predictor values may refer to a two-dimensional position reading of an image beam shift, a time stamp, a temperature, a sensor measurement from a surface of the imaging system, a sample position, or any other environmental or operational variables which may impact the imaging system or the optical state.
Steps 303 and 304 may happen simultaneously or sequentially. For example, if the criteria are met for a new aberration measurement, a new data acquisition may be made alongside an aberration determination acquisition, for example using one and the same exposure. In some examples, the aberration measurement at step 304 may be carried out following a data acquisition at step 303. In this way, the first part of the exposure is used for the acquisition of the charged particle microscope image, which is beneficial for samples which are radiation sensitive.
Once a particular area of the sample has been visited and the method progresses to step 308. The method 308 comprises a check of whether all areas of interest of the sample have been visited. If “no”, the method returns to method step 301, i.e., moving to a next area of interest on the sample and repeating the workflow as described above. If all areas of interest of the sample have been reach, the sample acquisition run is ended.
At step 406, a model is fit, using the pairs of aberration measurements and corresponding aberration predictor values, to predict aberration measurements from the aberration predictor values. The method includes initialising a model, which may be, for example, a parametric model, a physical model, a statistical model, a linear model, a polynomial model, a decision tree, a random forest, a neural network, or a machine learning model. The aberration model takes a set of aberration predictor values as an input and outputs a predicted aberration measurement. The model is trained by obtaining predicted aberration measurements from aberration predictor values, comparing the predicted measurements to the actual aberration measurements, and adjusting parameters of the model to reduce a discrepancy between the predicted measurements and the actual measurements. Various methods of training the model are known in the art and may be used to implement the methods disclosed herein. For example, the parameters of the model may be trained using suitable optimization techniques, such as gradient descent.
Once the model is fitted, the data acquisition storage 307 is accessed at step 407 and the acquired predictor values during an acquisition input into the model for inference to obtain aberration measurements for each acquisition. For example, the aberration model may infer that there is a 10 μm coma aberration present from a given set of aberration predictor values. At step 408, corrections for the aberration measurements output by the model are determined respectively for each data acquisition. Determining a correction for a given aberration measurement is known in the art, and may comprise, for example, a Fourier filter to correct for phase flips and/or phase changes caused by the predicted aberration measurements, a transformation or a distortion field which corrects for distortions caused by the predicted aberration measurements, or a tuning adjustment to the microscope column as will be described in more detail below with reference to
At step 404, the data acquisition storage 307 is accessed and the set of predictor values for each reconstruction acquisition are input into the model for inference to obtain respective aberration corrections. The inferred corrections are applied to the respective data acquisitions and the corrected reconstruction acquisitions are used to reconstruct an image or a model of the particle in a final step 410.
The workflows of
The steps of workflow of
In some examples, the workflows of
Step 502 may be the same or similar to steps 401 and 405 of
At step 506, each data acquisition is input into the model for inference to obtain predicted data about aberrations present in the data acquisition. At step 508, it is determined whether the predicted aberration data output by the model for each data acquisition meets certain criteria or conditions indicating that the charged particle beam microscope system needs to be tuned or that aberrations need to be reduced. For example, the predicted aberration data may comprise an estimated quantification of a type of aberration present in the data acquisition. Alternatively, the predicted aberration data may define an adjustment value (e.g., a voltage or current change) to be applied to one or more elements of the charged particle beam microscope system to reduce aberration. The criteria may comprise determining whether the predicted aberration data is within a confidence interval for the predicted aberration and/or whether the absolute value of the predicted aberration or absolute value of the adjustment value exceeds a predetermined threshold. The predetermined threshold may refer to, for example, a maximum value of aberration that can be sufficiently corrected for in the image data using the workflow of
At step 510, if the predicted aberration data exceeds the predetermined conditions, it is determined that the charged particle beam microscope system needs to be tuned. If the output predicted aberration data defines the parameters of a predicted aberration in the image, a corresponding adjustment to be applied to the charged particle beam microscope system is determined and then applied to the element(s) of the microscope system to mitigate aberrations. Similarly, if the output predicted aberration data instead defines the predicted adjustment to apply to the components of the microscope system, the adjustment is applied. By running the steps of
With reference to
A computing system may have additional features. For example, the computing system 700 can include storage 740, one or more input devices 750, one or more output devices 760, and one or more communication connections 770. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 700. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system 700, and coordinates activities of the components of the computing system 700.
The tangible storage 740 may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium which can be used to store information in a non-transitory way and which can be accessed within the computing system 700. The storage 740 stores instructions for the software 780 implementing one or more innovations described herein.
The input device(s) 750 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system 700. The output device(s) 760 may be a display, printer, speaker, CD-writer, or another device that provides output from the computing system 700.
The communication connection(s) 770 enable communication over a communication medium to another computing entity. The communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can use an electrical, optical, RF, or other carrier.
In view of the above-described implementations of the disclosed subject matter, this application discloses the additional examples enumerated below. It should be noted that one feature of an example in isolation or more than one feature of the example taken in combination and, optionally, in combination with one or more features of one or more further examples are further examples also falling within the disclosure of this application.
| Number | Date | Country | Kind |
|---|---|---|---|
| 23203083.3 | Oct 2023 | EP | regional |
