Patent Application
20250116634
The present disclosure relates to a method for automated defect classification of a sample by means of scanning acoustic microscopy, non-transitory computer program product as well as a scanning acoustic microscope.
Quality inspection by means of ultrasound, also called acoustic microscopy, enables the non-destructive measurement and inspection of materials and components. During image generation in a scanning acoustic microscope, a sample to be examined is scanned line by line and a short ultrasound pulse is generated on each pixel, whose ultrasound signal reflected by the sample is evaluated pixel by pixel. Typically, the chronological sequence of the signal is analyzed within a specified time range (gate). The time range to be analyzed may also be defined relative to the surface signal of the sample to ensure that the sample is imaged at a previously specified depth relative to the surface (surface trigger).
Generally, the maximum signal intensity resp. amplitude within the selected time range is represented by a gray scale value for each pixel resp. each raster point, and thus an image of the sample is generated. However, complex data processing operators such as filters or transformations may also be employed in order to generate this gray scale value. The produced micrographies image the sample in the plane perpendicular to the transducer and are referred to as a C-scan. A further imaging mode is the B-scan. A B-scan represents an acoustic cross section through the sample. A location coordinate is plotted along the X-axis of the image, the time of flight of the sound signal along the Y-axis.
In the case of using scanning acoustic microscopy in production control and statistical quality control, the evaluation of the C-scan images is generally performed by automated image processing. In this case, apart from conventional algorithms, like threshold value analyses, morphological filters and matrix-based image operations, also image-based solutions by means of artificial intelligence are employed. It is the purpose of image processing to reliably detect critical flaws in the components with as few misevaluations as possible in the form of false positive evaluated structures.
To this end, one or more images are transferred from the ultrasound inspection to an image processing, which then examines the received images for various features. The coordinates of the found flaws or as well properties are then transmitted to a manufacturing control system. In some cases, a final good/bad evaluation is performed and defective components are sorted out. In particular, in the field of semiconductor manufacturing, the communication standard SECS/GEM is used for the connection.
This method works well with classic semiconductor components such as individual components with a plastic socketing compound, DCB-based power electronics with classic solder joints, or wafers joined together by means of fusion bonding. In all these applications, it is possible to display defects with an unambiguous brightness compared to the remaining flawless regions.
Methodological limits for the conventional image processing are always reached when the brightness value of a pixel can no longer be unambiguously allocated to a flaw and the same gray scale value can occur both in the case of component defects and in the case of intact structures. Here, the capability of image processing can be improved by identifying the component regions and merely performing a local analysis.
The analysis of the ultrasound data of complex components is often no longer feasible with conventional image processing. In the case of complex image contents, image analysis based on machine learning may also be used. However, image-based machine learning requires large amounts of data for training neural networks. In particular, for training rarely occurring types of defects, it can be very elaborate to get the required high number of defective components. For example, a workaround is an artificial amplification of a few exemplary defects. However, this can lead to misinterpretations of the network, as non-representative data is used for training. Approaches with image-based artificial intelligence are therefore very complex and little reliable in the application of defect recognition in acoustic microscopy on materials and components, since only insufficient image material is available for training the artificial neural networks and insufficiently good data sets are available for comprehensive modeling.
An object of the present disclosure is to improve the automatic defect recognition in the non-destructive examination of even complex materials and components, for example complex semiconductor samples.
This object can be achieved by a method for automated defect classification of a sample by means of scanning acoustic microscopy, for example, in the frequency range of 10 MHz to 2000 MHz, wherein a sample can be scanned with an scanning acoustic microscope with one or more ultrasonic transducers, the sample can be positioned stepwise relative to the ultrasonic transducer or transducers at raster points, and one or more ultrasound signals can be generated at each raster point and recorded after reflection on and/or in the sample and/or after transmission by the sample, wherein the chronological sequences of the recorded ultrasound signals reflected and/or transmitted can be digitized and analyzed, such as, classified, with regard to defects by means of at least one neural network, which had previously been trained unsupervised in an initial training by means of a deep learning algorithm with the help of scanning acoustic microscope scans of one or more control samples of the same type as the sample or with known and labeled defects. The method according to the present disclosure is applicable to diverse types of samples, from simple material samples to complex semiconductor samples, and can be based on the fact that scanning acoustic microscopes, which can correspond to the current state of the art, can allow the sound signals to be detected at each scanning point, processed in real time and stored. These chronological sequences can also be referred to as A-scan signals. Therewith, instead of a previously image-based analysis, a signal- or volume-based analysis of the complete chronological signal sequences can be performed.
In principle, the problem of unavoidable data reduction in the prior image-based analysis is hereby avoided, which is caused in that during the image generation for each pixel, thus each raster point, the respectively detected and perhaps complex ultrasound signal is reduced to a single value, for example the maximum of the amplitude in the time frame considered. With this reduction, the comprehensive information contained in the ultrasound signal can be lost for the analysis. This leads, among other things, to the noise of a single gray scale value reaching a critical magnitude in comparison to the required threshold values in a conventional image analysis, such as at high amplifications, and negatively affecting the certainty of a flaw decision.
The ultrasound signal of complex semiconductor samples often contains signal portions that not only correspond directly to the coupled compression wave, but signal portions are also generated by other wave types. In fast materials, a so-called mode conversion may occur. Here, a coupled compression wave generates both a compression wave portion and a shear wave portion at the interface, both of which move at different propagation speeds in the sample and thus generate temporally shifted echo signals. If two interfaces with a large reflection coefficient are opposite to each other, multiple echoes are formed, which are caused by sound waves reflected back and forth. Transducers with a large aperture angle can also cause Rayleigh and Lamb waves in solid bodies. All of these signals can be used for a defect/non-defect decision. A weakly developed defect thus generates characteristic signatures at many different locations of the signal. These signatures supply a better data base for identification of defects than just a single gray scale value, as different parts of the signal can be used for the analysis.
The use of an artificial neural network that can evaluate and classify the overall signal of an interface increases the reliability of the network's decisions in its analysis of the data and exceeds that of image-based methods. In contrast to the often limited data availability and the considerable human effort in the case of labeling image data, which are required for training an artificial neural network for image data analysis, the presented method utilizes the classification of individual A-scan signals. For this purpose, it can be sufficient to train an artificial neural network with a plurality of A-scan signals from representative and relevant regions. Even a 4-digit number of such signals represents only a small percentage of the total amount of data with respect to a complete C-scan image.
The method according to the present disclosure is directed to training the network with signals from the relevant region, so that the network can derive information and examine the usability of the complete information of the signal for the deep learning network training. Deep learning networks are very efficient in automated feature extraction and often obtain better results herein than can be attained in the case of a manual selection. By correct optimization and continuous learning, deep learning networks are able to draw more conclusions than a human operator. 3D data that are detected by the scanning acoustic microscope are used as raw data for training the statistical model. These data make it possible to reconstruct all individual gate settings like A-scan, C-scan, B-scan, slice and 3D-scan (including amplitude and time-of-flight data) so that the sample can be displayed in individual layers.
While previous methods are reliant on the clean signals of ultrasonic transducers in their applicability and are overwhelmed in the case of broadband ultrasonic transducers due to the many signals of different origin, in embodiments of the method according to the present disclosure, the ultrasonic transducer or transducers can be designed as broadband ultrasonic transducer, such as with a bandwidth of at least 10%, in some examples at least 20%, wherein the ultrasonic transducer or transducers can be designed to record signals, which are produced from mode conversion, multiple echoes, Rayleigh waves, Lamb waves and/or due to intrinsic properties of the transducer, and to forward them to a receiver.
The method according to the present disclosure uses neural networks that have been trained either supervised or unsupervised. In the first case, a control sample or several control samples, at least one of which is defective, can be scanned, the defects of which are either visible to an expert or are known from other examinations. The expert then marks one or more regions of the control sample that can be flawless and one or more defective regions. The training of the neural network can prepare it to assign corresponding ultrasound signals to one of the categories. In the case of the deep learning, it is not preset, which signal structures are concerned, but the neural network can find the most meaningful signal structures by itself.
In the case of unsupervised learning, a control sample can be scanned that has no defects. The ultrasound signals of this control sample can show the neural network trained therewith how the ultrasound signals should ideally look like. In the training phase, the neural network has learned thereby the full range of acceptable signals for different structures of the control sample. This can be a large plurality of different structures and corresponding signal sequences that would not have been imageable in conventional manner. If a defective sample is scanned later, the ultrasound signals at the positions of the defects will not be in the spectrum of the acceptable signals, but will differ from them. With the help of a criterion, which describes the deviation from the ideal signal image, it can then be assessed whether or not a defect is present.
In this context, it can be sensible to train the neural network with ultrasound signals that have been recorded with the same parameters as the signals to be classified later. This concerns, among other things, the bandwidth and transmission characteristics of the ultrasonic transducers, but also the signal frequency of the ultrasound pulse.
In embodiments, the ultrasonic transducer or transducers can be designed as broadband ultrasonic transducer or transducers, such as with a bandwidth of at least 10%, in some examples at least 20%, wherein the ultrasonic transducer or transducers can be designed to record signals, which are produced from mode conversion, multiple echoes, Rayleigh waves, Lamb waves and/or due to intrinsic properties of the transducer, and to forward them to a receiver. Due to their high bandwidth, the ultrasound signals transmitted by broadband ultrasonic transducers can include substantially more information than the usually used ultrasonic transducers, which comprise a smaller bandwidth or are operated with a smaller bandwidth. In the case of using broadband ultrasonic transducers, this information density has made it impractical so far to apply a corresponding image analysis based on neural networks to data generated with the help of broadband ultrasonic transducers. The method according to the present disclosure, on the other hand, can utilize the additionally available information for improvement of the sharpness of separation of the classification. The high information density of the broadband signals also makes it possible to enable the training of the neural networks with the help of only one or very few scans.
In embodiments, the neural network can be a convolutional neural network (CNN) with a 1-D-resnet architecture with 1-dimensional convolutional blocks, wherein the deep learning algorithm can include an automated feature extraction. Alternatively, the neural network can be a recurrent neural network (RNN), with a Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU)-based architecture, where the deep learning algorithm can include an adaptation of a feed-forward stage of the RNN. The neural network may also be designed as a hybrid of a CNN and an RNN.
In embodiments, the ultrasound signals can be smoothed by means of a deterministic algorithm prior to the analysis, such as by wavelet filtering, on the basis of Daubechies wavelets or Mexican hat wavelets. Also, the ultrasound signals, with the help of which the neural network has been trained, should have been subjected to the same preprocessing in this case. Smoothing may serve for elimination of signal artifacts that may be produced due to instrument design.
In embodiments, the chronological sequence of the ultrasound signals is analyzed as a discrete time series in the form {signal length×1}. Due to the digitization, the discrete time series may contain a length of several thousand data points, resulting in a vector of the same length. In this case, each raster point resp. each pixel is individually analyzed as to whether or not it belongs to a defect. Additionally or alternatively, a volumetric analysis may take place, wherein several ultrasound signals of adjacent raster points are combined and analyzed as discrete time series of raster points arranged in a rectangle of the edge lengths n×m, as a data set of the form {signal length×n×m}, where at least one of n and m is ≥2, n=m, and n and m can be odd numbers. This includes a row-by-row or column-by-column combination of several adjacent pixels resp. a combination of pixels in a rectangular resp. square arrangement, wherein, for example, if an odd number of pixels is selected as the edge length in the case of a square arrangement, the center of the square in turn coincides with the central pixel. In the case of the combination of adjacent pixels, regardless of whether this takes place in a linear or in a planar form, the analysis resp. training of the neural network can be adapted, in that central pixels are weighted higher than peripheral pixels.
A gain in processing performance is attained in embodiments, if the data sets of the discretized ultrasound signals are two-dimensional, with the discretized ultrasound signal in the first dimension and time steps of 1 in the second dimension. The actual value of the time steps depends on the digitization rate of the analog ultrasound signal.
In embodiments, the classification can include one or more classes for the absence of defects, such as with one or more classes for one or more structures, and one or more classes for defects, and/or different kinds of defects.
During the classification, in embodiments, a confidence can be determined for the respective classification, which can be represented in particular in a representation for each raster point by an intensity value.
Among other things, for the systematic analysis of flaws in the production of the samples, it is provided in embodiments to superimpose the classification, such as a confidence, onto a C-scan image as a color or gray scale value for raster points with defect classification, the confidence can be integrated as a brightness parameter, color parameter or transparency parameter.
In embodiments, prior to the processing by the neural network, the ultrasound signals can be converted by means of regression, such as by interpolation or by means of a trained statistical model, into a signal of a predefined length and/or predefined chronological increment, without loss of information with regard to the sample, where noise, signal artifacts and/or interference signals and distortions due to design can be reduced. According to the present disclosure, a loss of information can be the loss of information indicative of the sample, while interference signals and distortions can be inherent in the system and do not contain any information about the sample itself. Although ideally the training data on the control samples are recorded under the same circumstances and parameters as later the ultrasound signals on the samples to be tested, in practice not all test parameters can be accurately repeated, so that it may be useful first to record and analyze the system response to different situations and to remove those portions of the system response that are known during the preprocessing of the ultrasound signals prior to the training and prior to the analysis, such as noise, signal artifacts, interference signals and distortions.
Additionally or alternatively to that, the ultrasound signals may be anomaly-corrected using an encoder-decoder architecture, where the ultrasound signals can be decomposed into components and reconstructed again based on trained domain knowledge, a reconstruction error can be derived from the difference of the input signal and the reconstructed signal. The encoder-decoder architecture can image the received ultrasound signals onto good ultrasound signals, thus, can be an example of a regression. For this purpose, first a statistical model, for example a neural network, can be trained such that it images in turn good signals onto good signals. This is an example of unsupervised learning, which generates the mentioned domain knowledge, which is then contained in the neural network. Anomalous signals that deviate from the good signals due to the presence of defects are in turn imaged onto the previously learned good signals by the encoder-decoder architecture, thus anomaly-corrected. The difference between the output signals and the input signals can form a basis for a defect classification, which can be represented, for example, via threshold values for a degree of the deviation.
In embodiments, a reconstructed 3D data set of the sample volume of the sample can be generated by a trained statistical model in consideration of the properties of the ultrasound propagation in the sample, which can be corrected for effects by defocusing, multiple echoes, mode-converted signals, and intrinsic transducer signals, allowing the generation of sectional images along any planes. These effects can be modeled and calculated back from the ultrasound signals, thus removed.
In embodiments, an untrained neural network or a neural network pre-trained with the help of at least one control sample of foreign types of at least one foreign sample type can be used as the basis for the initial training. The training in the first case can take a bit longer, since the neural network is trained ab initio, while in the latter case there is already information available that only needs to be adapted to the present control sample. Thereby, the learning converges comparatively fast, depending on the similarity with previously trained sample types. After completion of the training for the current sample type, the current version of the neural network can be stored for later use on the same sample type, so that a database for neural networks for different prototypes can be set up without having to train a neural network every time anew.
For the initial training of the neural network, in embodiments, the labeling of defects and other parts of the control sample takes place by hand, such as on a C-scan section, where the regions marked by hand can account for less than 50%, such as less than 20%, or less than 10%, of the area of the scan of the control sample. The method according to the present disclosure thus manages with significantly less starting material, since the full information content of the ultrasound signals for each pixel can be used. Therefore, a restriction to relatively small regions of the control sample suffices.
The object of the present disclosure is also achieved by a non-transitory computer program product with program code means, which, upon execution on a data processing installation of a scanning acoustic microscope, cause the scanning acoustic microscope to execute a method according to the present disclosure described above. Alternatively, the training of the neural network and the analysis of the ultrasound signals may also take place on a separate data processing installation.
Furthermore, the object of the present disclosure is also achieved by an scanning acoustic microscope having a positioning system with a holder for a sample, at least one ultrasonic transducer, such as a broadband ultrasonic transducer, a pulse generating and receiving unit, which is connected or connectible with the at least one ultrasonic transducer, a data processing installation and an analog-to-digital converter unit, wherein the scanning acoustic microscope is designed and configured for performing the methods described according to the present disclosure, such as by a non-transitory computer program product executable and stored in the data processing installation as described in the present disclosure.
The non-transitory computer program product and the scanning acoustic microscope realize the same advantages, properties and features as the method according to the embodiments previously described.
Further features of the present disclosure will become apparent from the description of the embodiments according to the present disclosure together with the claims and the attached drawings. Embodiments according to the present disclosure may fulfill individual features or a combination of several features.
In the context of the present disclosure, features being identified with “in particular” or “preferably” are to be understood as optional features.
Further features become clear from the description of embodiments, together with the claims and the accompanying drawings. Individual features or a combination of a plurality of features can fulfil embodiments.
The embodiments will be described below without restricting the general inventive idea with the help of exemplary embodiments with reference to the drawings, and regarding any details according to the present disclosure which are not explained further in the text reference is expressly made to the drawings. They show in:
In the drawings, the same or similar types of elements and/or parts are provided with the same reference numbers so that a corresponding re-introduction is omitted, respectively.
In the left-hand half of the figure, the chronological sequence of the amplitude of the reflected signal is represented, the so-called A-scan. Corresponding to the shorter travel time, first the echo 32 comes back from the front surface. This one also has the largest amplitude. The substantially weaker echo 36 from the interior of the sample follows, and at last the echo 34 of the pulse comes from the rear surface 24. This signal sequence is also shown schematically in vertical alignment corresponding to the depth profile of the sample 20.
For the analysis of the sample with regard to defects, the middle part of the signal is interesting. Therefore, a temporal gate 40 is applied that excludes the echoes 32, 34 of the pulse 30 from the front surface 22 and the rear surface 24. The signal located within the gate 40 contains a variety of information about the inner life of the sample 20 at the current raster point resp. pixel.
In
In
In
Instead of unsupervised learning, supervised learning may also take place. In this case, a defective control sample is scanned instead of a defect-free control sample. This is followed by a labeling step in which an expert or another system, which is used for defect recognition, marks regions on the control sample that are characteristic of, if applicable, different, defects, on the one hand, and of, if applicable, different, defect-free structures of the sample. The neural network is then trained towards the aim of reproducing the corresponding defect resp. structure classification as accurately as possible. Procedures for supervised learning are also known.
In
After selection of the procedure 112, the quantity of the recorded signals is fed to an evaluation. This may take place online and in real time, but also downstream and, if applicable, offline. Therein, the recorded ultrasound signals may be subjected to a preprocessing 116, for instance, an upsampling, a downsampling and/or a filtering, in particular wavelet filtering. A regression 118 may take place, for example, by using an autoencoder 119. An example of this is a neural network that has been trained unsupervised. Also, a classification 120 may take place by using a CNN 50 or RNN 70 trained by means of supervised learning.
Context-based functions concern the collective evaluation of the ultrasound signals of adjacent scan points resp. pixels. These may as well be classified by means of correspondingly trained neural networks. Due to the larger number of input data, the neural networks are dimensioned correspondingly larger, but the larger number of available signals also ensures a possibly greater discriminatory power than in the case of the individual signal analysis. These more complex signals collectively evaluated are also very suitable for a 2-D or 3-D reconstruction of the sample, since uncorrelated signal noise is suppressed by the combination of the signals of adjacent raster points, so that a clearer reconstructed image may be produced. A reconstruction may, of course, also take place with the help of the individual signals.
While there has been shown and described what is considered to be preferred embodiments of the invention, it will, of course, be understood that various modifications and changes in form or detail could readily be made without departing from the spirit of the invention. It is therefore intended that the invention be not limited to the exact forms described and illustrated, but should be constructed to cover all modifications that may fall within the scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10 2022 125 479.8 | Oct 2022 | DE | national |
The present application is a continuation of PCT/EP2023/076671 filed on Sep. 27, 2023, which claims priority to DE 10 2022 125 479.8, filed on Oct. 4, 2022, the entire contents of each of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/EP2023/076671 | Sep 2023 | WO |
| Child | 18982505 | US |
