Patent Application
20190228519
The present invention relates to component inspection, and more specifically, to anomaly detection using image-based physical characterization.
Visual inspection methods can capture images for components to be inspected and determine pass/fail criteria or identify whether one or more various anomaly types are present. In complex systems, there can be many components to inspect for a wide range of anomalies. Images can be captured under different conditions depending upon an operator, image capturing devices used, and other such factors.
Embodiments of the present invention are directed to a computer-implemented method for anomaly detection. A non-limiting example of the method includes reading, by a processor, a scale in image data representing an image of a plurality of physical characteristics. The processor resizes at least a portion of the image data to align with target image data representing a target image of one or more structures based at least in part on the scale to form resized image data representing one or more resized images. The processor applies noise reduction to the resized image data to produce test image data representing one or more test images and performs a best fit analysis on the test image data with respect to the target image data. Test image data of at least one of the test images having the best fit are stored with training image data representing a plurality of classification training images indicative of one or more recognized features. An anomaly in unclassified image data representing an unclassified image is identified based at least in part on an anomaly detector as trained using the classification training images.
Embodiments of the present invention are directed to a computer program product for anomaly detection. The computer program product includes a computer readable storage medium readable by a processing circuit and storing program instructions for execution by the processing circuit for performing a method. A non-limiting example of the method includes reading a scale in image data representing an image of a plurality of physical characteristics and resizing at least a portion of the image data to align with target image data representing a target image of one or more structures based at least in part on the scale to form resized image data representing one or more resized images. Noise reduction is applied to the resized image data to produce test image data representing one or more test images. A best fit analysis is performed on the test image data with respect to the target image data. Test image data of at least one of the test images having the best fit are stored with training image data representing a plurality of classification training images indicative of one or more recognized features. An anomaly in unclassified image data representing an unclassified image is identified based at least in part on an anomaly detector as trained using the classification training images.
Embodiments of the present invention are directed to a processing system for anomaly detection. The processing system includes one or more types of memory and at least one processor communicatively coupled with the one or more types of memory. The at least one processor is configured to perform a method. A non-limiting example of the method includes reading a scale in image data representing an image of a plurality of physical characteristics and resizing at least a portion of the image data to align with target image data representing a target image of one or more structures based at least in part on the scale to form resized image data representing one or more resized images. Noise reduction is applied to the resized image data to produce test image data representing one or more test images. A best fit analysis is performed on the test image data with respect to the target image data. Test image data of at least one of the test images having the best fit are stored with training image data representing a plurality of classification training images indicative of one or more recognized features. An anomaly in unclassified image data representing an unclassified image is identified based at least in part on an anomaly detector as trained using the classification training images.
Additional technical features and benefits are realized through the techniques of the present invention. Embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.
The specifics of the exclusive rights described herein are particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of the embodiments of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
The diagrams depicted herein are illustrative. There can be many variations to the diagram or the operations described therein without departing from the spirit of the invention. For instance, the actions can be performed in a differing order or actions can be added, deleted or modified. Also, the term “coupled” and variations thereof describes having a communications path between two elements and does not imply a direct connection between the elements with no intervening elements/connections between them. All of these variations are considered a part of the specification.
In the accompanying figures and following detailed description of the described embodiments, the various elements illustrated in the figures are provided with two or three digit reference numbers.
Various embodiments of the invention are described herein with reference to the related drawings. Alternative embodiments of the invention can be devised without departing from the scope of this invention. Various connections and positional relationships (e.g., over, below, adjacent, etc.) are set forth between elements in the following description and in the drawings. These connections and/or positional relationships, unless specified otherwise, can be direct or indirect, and the present invention is not intended to be limiting in this respect. Accordingly, a coupling of entities can refer to either a direct or an indirect coupling, and a positional relationship between entities can be a direct or indirect positional relationship. Moreover, the various tasks and process steps described herein can be incorporated into a more comprehensive procedure or process having additional steps or functionality not described in detail herein.
The following definitions and abbreviations are to be used for the interpretation of the claims and the specification. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains” or “containing,” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, a mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.
Additionally, the term “exemplary” is used herein to mean “serving as an example, instance or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms “at least one” and “one or more” can include any integer number greater than or equal to one, i.e. one, two, three, four, etc. The terms “a plurality” can include any integer number greater than or equal to two, i.e. two, three, four, five, etc. The term “connection” can include both an indirect “connection” and a direct “connection.”
The terms “about,” “substantially,” “approximately,” and variations thereof, are intended to include the degree of error associated with measurement of the particular quantity based upon the equipment available at the time of filing the application. For example, “about” can include a range of ±8% or 5%, or 2% of a given value.
For the sake of brevity, conventional techniques related to making and using aspects of the invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs to implement the various technical features described herein are well known. Accordingly, in the interest of brevity, many conventional implementation details are only mentioned briefly herein or are omitted entirely without providing the well-known system and/or process details.
Turning now to an overview of technologies that are more specifically relevant to aspects of the invention, a number of challenges can arise in performing recognition of physical characteristics in digital images. For example, when using a target image to compare with other images to identify similar features, there can be differences due to image rotation, image scale, image resolution, image noise, image intensity, and other such differences. Further, known visual recognition techniques that work well with color images cannot work well with gray-scale images (also referred to as black-and-white images). Some image capturing techniques use alternatives to visible light, such as X-ray based imaging or transmission electron microscopy (TEM). These alternative image capturing techniques can include a larger degree of noise or blurriness as compared to cameras that operate in the visible light spectrum.
Turning now to an overview of the aspects of the invention, one or more embodiments of the invention address the above-described shortcomings of the prior art by providing processing of images with varying resolution and orientation for developing training sets of various identified physical characteristics to support anomaly detection using machine learning. A number of sample images can be captured from an imaging device with sub-images extracted and normalized to construct a training image repository of classification training images that best fit a target image of one or more structures. The best fit can be determined by performing a best fit analysis. In embodiments of the invention, the best fit analysis can include image shifting and/or rotation. The process can be repeated for multiple target images having various physical characteristics indicative of “good” and anomalous features to collect groups of corresponding classification training images. Once image processing and collection of classification training images are performed, an anomaly detector can be trained using machine learning based at least in part on the training image repository. After training, the anomaly detector can be used to identify one or more anomalies in an unclassified image, for instance, during a visual inspection process of a component or article under analysis.
The above-described aspects of the invention address the shortcomings of the prior art by combining a number of image processing approaches to build a normalized training set for machine learning to support an anomaly detector. For example, identification of scaling information on an image itself can support resizing/rescaling of images. Random sampling of images by taking cuts proportional to a target image, along with noise reduction can be used to generate multiple test images from an original image. A test image most closely aligning to a target image can be identified by rotation and performing a best fit comparison, which can include image shifting and/or rotation. Multiple iterations can be performed to develop the training image repository of classification training images for test images indicative of recognized features with multiple classified physical characteristics. Technical benefits include enhanced anomaly detection for various physical characteristics captured in image data. The described processes can efficiently analyze gray-scale images, such as TEM images, where color image processing techniques are not as effective and would typically require a larger number of training sets as compared to embodiments described herein.
Turning now to a more detailed description of aspects of the present invention,
Thus, as configured in
The training data generator 402 can perform image processing 414 using one or more images 416 of the article under analysis 204 that include various physical characteristics. The image processing 414 can perform normalization operations to better align features of the images 416 with one or more target images 418. Upon performing normalization and best fit matching between the images 416 and target images 418, the image processing 414 outputs classification training images 406 for storage in the training image repository 404. Further details of the image processing 414 and identification of one or more anomaly 412 are described in reference to
Referring now to
At block 502, processor 101 can read a scale in image data representing an image 600 of
In embodiments, processor 101 can extract a plurality of image cuts from the image data representing image 600 as part of the image processing 414. As an example, image 700 of
At block 504, processor 101 can resize at least a portion of the image data representing image 700, such as the image cuts 704-710, to align with image data representing a target image 800 of
At block 506, processor 101 can apply a noise reduction to the resized image data representing one or more resized images (e.g., rescaled image cuts 704-710) to produce test image data representing a plurality of test images 904, 906, 908, 910 of
At block 508, processor 101 can perform a best fit analysis on the test image data representing test images 904-910 with respect to the target image data representing target image 800 as part of the image processing 414. The best fit analysis can include image shifting and/or rotation. Target image 1300 of
At block 510, test image data representing one or more of the test images 904-910 having the best fit can be stored with training image data representing classification training images 406 indicative of one or more recognized features. At block 512, the training image repository 404 including a plurality of classification training images 406 can be formed based at least in part on identifying a plurality of test images 904-910 as best fit with respect to a plurality of target images 418 indicative of a plurality of recognized features. For example, the training image repository 404 can be formed by repeating the process steps of blocks 502-512 for a plurality of test images 904-910 as best fit with respect to a plurality of target images 418 indicative of a plurality of recognized features. The recognized features include physical characteristics identified as ‘good’/expected results and examples of known anomalies.
At block 514, the anomaly detector 408 can be trained using machine learning based at least in part on the training image data representing the classification training images 406 in the training image repository 404. For example, one or more known machine learning techniques can be applied to map groups of similar features in the training image repository 404 to support feature identification and classification as unclassified images 410 are received. Training for structural and defect recognition can enable the anomaly detector 408 to both identify features within an image and determine which target type the feature most closely matches (e.g., an open circuit, a short circuit, good yield, and the like).
At block 516, an anomaly 412 can be identified in unclassified image data representing an unclassified image 410 based at least in part on the anomaly detector 408 as trained using the classification training images 406. A number of examples are depicted schematically in
The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user' s computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instruction by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments described. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments described herein.
