1. Technical Field of the Invention
The present invention relates generally to the field of image acquisition inspection systems. More particularly, the present invention relates to adjustable image acquisition inspection systems using feedback mechanisms.
2. Description of Related Art
Inspection systems are used in many different types of industries for a wide variety of purposes. For example, automated inspection systems are commonly employed in the manufacturing industry to inspect objects, such as solder joints and other components, on printed circuit boards (PCBs) for quality control purposes. In many automated inspection systems, the output is a classification of an inspected object into one of a few categories. As an example, in the PCB (printed circuit board) manufacturing industry, a component can be categorized as present or absent, while a solder-joint can be categorized as good or bad. The final classification of the object is made after an image of the object is acquired and processed.
Traditionally, the flow of information in inspection systems is one way, from the image acquisition system to the image processing system to the classifier. The performance of the overall inspection system is measured by the output of the classifier. However, the performance of the classifier is directly affected by the performance of the image acquisition system and the image processing system. Therefore, to improve the performance of the overall system, adjustments are typically made to the image acquisition and image processing systems. For example, adjustments can be made to compensate for variations in the illumination intensity between inspection systems, variations in the PCB design and layout, thickness variations between different PCBs, material changes between different objects and customer-specific requirements.
Currently, the operator of the inspection system manually performs such adjustments. The manual adjustment process is labor-intensive, time consuming and error-prone. Therefore, what is needed is an inspection system capable of automatically making adjustments to the image acquisition and image processing systems based on information from the later stages of the inspection, such as the classification stage.
Embodiments of the present invention provide an inspection system and method for providing feedback during an inspection of an object. The inspection system includes a device to capture an image and a processor that receives image data representing the object. The processor is operable to determine parameter modification information from the image data and modify an image parameter used during the production of the image data with the parameter modification information. The modified image parameter is used during the production of subsequent image data representing the object.
In one embodiment, the image parameter is an image acquisition parameter. Examples of image acquisition parameters include illumination parameters, such as the intensity of illumination, angle of illumination or duration of illumination, image view parameters, such as the positional relationship between the object and a sensor operable to capture an image of the object, and sensor parameters, such as the exposure duration of the sensor or resolution of the sensor. In one feedback embodiment, the processor is an image processor operable to process the image data and calculate the parameter modification information for the image acquisition parameter based on the processed image data. In another feedback embodiment, the processor is a classification processor operable to output a classification and calculate the parameter modification information for the image acquisition parameter based on the classification. For example, if the classification is incorrect as a result of an original setting of the image acquisition parameter, the classification processor is operable to calculate the parameter modification information to correct the incorrect classification and modify the original setting of the image acquisition parameter to a modified setting based on the parameter modification information.
In another embodiment, the image parameter is an image processing parameter. Examples of image processing parameters include processing type parameters, such as the type of algorithm used to process the image data, and processing complexity parameters, such as the complexity of the algorithm used to process the image data. In this embodiment, the processor is a classification processor operable to output a classification and calculate the parameter modification information for the image processing parameter.
By providing feedback between the different parts of the inspection system, adjustments can be made automatically in real-time with improved reliability and increased speed. Furthermore, the invention provides embodiments with other features and advantages in addition to or in lieu of those discussed above. Many of these features and advantages are apparent from the description below with reference to the following drawings.
The disclosed invention will be described with reference to the accompanying drawings, which show sample embodiments of the invention and which are incorporated in the specification hereof by reference, wherein:
The inspection system 100 further includes a processor 150 for receiving the raw image data 145 representing the image of the object 130. The processor 150 can be a microprocessor, microcontroller, programmable logic device or other type of processing device capable of performing the functions described herein. In addition, the processor 150 can include multiple processors or be a single processor having one or more processing elements (referred to herein as separate processors). For example, as shown in
The image processor 160 is connected to receive as input the raw image data 145 from the sensor 140, and operates to process the raw image data 145 and output processed image data 165. For example, if the sensor 140 is a color sensor incorporating a color filter array, the image processor 160 can demosaic the image. Demosaicing is a process by which missing color values for each pixel location are interpolated from neighboring pixels. There are a number of demosaicing methods known in the art today. By way of example, but not limitation, various demosaicing methods include pixel replication, bilinear interpolation and median interpolation. Other types of processing that the image processor 160 can perform include noise filtering, image enhancement, image reconstruction for three-dimensional or X-ray images and extraction of features of the object 130 that are of interest. It should be understood that as used herein, the phrase “features of the object” includes measurements of the object 130, components on a surface of or within the object 130 or other indicia of the object 130. An example of an image reconstruction process for three-dimensional images is described in co-pending and commonly assigned U.S. Application for Patent, Ser. No. ______ (Attorney Docket Number 10021084), which is hereby incorporated by reference.
The classification processor 170 is connected to receive as input the processed image data 165 from the image processor 160, and operates to classify one or more features of the object 130 from the processed image data 165 and output a classification 175 of the object feature(s). For example, if an extracted feature is a solder joint on a printed circuit board, the classification processor 170 can analyze the processed image data 165 and output a classification 175 indicating whether the solder joint is good or bad. As another example, if an extracted feature is another component on a printed circuit board, the classification processor 170 can analyze the processed image data 165 and output a classification 175 indicating whether the component is present or absent. Other types of classifications 175 that can be output by the classification processor 170 include measurements of specific aspects of one or more features of the object 130, such as the size of a solder joint or another component, volume of a solder joint or another component, location or position of a component or size of the object 130.
The classification 175 is also stored in a computer-readable medium 180 for later processing or display. The computer-readable medium 180 can be a memory device, such as random access memory (RAM), read-only memory (ROM), flash memory, EEPROM, disk drive, compact disk, floppy disk or tape drive, or any other type of storage device. Additional processing information can also be stored on the computer-readable medium 180 and accessed by the image processor 160 and classification processor 170. For example, such processing information can include various processing parameters, such as algorithms that can be used to process the raw or processed image data 145 or 165, respectively, and varying complexities of the algorithms used to process the raw or processed image data 145 or 165, respectively. As another example, such processing information can include feature specifications or other metrics against which the processed image data 165 can be measured to determine the classification 175.
In traditional inspection systems, the flow of data is in one direction, from the image acquisition system 120 to the image processor 160 to the classification processor 170. The performance of the inspection system is measured by the output of the classification processor 170, and any adjustments that need to be made to the image acquisition system 120 or image processor 160 are conventionally made manually, off-line. The manual adjustment process is labor-intensive, time consuming and error-prone.
Therefore, in accordance with embodiments of the present invention, adjustments are made automatically with improved reliability and increased speed by altering the flow of data to provide feedback between the various parts of the inspection system 100. For example, as shown in
In one implementation embodiment, feedback is provided only in a “tuning” mode of the inspection system 100 instead of during the run-time to prevent any impact to the in-line running time. In another implementation embodiment, feedback is provided in real time during the operation of the inspection system 100, to compensate for drift in various parameters. In a further implementation embodiment, feedback is provided during a “learning” mode for a batch of objects, such as printed circuit boards. For example, while in a “learning” mode, the inspection system 100 acquires multiple images of the object, each image being taken using a different illumination setting and/or view. The image processor 160 and classification processor 170 determine the optimal images and algorithms that are needed to produce an optimal classification of the object. The inspection system 100 acquires the optimal images and uses the optimal algorithms when running at full production speed.
In one embodiment, if the received image data includes data outside a predetermined tolerance for a specific image parameter, parameter modification information is calculated to modify the specific image parameter to produce image data within the tolerance for that specific image parameter. For example, if the average reflected illumination intensity over the image or over a portion of the image is below a predetermined threshold, the illumination intensity of the illumination source is increased by an amount sufficient to produce image data having an average reflected illumination intensity above the threshold. Other examples are discussed below in connection with
In another embodiment, if the classification of the object is incorrect, e.g., as determined by the operator of the inspection system or if the inspection system is unable to classify the object, parameter modification information is calculated to modify one or more of the image parameters used to capture the first image data to correct the classification of the object or to enable the inspection system to classify the object. For example, if it is determined that the incorrect classification is due to the type of algorithm used to process the image data, the parameter modification information identifies a different algorithm to be used to process the image data and produce a correct classification. The cause of the incorrect classification is determined by the operator or automatically by the system using a diagnostic algorithm that analyzes the image data and compares the image data to predetermined criteria (e.g., thresholds) for each of the image parameters.
At block 350, second image data representing the object is received. The second image data is produced using the modified image parameter. The feedback process ends at block 360. It should be understood that in other embodiments, the feedback process can continually provide feedback to make adjustments to one or more of the image parameters, as needed.
Using an original setting of the image acquisition parameter 400, the image acquisition system 120 captures an image of the object and provides raw image data 145 representing the image of the object to the image processor 160. The image processor 160 processes the raw image data 145 and determines parameter modification information 450 based on the raw and/or processed image data. The parameter modification information 450 is fed back to the image acquisition system 120 to modify the image acquisition parameter 400 for capturing a subsequent image of the object. Examples of parameter modification information 450 include, but are not limited to, a modification to the power of the X-ray, a modification to the particular areas on the object to focus on, a modification to the integration time of the X-ray, a modification to the view of the object and a modification to the illumination.
For example, the image processor 160 can detect that the image is too dark, too bright or has insufficient dynamic range from the raw image data 145 and can provide parameter modification information 450 to the image acquisition system 120 to adjust the illumination power and duration, and the mode of operation of the sensor. As another example, the image processor 160 can detect that the image has excessive resolution or dynamic range in certain portions of the image, and can provide parameter modification information 450 to the image acquisition system 120 to lower the resolution or dynamic range. In this case, by reducing the acquired resolution or dynamic range, the speed of the image acquisition system 120 is increased. As a further example, the image processor 160 can determine from the raw and/or processed image data that wider-angle images are needed for reliable X-ray reconstruction. In this case, the parameter modification information 450 is fed back to the image acquisition system 120 to acquire more images using a different Z-height. Similarly, for three-dimensional reconstruction of images, the image processor 160 can detect that information is missing in certain areas, due to shadowing or steep angles. In this case, the parameter modification information 450 is fed back to the image acquisition system 120 to change the angle of illumination or types of lighting rings used to illuminate the object.
Using an original setting of the image acquisition parameter 400, the image acquisition system 120 captures an image of the object and provides raw image data 145 representing the image of the object to the image processor 160. The image processor 160 processes the raw image data 145 using an original setting of the image processing parameter 600 and outputs processed image data 165 to the classification processor 170. Using the processed image data 165, the classification processor 170 classifies the object and determines parameter modification information 450 based on the classification of the object and/or processed image data 165. The parameter modification information 450 is fed back to either or both of the image acquisition system 120 to modify the image acquisition parameter 400 for capturing a subsequent image of the object and the image processor 160 to modify the image processing parameter 600 for processing the raw image data 145 representing the current image and/or the raw image data 145 representing a subsequent image of the object.
If modification is necessary, at block 770, parameter modification information is determined, and at block 780, the parameter modification information is used to modify the image acquisition parameter. The modified image acquisition parameter is used to produce subsequent raw image data representing a subsequent image of the object at block 730. If no modification to the image acquisition parameter is necessary, the feedback process ends at block 790. It should be understood that in other embodiments, the feedback process can continually provide feedback to make adjustments to one or more of the image acquisition parameters, as needed.
If modification is necessary, at block 870, parameter modification information is determined, and at block 880, the parameter modification information is used to modify the image processing parameter. The modified image processing parameter is used to process the raw image data representing the current image of the object and/or a subsequent image of the object at block 840. If no modification to the image processing parameter is necessary, the feedback process ends at block 890. It should be understood that in other embodiments, the feedback process can continually provide feedback to make adjustments to one or more of the image processing parameters, as needed.
If modification of an image acquisition parameter is not necessary, at block 940, one or more features of the object are classified based on the processed image data. At block 945, a determination is made whether one or more of the image acquisition parameters should be modified due to an incorrect classification of one or more of the features. If modification is necessary, at block 935, parameter modification information is determined and the parameter modification information is used to modify the image acquisition parameter. The modified image acquisition parameter is used to capture a subsequent image of the object at block 915. If no modification to the image acquisition parameter is necessary, at block 950, a determination is made whether one or more of the image processing parameters should be modified due to an incorrect classification of one or more of the features.
If modification is necessary, at block 955, parameter modification information is determined and the parameter modification information is used to modify the image processing parameter. The modified image processing parameter is used to process the raw image data representing the current image of the object and/or a subsequent image of the object at block 925. If no modification to the image processing parameter is necessary, the feedback process ends at block 960. It should be understood that in other embodiments, the feedback process can continually provide feedback to make adjustments to one or more of the image parameters, as needed.
Due to the large number of features 1000 of the object 130, it is usually not feasible for the operator to inspect all of the features 1000. Therefore, only images of the features 1000 that were automatically classified by the system 100 as defective or indicating a problem are typically presented to the operator on a display 1020. The image itself, processed image data representing the image and/or the classification is displayed on the display 1020. A user interface 1030 (e.g., keyboard, mouse, touch screen, light pen or other interface) allows the operator to control the information displayed on the display 1020. In addition, the user interface 1030 enables the operator to cause the classification processor to provide parameter modification information to the image processor and/or image acquisition system when the operator determines the displayed classification is incorrect. With the automatic provision of parameter modification information in accordance with the present invention, the quality of the information displayed to the user is improved, thus reducing or eliminating the time needed for manual adjustments by the user. As a result, throughput is increased and errors are minimized.
The beam 1120 impinges upon a sensor 140 to produce an image based on the cross-sectional density of the object 130, including feature 1000. In one embodiment, the sensor 140 includes a row of detector elements forming a linear array. Typically, there are between 300 and 700 individual detector elements arranged in a linear array. The linear array is sequentially scanned as the object 130 is moved over the sensor by the conveyor belt 1010, and the image generated is a two-dimensional “gray scale” raster image. Raw image data representing the raster image is sent to a processor (not shown) in accordance with the present invention for analysis and classification of the feature and/or object, as described above in connection with
As will be recognized by those skilled in the art, the innovative concepts described in the present application can be modified and varied over a wide rage of applications. Accordingly, the scope of patents subject matter should not be limited to any of the specific exemplary teachings discussed, but is instead defined by the following claims.
This application is related by subject matter to U.S. Utility Application for patent Ser. No. 10/699,542, entitled METHOD FOR CHOOSING TESTER DESIGNS AND USE MODEL USING OPERATING CHARACTERISTICS, filed on Oct. 31, 2003.