Patent Application
20050226490
1. Field of the Invention
The invention relates to automated visual event detection, object detection and inspection, and the related fields of industrial machine vision and automated image analysis.
2. Description of the Related Art
It is well-known in the art to use machine vision systems and methods to analyze objects, for example to identify, locate, or inspect objects being manufactured on a production line. Such systems generally receive a trigger signal from an external device, such as a photodetector, to indicate that an object is present in the field of view of the machine vision system. Upon receiving a trigger signal, the machine vision system will capture a digital image of the object, analyze the image using a variety of well-known methods to produce useful information, such as identity, location, and/or quality or the object, and report the information to external automation equipment for use in the manufacturing process.
Speed of operation is almost always at a premium for machine vision systems and methods. Objects are generally manufactured at high rates, and image analysis operations usually require sophisticated computational shortcuts to meet time budgets. The rapid rise in digital computer speed over the years has not eased the burden of providing high-speed operation for machine vision systems, but rather has been applied to ever-increasing sophistication of image analysis methods so that more accurate and reliable decisions can be made.
One common method for increasing the speed of a machine vision system is to allow an image of an object to be captured simultaneously with the analysis of the image corresponding to the previous object. With such an arrangement, the rate at which a machine vision system can analyze objects is determined by the longer of the capture time and the analysis time, instead of the sum of those times. In cases where capture and analysis times are roughly equal, an improvement of almost a factor of two in object analysis rate can be achieved. If the object presentation rate exceeds the analysis rate of the machine vision system, trigger signals will be ignored and the objects corresponding to those signals will not be analyzed.
Systems that allow simultaneous capture and analysis use two digital memory buffers, one to hold a new image being captured and one to hold the previous image being analyzed. The two buffers switch roles after each capture and analysis cycle is complete, and so they are often called “ping-pong” buffers.
It is also known in the art to capture images asynchronously relative to image analysis, so that images are captured whenever a trigger signal arrives and stored for analysis at some later time. Such an arrangement is suitable where the image capture time is significantly shorter than the image analysis time, where the object presentation rate and/or the image analysis time is variable and the average object presentation rate does not exceed the average image analysis rate, and where the information produced by the analysis is not needed too soon after the trigger signal. The captured images are stored in a first-in, first-out (FIFO) memory buffer for subsequent analysis. If the average object presentation rate exceeds the average image analysis rate over a period of time whose length is determined by the size of the FIFO buffer, then the FIFO will overflow and triggers will be ignored.
The Vision Detector Method and Apparatus teaches novel methods and systems for analyzing objects by capturing and analyzing digital images of those objects. These teachings provide fertile ground for innovation leading to improvements beyond the scope of the original teachings. In the following section the Vision Detector Method and Apparatus is briefly summarized, and a subsequent section lays out the problems to be addressed by the present invention.
Vision Detector Method and Apparatus
The Vision Detector Method and Apparatus provides systems and methods for automatic optoelectronic detection and inspection of objects, based on capturing digital images of a two-dimensional field of view in which an object to be detected or inspected may be located, and then analyzing the images and making decisions. These systems and methods analyze patterns of brightness reflected from extended areas, handle many distinct features on the object, accommodate line changeovers through software means, and handle uncertain and variable object locations. They are less expensive and easier to set up than prior art machine vision systems, and operate at much higher speeds. These systems and methods furthermore make use of multiple perspectives of moving objects, operate without triggers, provide appropriately synchronized output signals, and provide other significant and useful capabilities that will be apparent to those skilled in the art.
One aspect of the Vision Detector Method and Apparatus is an apparatus, called a vision detector, that can capture and analyze a sequence of images at higher speeds than prior art vision systems. An image in such a sequence that is captured and analyzed is called a frame. The rate at which frames are captured and analyzed, called the frame rate, is sufficiently high that a moving object is seen in multiple consecutive frames as it passes through the field of view (FOV). Since the object moves somewhat between successive frames, it is located in multiple positions in the FOV, and therefore it is seen from multiple viewing perspectives and positions relative to the illumination.
Another aspect of the Vision Detector Method and Apparatus is a method, called dynamic image analysis, for inspecting objects by capturing and analyzing multiple frames for which the object is located in the field of view, and basing a result on a combination of evidence obtained from each of those frames. The method provides significant advantages over prior art machine vision systems that make decisions based on a single frame.
Yet another aspect of the Vision Detector Method and Apparatus is a method, called visual event detection, for detecting events that may occur in the field of view. An event can be an object passing through the field of view, and by using visual event detection the object can be detected without the need for a trigger signal.
Additional aspects of the Vision Detector Method and Apparatus will be apparent by a study of the figures and detailed descriptions given therein.
In order to obtain images from multiple perspectives, it is desirable that an object to be detected or inspected moves no more than a small fraction of the field of view between successive frames, often no more than a few pixels. According to the Vision Detector Method and Apparatus, it is generally desirable that the object motion be no more than about one-quarter of the FOV per frame, and in typical embodiments no more than 5% or less of the FOV. It is desirable that this be achieved not by slowing down a manufacturing process but by providing a sufficiently high frame rate. In an example system the frame rate is at least 200 frames/second, and in another example the frame rate is at least 40 times the average rate at which objects are presented to the vision detector.
An exemplary system is taught that can capture and analyze up to 500 frames/second. This system makes use of an ultra-sensitive imager that has far fewer pixels than prior art vision systems. The high sensitivity allows very short shutter times using very inexpensive LED illumination, which in combination with the relatively small number of pixels allows very short image capture times. The imager is interfaced to a digital signal processor (DSP) that can receive and store pixel data simultaneously with analysis operations. Using methods taught therein and implemented by means of suitable software for the DSP, the time to analyze each frame generally can be kept to within the time needed to capture the next frame. The capture and analysis methods and apparatus combine to provide the desired high frame rate. By carefully matching the capabilities of the imager, DSP, and illumination with the objectives of the invention, the exemplary system can be significantly less expensive than prior art machine vision systems.
The method of visual event detection involves capturing a sequence of frames and analyzing each frame to determine evidence that an event is occurring or has occurred. When visual event detection is used to detect objects without the need for a trigger signal, the analysis would determine evidence that an object is located in the field of view.
In an exemplary method the evidence is in the form of a value, called an object detection weight, that indicates a level of confidence that an object is located in the field of view. The value may be a simple yes/no choice that indicates high or low confidence, a number that indicates a range of levels of confidence, or any item of information that conveys evidence. One example of such a number is a so-called fuzzy logic value, further described therein. Note that no machine can make a perfect decision from an image, and so will instead make judgments based on imperfect evidence.
When performing object detection, a test is made for each frame to decide whether the evidence is sufficient that an object is located in the field of view. If a simple yes/no value is used, the evidence may be considered sufficient if the value is “yes”. If a number is used, sufficiency may be determined by comparing the number to a threshold. Frames where the evidence is sufficient are called active frames. Note that what constitutes sufficient evidence is ultimately defined by a human user who configures the vision detector based on an understanding of the specific application at hand. The vision detector automatically applies that definition in making its decisions.
When performing object detection, each object passing through the field of view will produce multiple active frames due to the high frame rate of the vision detector. These frames may not be strictly consecutive, however, because as the object passes through the field of view there may be some viewing perspectives, or other conditions, for which the evidence that the object is located in the field of view is not sufficient. Therefore it is desirable that detection of an object begins when an active frame is found, but does not end until a number of consecutive inactive frames are found. This number can be chosen as appropriate by a user.
Once a set of active frames has been found that may correspond to an object passing through the field of view, it is desirable to perform a further analysis to determine whether an object has indeed been detected. This further analysis may consider some statistics of the active frames, including the number of active frames, the sum of the object detection weights, the average object detection weight, and the like.
The method of dynamic image analysis involves capturing and analyzing multiple frames to inspect an object, where “inspect” means to determine some information about the status of the object. In one example of this method, the status of an object includes whether or not the object satisfies inspection criteria chosen as appropriate by a user.
In some aspects of the Vision Detector Method and Apparatus dynamic image analysis is combined with visual event detection, so that the active frames chosen by the visual event detection method are the ones used by the dynamic image analysis method to inspect the object. In other aspects of the Vision Detector Method and Apparatus, the frames to be used by dynamic image analysis can be captured in response to a trigger signal.
Each such frame is analyzed to determine evidence that the object satisfies the inspection criteria. In one exemplary method, the evidence is in the form of a value, called an object pass score, that indicates a level of confidence that the object satisfies the inspection criteria. As with object detection weights, the value may be a simple yes/no choice that indicates high or low confidence, a number, such as a fuzzy logic value, that indicates a range of levels of confidence, or any item of information that conveys evidence.
The status of the object may be determined from statistics of the object pass scores, such as an average or percentile of the object pass scores. The status may also be determined by weighted statistics, such as a weighted average or weighted percentile, using the object detection weights. Weighted statistics effectively weight evidence more heavily from frames wherein the confidence is higher that the object is actually located in the field of view for that frame.
Evidence for object detection and inspection is obtained by examining a frame for information about one or more visible features of the object. A visible feature is a portion of the object wherein the amount, pattern, or other characteristic of emitted light conveys information about the presence, identity, or status of the object. Light can be emitted by any process or combination of processes, including but not limited to reflection, transmission, or refraction of a source external or internal to the object, or directly from a source internal to the object.
One aspect of the Vision Detector Method and Apparatus is a method for obtaining evidence, including object detection weights and object pass scores, by image analysis operations on one or more regions of interest in each frame for which the evidence is needed. In an example of this method, the image analysis operation computes a measurement based on the pixel values in the region of interest, where the measurement is responsive to some appropriate characteristic of a visible feature of the object. The measurement is converted to a logic value by a threshold operation, and the logic values obtained from the regions of interest are combined to produce the evidence for the frame. The logic values can be binary or fuzzy logic values, with the thresholds and logical combination being binary or fuzzy as appropriate.
For visual event detection, evidence that an object is located in the field of view is effectively defined by the regions of interest, measurements, thresholds, logical combinations, and other parameters further described herein, which are collectively called the configuration of the vision detector and are chosen by a user as appropriate for a given application of the invention. Similarly, the configuration of the vision detector defines what constitutes sufficient evidence.
For dynamic image analysis, evidence that an object satisfies the inspection criteria is also effectively defined by the configuration of the vision detector.
Discussion of the Problem
It is clear from the above summary and from a detailed reading of Vision Detector Method and Apparatus that it is advantageous for a vision detector to maintain a high frame rate. As taught therein, the high frame rate can be achieved in part by capturing a frame simultaneously with the analysis of the previous frame, using for example a ping-pong arrangement, and by keeping the analysis time comparable to or below the capture time. While it is not required that the analysis time be kept comparable to or below the capture time, if analysis takes more time than capture then the frame rate will be lower.
In some applications it is desirable that the analysis time be, under certain conditions, significantly longer than the capture time, and yet the frame rate be determined by the capture time only, so that it remains high even under those conditions. The prior art method of triggered asynchronous capture with FIFO buffer, used by some machine vision systems, is not sufficient for vision detectors, where images may be captured and analyzed continuously, whether or not an object is present, and not in response to a trigger. Thus there is a need for improved methods and systems that can allow long analysis times while still capturing frames at the highest rate.
The invention teaches improvements on methods and systems taught in Vision Detector Method and Apparatus for detecting and inspecting objects in the field of view of a vision detector. The invention replaces the synchronous, overlapped image capture and analysis methods and systems taught therein with asynchronous capture and analysis methods and systems that permit higher frame rates than previously possible under conditions where the time to analyze frames exceeds the time to capture them.
The invention provides for capturing and analyzing a sequence of frames, where each frame is an image of the field of view of a vision detector. Some of these frames will correspond to periods of time, called inactive states, where no object appears to be present in the field of view, and others will correspond to periods of time, called active states, where an object does appear to be present in the field of view. There may also be periods of time, called idle states, where no frames are being captured and analyzed. Objects are detected and inspected in response to an analysis of the frames during both inactive and active states.
In an illustrative embodiment, capture and analysis are controlled by separate threads in a multi-threaded software environment. The capture and analysis threads run concurrently and asynchronously, communicating by means of various data structures further described below. The capture thread puts frames into a FIFO buffer, and the analysis thread removes them after analysis.
The invention recognizes that an arbitrary and potentially unlimited number of frames are captured and analyzed for each object, some in the active state and most in the inactive state. It is desirable to capture frames at the highest rate during active states, but frames cannot be captured as fast as possible all the time, because the FIFO would quickly overflow in many situations. It is also desirable that the analysis of frames does not lag too far behind their capture in inactive states, as further explained below, in order to maintain the utility of output signals synchronized to the mark time. Thus the invention recognizes that it is desirable to control frame capture differently depending on whether or not an object appears to be present in the field of view.
The invention provides a variety of methods and systems to prevent FIFO overflow, manage the lag time, maintain high frame rate, and provide other useful capabilities that will be apparent to one skilled in the art.
One aspect of the invention includes an analysis step or process that takes less time when no object appears to be present in the field of view than when a object does appear to be present, which is desirable for maintaining a high frame rate in inactive states. In an illustrative embodiment, analyzing a frame comprises one or more detection substeps and one or more inspection substeps. If the detection substeps reveal that no object appears to be present in a given frame, the inspection substeps can be skipped, resulting in quicker analysis for that frame. Furthermore, in many cases it is not necessary to perform all of the detection substeps in order to judge that no object appears to be present.
In an illustrative embodiment, the detection and inspection substeps are asynchronous, and capture, detection, and inspection are controlled by three separate threads in a multi-threaded environment.
Another aspect of the invention includes a step or process that limits the number of frames to be captured and analyzed in an active state, so that the FIFO does not overflow and the time needed to inspect the object is predictable.
Yet another aspect of the invention includes a step or process that limits the lag time between the capture and analysis of a frame in inactive states. This step or process may result in a higher frame rate during active states, when high frame rate is most desirable, than inactive states, when other considerations apply.
A further aspect of the invention includes a step or process that flushes (discards) without analysis certain frames that have been captured, because analysis of previous frames reveals that analysis of the flushed frames is not necessary to detect or inspect an object.
The invention will be more fully understood from the following detailed description, in conjunction with the accompanying figures, wherein:
Discussion of Prior Art
The trigger 120 is some event external to the vision system, such as a signal from a photodetector, or a message from a Programmable Logic Controller (PLC), computer, or other piece of automation equipment.
The image capture step 130 starts by exposing a two-dimensional array of photosensitive elements, called pixels, for a brief period, called the integration or shutter time, to an image that has been focused on the array by a lens. Each pixel measures the intensity of light falling on it during the shutter time. The measured intensity values are then converted to digital numbers and stored in the memory of the vision system.
During the analyze step 140 the vision system operates on the stored pixel values using methods well-known in the art to determine the status of the object being inspected. During the report step 150, the vision system communicates information about the status of the object to appropriate automation equipment, such as a PLC.
Trigger row 200 shows trigger signals input to the machine vision system that indicate that an object is present in the field of view, including example trigger 202. Note that trigger steps in trigger row 200 are equivalent to the trigger 120 steps of
Capture row 210 shows image capture steps, including example capture step 212 that captures image 3 in the illustrated sequence (the image numbers are arbitrary). Note that each trigger signal starts an image capture, so that example trigger 202 starts example capture step 212.
Analysis row 220 shows image analysis steps, including example analysis step 222 that analyzes image 3. Note that in the example of
FIFO row 230 shows the contents of the FIFO, including example FIFO state 244 that shows the FIFO containing images 2, 3, and 4. An image is added to the bottom of the FIFO at the end of a capture step, and removed from the top at the end of an analysis step. For example, at the end of example capture step 212, corresponding to first time mark 240, image 3 is added to the FIFO, and at the end of example analysis step 222, corresponding to second time mark 242 image 3 is removed from the FIFO.
Note that the timing of the steps in analysis row 220 is asynchronous with the timing of steps in capture row 210. Analysis proceeds whenever the FIFO is not empty, using the oldest (“first-in”) image shown at the top in FIFO row 230, regardless of what is currently being captured. This arrangement can handle a temporary condition where the analysis times are much longer than the object presentation intervals, as shown in the example of
Selected Teachings of Vision Detector Method and Apparatus
If capture and analysis are overlapped, the rate at which a vision detector can capture and analyze images is determined by the longer of the capture time and the analysis time. This is the “frame rate”.
The Vision Detector Method and Apparatus allows objects to be detected reliably without a trigger signal, such as that provided by a photodetector. Note that in
Referring again to
Each analysis step first considers the evidence that an object is present. Frames where the evidence is sufficient are called active. Analysis steps for active frames are shown with a thick border, for example analysis step 340. In an illustrative embodiment, inspection of an object begins when an active frame is found, and ends when some number of consecutive inactive frames are found. In the example of
At the time that inspection of an object is complete, for example at the end of analysis step 348, decisions are made on the status of the object based on the evidence obtained from the active frames. In an illustrative embodiment, if an insufficient number of active frames were found then there is considered to be insufficient evidence that an object was actually present, and so operation continues as if no active frames were found. Otherwise an object is judged to have been detected, and evidence from the active frames is judged in order to determine its status, for example pass or fail. A variety of methods may be used to detect objects and determine status within the scope of Vision Detector Method and Apparatus; some are described therein and many others will occur to those skilled in the art.
Once an object has been detected and a judgment made, a report may be made to appropriate automation equipment, such as a PLC, using signals well-known in the art. In such a case a report step similar to report 150 in
Note that the report 360 may be delayed well beyond the inspection of subsequent objects such as that corresponding to second portion 310. The vision detector uses well-known first-in first-out (FIFO) buffer methods to hold the reports until the appropriate time.
Once inspection of an object is complete, the vision detector may enter an idle step 380. Such a step is optional, but may be desirable for several reasons. If the maximum object rate is known, there is no need to be looking for an object until just before a new one is due. An idle step will eliminate the chance of false object detection at times when an object couldn't arrive, and will extend the lifetime of the illumination system because the lights can be kept off during the idle step.
In another embodiment, the report step is delayed in a manner equivalent to that shown in
Operation of the Present Invention
For first example analysis step 500, an object is found and inspected-all six detection and inspection substeps are executed. This corresponds to an active frame. Most frames are inactive, however—no object is found and the inspection substeps need not be carried out. In second example analysis step 510 the three detection substeps are executed, but no object is found and so the inspection substeps are not done, resulting in a shorter duration for the analysis.
Furthermore, in many frames it is possible to decide that no object is found without executing all of the detection substeps. In third example analysis step 520, D1 substep 530 reveals that no object is found, and so no other substeps need be executed. Thus the average time to decide that no object is found may be significantly shorter than the longest time to make that decision.
Following the teachings of Vision Detector Method and Apparatus, it is important to distinguish between finding an object in a given frame, and detecting an object based on the evidence of many frames. An object may be present in a given frame but not found, or appear to be found but not actually present, because viewing conditions in that frame are poor or misleading for some reason. The methods of visual event detection and dynamic image analysis are designed in part to prevent an isolated poor frame from producing false results. Thus it is typically the case that an object is found in a given frame if and only if one is actually present, but this is not guaranteed. The following discussions of the invention are based on examples where an object is found if and only if one is actually present, but the reader will understand that methods and systems herein described will still work even when that condition does not hold for every frame.
Referring to
As taught in Vision Detector Method and Apparatus:
Thus, for a frame where ObjectDetect Judge 630 determines that no object has been found, the logic output of AND Gate 622 is not needed and therefore the logic outputs of I1 Photo 610, I2 Photo 612, and I3 Photo 614 are not needed. Such a case may correspond to second example analysis step 510 or third example analysis step 520. For a frame where D1 Photo 600 produces a zero output, AND Gate 620 may decide that the logic outputs of D2 Photo 602 and D3 Photo 604 are not needed. Such a case may correspond to third example analysis step 520. In an alternate embodiment, the same decision may be reached if D1 Photo 600 produces a output less than some object detection threshold such as 0.5.
It will be understood that the foregoing examples are illustrative and not limiting. Similar examples can be made using any number of Photos and any combination of AND Gates and OR Gates, and with different rules for breaking an image analysis step into substeps and deciding which substeps to execute. Furthermore, while it is desirable that the duration of analysis steps is shortened when no object is detected, it is not necessary for practice of the invention, as will become clear below.
Note that the analysis steps are of varying duration, some shorter and some longer than the substantially fixed image capture time, due in part to decisions that no object has been found as explained above for
In the example of
The decision delay 760, measured from mark time 740 to decision time 742, will be somewhat variable from object to object. When a synchronized output signal, such as report 360 (
Note that the active frames 38-41, where an object is found and inspected, are of substantially longer duration than inactive frames as explained above. These frames are also of substantially longer duration than image capture, but as can been seen this has no effect on the frame rate, which is determined solely by the capture time. Without the present invention, the frame rate would have to be slowed down to match the analysis. The higher frame rate provides more images for dynamic image analysis and visual event detection, and greater accuracy for mark time calculation. Note that since mark time is calculated based on recorded frame capture times, it doesn't matter that the analysis is done much later.
Note that the analysis of inactive frames may also be of longer duration than image capture, such as for frame 33 corresponding to example analysis step 722, without effecting the frame rate.
Note further that decision time 742 happens somewhat sooner with the present invention than with the ping-pong capture/analyze arrangement taught in Vision Detector Method and Apparatus. With a ping-pong arrangement, capture of frame 43 would begin when analysis of frame 42 begins, but since the analysis of frame 42 is shorter than the capture of frame 43, the analysis of frame 43 would happen somewhat later than the arrangement of
In an illustrative embodiment, a vision detector is in an active state for intervals during which an object appears to be present in the field of view, an inactive state for intervals during which frames are being captured and analyzed to detect an object but none appears to be present, and an idle state for intervals during which frames are not being captured and analyzed. In the example of
With the present invention, capture and analysis are substantially asynchronous. Herein substantially asynchronous means that the relative timing of capture and analysis is not predetermined, but rather is determined dynamically by conditions unfolding in the field of view. There may be conditions wherein capture and analysis do proceed in what appears to be synchronization, or when capture and analysis are deliberately synchronized to achieve a desirable result, but these conditions are not predetermined and occur in response to what is happening in the field of view.
In an illustrative embodiment, a conventional FIFO buffer is used to hold frames, following an arrangement similar to that used for the prior art machine vision system of
Adding asynchronous capture/analysis with a FIFO buffer to a vision detector, however, is not sufficient to produce a practical method or system, particularly when visual event detection is being used. The problems that might arise are not obvious, nor are the solutions. The problems arise in part because an arbitrary and potentially unlimited number of frames are captured and analyzed for each object, some when the object is present in the field of view (active state) and most when no object is present (inactive state). There is no trigger signal to indicate that an object is present and therefore a frame should be captured. One cannot, however, simply capture frames as fast as possible all the time, because the FIFO would quickly overflow in many situations. Even when the FIFO does not overflow, if the analysis of frames lags too far behind their capture in certain situations, the decision delay 760 will become long and unpredictable, severely reducing the utility of output signals synchronized to the mark time.
The invention recognizes that it is desirable to control frame capture differently depending on whether or not an object appears to be present in the field of view. It is desirable for robustness and mark timing accuracy to capture frames as fast as possible during active states. While analysis of those frames may be lengthy because most are active frames with all detection and inspection substeps being executed, the number of frames to be captured and analyzed during an active state is predictable based on the expected speed of objects and the known size of the field of view, and can be controlled by appropriate configuration parameters so as not to exceed predefined limits. The ability to predict and control the active state frame count is part of a structure used in an illustrative embodiment to insure that the FIFO will not overflow, and that the decision delay is short and predictable.
Another part of the above structure used in an illustrative embodiment keeps the count of frames in the FIFO (which corresponds to the time lag from capture to analysis) predictable during an inactive state, so that the count is predictable when an active state begins. This is accomplished in the illustrative embodiment by providing that frame capture in the inactive state waits until the FIFO contains no more than a configurable number of frames. Since analysis is generally faster in an inactive state (no object is detected), it is typically the case that the FIFO stays nearly empty even at the maximum frame rate. If analysis takes longer than capture for some reason, due for example to some temporary condition or because object detection requires significant analysis, frame capture will wait as necessary to prevent it from getting too far ahead of analysis, and the frame rate will slow down. Note that it is usually desirable that the FIFO be nearly empty during an inactive state.
Yet another part of the above structure used in an illustrative embodiment provides that the FIFO be flushed (all frames discarded) during an idle state. Analysis during an active state may get significantly behind frame capture, with the FIFO nearly filling up, and flushing the FIFO insures that the next inactive state begins with an empty FIFO and with analysis fully caught up to frame capture.
In an illustrative embodiment, frame capture and analysis are controlled by a suitable programmable digital processor, using software instructions that provide for multi-threaded operation of conventional and well-known design. Two threads run concurrently, one for frame capture and one for analysis. It is desirable that the capture thread run at higher priority than the analysis thread. Other threads may be running as well, depending on the application. The threads share certain data structures that provide for communication and, when necessary, synchronization between the threads. These data structures reside in the memory of the programmable digital processor, and include the FIFO and the state previously discussed, as well as other items to be introduced below.
Idle test block 820 tests whether state 800 is idle. If idle, flush block 840 flushes FIFO 802, idle wait block 842 waits for a time (or encoder count) specified by idle interval 804, and signal block 844 sets state 800 to inactive to signal the analyze thread (further described below) that the idle interval has ended. If not idle, lag limit block 830 waits for either state 800 to be not inactive, or FIFO 802 to contain fewer than a number of frames specified by inactive lag limit 806. In an illustrative embodiment, inactive lag limit 806 is 3. Capture block 850 captures the next frame, and put block 852 puts it into FIFO 802.
Idle interval 804 specifies the length of idle step 750 (
FIFO wait block 920 waits for FIFO 802 to contain at least one frame, so that there is something to analyze, get block 922 gets the first-in (oldest) frame from FIFO 802, and detection analysis block 924 runs object detection substeps of frame analysis to compute an object detection weight d.
Detection test block 930 compares the object detection weight to detection threshold 900 to see if an object appears to be present (i.e. see if the frame is active). If so, first active test block 940 tests whether state 800 is active. If not active, the first active frame of a possible new object has been found, and active set block 942 sets state 800 to active and initialize block 944 initializes statistics 902. If state 800 was already active, active set block 942 and initialize block 944 are skipped. Inspection analysis block 950 runs object inspection substeps of frame analysis to compute an object pass score p, and update block 952 updates statistics 902 based on the object detection weight d and object pass score p for the current frame. Clear block 954 sets inactive frame count 904, which counts consecutive inactive frames found during an active state, to zero.
Statistics 902 contains various statistics of a set of active frames that might be useful in judging whether an object has actually been detected, and if so whether it passes inspection. Examples of useful statistics can be found in Vision Detector Method and Apparatus, and others will occur to one of ordinary skill in the art. Statistics 902 includes a count of the active frames found during the current active state, and may also include a count of all frames found during the current active state.
Limit test block 956 compares a frame count, part of statistics 902, to max frames parameter 906 to see if a sufficient number of frames has been seen to make an inspection decision, and to control the number of frames analyzed during an active state so that object detection and inspection will not take too long and FIFO 802 will not overflow.
If detection test block 930 judges that no object appears to be present (inactive frame), second active test block 960 tests whether state 800 is active. If so, termination test block 962 compares inactive frame count 904 to missing frames parameter 908 to see if enough consecutive inactive frames have been found to terminate object detection and inspection. If object detection and inspection will continue, increment block 966 adds 1 to inactive frame count 904. If object detection and inspection will terminate, object test block 970 examines statistics 902 to judge whether an object has actually been detected. If not, inactive set block 964 sets state 800 inactive, and the vision detector is ready to detect another object. If so, output block 972 computes a mark time and schedules appropriate output pulses to occur at a later time or encoder count synchronized with the mark time, or provides for other output reports as required. Idle set block 974 sets state 800 to idle to signal the capture thread that an idle step should begin, and inactive wait block 976 waits for state 800 to be inactive, which is a signal from the capture thread that FIFO 802 has been flushed and a new detection and inspection cycle can begin.
Max frames parameter 906 and missing frames parameter 908 can be set by means of a human-machine interface, such as that shown in
It is essential that the capture and analysis threads be designed according to good multi-threaded programming practices to avoid critical races. One must assume that execution of the threads is completely asynchronous. For example, one must assume that state 800 might be idle during lag limit block 830, even though it appears that the capture thread cannot get to that block if state 800 is idle. The analysis thread can change state 800 at any time, however. If lag limit block 830 waited for state≠active instead of state # inactive, which might seem an equivalent test, the capture thread could be in lag wait block 830 in the idle state, with the acquire thread waiting at inactive wait block 976, which would cause both threads to hang forever.
Note that capture and analysis are overlapped (happen at the same time), as they are in the timelines of
Note further that the inspection substeps of certain frames, for example frames 38 and 39, are spread out over multiple separate non-contiguous intervals of time. Inspection of frame 38, for example, occurs during four separate intervals contained within example interval 1034. These separate intervals do not correspond to individual inspection substeps, they simply represent time during which the processor is available to perform inspection, which is a lower priority than detection. In the example of
In the example of
In some embodiments, image capture may slow down simultaneous analysis somewhat, due to competition for access to memory (such is the case for the illustrative embodiment of
The advantages of the alternate illustrative embodiment of
The disadvantages of the alternate illustrative embodiment of
In an illustrative embodiment according to
Note that a simple FIFO, as used for the prior art machine vision system of
It is also possible to use two FIFOs. Frames would be added to the first FIFO by the capture thread, removed and added to the second FIFO by the detection thread, and removed from the second FIFO by the inspection thread. Pointer manipulation methods would be used to avoid actually copying any frames. These and other alternatives are straightforward programming tasks for one of ordinary skill in the art.
Illustrative Apparatus
The DSP 1300 can be any device capable of digital computation, information storage, and interface to other digital elements, including but not limited to a general-purpose computer, a PLC, or a microprocessor. It is desirable that the DSP 1300 be inexpensive but fast enough to handle a high frame rate. It is further desirable that it be capable of receiving and storing pixel data from the imager simultaneously with image analysis.
In the illustrative embodiment of
The high frame rate desired by a vision detector suggests the use of an imager unlike those that have been used in prior art vision systems. It is desirable that the imager be unusually light sensitive, so that it can operate with extremely short shutter times using inexpensive illumination. It is further desirable that it be able to digitize and transmit pixel data to the DSP far faster than prior art vision systems. It is moreover desirable that it be inexpensive and have a global shutter.
These objectives may be met by choosing an imager with much higher light sensitivity and lower resolution than those used by prior art vision systems. In the illustrative embodiment of
It is desirable that the illumination 1340 be inexpensive and yet bright enough to allow short shutter times. In an illustrative embodiment, a bank of high-intensity red LEDs operating at 630 nanometers is used, for example the HLMP-ED25 manufactured by Agilent Technologies. In another embodiment, high-intensity white LEDs are used to implement desired illumination.
In the illustrative embodiment of
As used herein an image capture device provides means to capture and store a digital image. In the illustrative embodiment of
As used herein an analyzer provides means for analysis of digital data, including but not limited to a digital image. In the illustrative embodiment of
As used herein an output signaler provides means to produce an output signal responsive to an analysis. In the illustrative embodiment of
As used herein a process refers to systematic set of actions directed to some purpose, carried out by any suitable apparatus, including but not limited to a mechanism, device, component, software, or firmware, or any combination thereof that work together in one location or a variety of locations to carry out the intended actions.
In an illustrative embodiment, various processes used by the present invention are carried out by an interacting collection of digital hardware elements and computer software instructions. These hardware elements include
In an illustrative embodiment the computer software instructions include those for carrying out actions described herein, and in Vision Detector Method and Apparatus, Parameter Setting Method and Apparatus, and Event Detection Method and Apparatus, for such functions as:
Furthermore, it will be understood by those skilled in the art that the above is a list of examples only. It is not exhaustive, and suitable computer software instructions may be used in illustrative embodiments to carry out any suitable process.
Examples of processes described herein include:
It will be understood by one of ordinary skill that there are many alternate arrangements, devices, and software instructions that could be used within the scope of the invention to implement an image capture device 1380, analyzer 1382, and output signaler 1384. Similarly, many alternate arrangements, devices, and software instructions could be used within the scope of the invention to carry out the processes described herein.
A variety of engineering tradeoffs can be made to provide efficient operation of an apparatus according to the present invention for a specific application. Consider the following definitions:
From these definitions it can be seen that
To achieve good use of the available resolution of the imager, it is desirable that b is at least 50%. For dynamic image analysis, n should be at least 2. Therefore it is further desirable that the object moves no more than about one-quarter of the field of view between successive frames.
In an illustrative embodiment, reasonable values might be b=75%, e=5%, and n=4. This implies that m≦5%, i.e. that one would choose a frame rate so that an object would move no more than about 5% of the FOV between frames. If manufacturing conditions were such that s=2, then the frame rate r would need to be at least approximately 40 times the object presentation rate p. To handle an object presentation rate of 5 Hz, which is fairly typical of industrial manufacturing, the desired frame rate would be at least around 200 Hz. This rate could be achieved using an KAC-9630 with at most a 3.3 millisecond shutter time, as long as the image analysis is arranged so as to fit within the 5 millisecond frame period. Using available technology, it would be feasible to achieve this rate using an imager containing up to about 40,000 pixels.
With the same illustrative embodiment and a higher object presentation rate of 12.5 Hz, the desired frame rate would be at least approximately 500 Hz. An KAC-9630 could handle this rate by using at most a 300 microsecond shutter.
In another illustrative embodiment, one might choose b=75%, e=15%, and n=5, so that m≦2%. With s=2 and p=5 Hz, the desired frame rate would again be at least approximately 500 Hz.
The foregoing has been a detailed description of various embodiments of the invention. It is expressly contemplated that a wide range of modifications and additions can be made hereto without departing from the spirit and scope of this invention. For example, the processors and computing devices herein are exemplary and a variety of processors and computers, both standalone and distributed can be employed to perform computations herein. Likewise, the imager and other vision components described herein are exemplary and improved or differing components can be employed within the teachings of this invention. The timing diagrams can all be modified or replaced with equivalents as appropriate for specific applications of the invention. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.
This application is a continuation-in-part of the following co-pending U.S. patent applications: Ser. No. 10/865,155, entitled METHOD AND APPARATUS FOR VISUAL DETECTION AND INSPECTION OF OBJECTS, by William M. Silver, filed Jun. 9, 2004, the teachings of which are expressly incorporated herein by reference, and referred to herein as the “Vision Detector Method and Apparatus”; Ser. No. 10/979,535, entitled METHOD FOR SETTING PARAMETERS OF A VISION DETECTOR USING PRODUCTION LINE INFORMATION, by Brian Mirtich and William M. Silver, filed Nov. 2, 2004, a continuation-in-part of Vision Detector Method and Apparatus, the teachings of which are expressly incorporated herein by reference, and referred to herein as the “Parameter Setting Method and Apparatus”; and Ser. No. 11/059,512, entitled Method and Apparatus for Automatic Visual Detection, Recording, and Retrieval of Events, by William M. Silver and Brian S. Phillips, filed Feb. 16, 2005, a continuation-in-part of Vision Detector Method and Apparatus, the teachings of which are expressly incorporated herein by reference, and referred to herein as the “Event Detection Method and Apparatus”.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10865155 | Jun 2004 | US |
| Child | 11094650 | Mar 2005 | US |
| Parent | 10979535 | Nov 2004 | US |
| Child | 11094650 | Mar 2005 | US |
| Parent | 10059512 | Jan 2002 | US |
| Child | 11094650 | Mar 2005 | US |
