Inspection apparatus method and apparatus comprising selective frame output

Description

FIELD OF THE INVENTION

The invention relates to inspection apparatuses generally and more particularly to visual inspection apparatuses.

BACKGROUND OF THE PRIOR ART

Inspection apparatuses can be used to develop streaming video image representation of areas to be inspected. In one embodiment an inspection apparatus can be used to inspect industrial equipment articles. At certain times during operation of an inspection apparatus an inspector may initiate a freeze frame control signal. When a freeze frame control signal is initiated, a buffered frame of image data retained in a frame buffer can continually be read out to a display. At other times during operation of an inspection apparatus an inspector may initiate a save frame control signal. When a save frame control signal is initiated, a buffered frame of image data retained in a frame buffer can be written to a memory location of a memory device for later use, e.g., a volatile memory device, a non-volatile memory device, or to a long term storage device.

In typical operation the first frame of image data having a capture initiation time subsequent to the time of initiation of a control signal to output a frame of image data to a display or memory is the frame that is subject to output. Unfortunately, the frame having the first capture initiation time subsequent to an initiation of a control signal to output a frame is not always a high quality frame of image data. If the inspection apparatus is being moved at the time of initiation of a control signal to output a frame of image data, low quality image may be saved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram illustrating a method that can be carried out with one of an inspection apparatus.

FIG. 2 is as electrical block diagram illustrating an exemplary set of circuits that can be incorporated in an inspection apparatus.

FIG. 3 is an alternative physical form view of an inspection apparatus.

FIG. 4 is an alternative physical form view of as inspection apparatus.

FIG. 5 is alternative electrical block diagram illustrating an exemplary set of circuits that can be incorporated in an inspection apparatus.

FIG. 6 is an alternative physical form view of an inspection apparatus.

FIG. 7 is an alternative physical form view of an inspection apparatus.

FIG. 8 is a timing diagram illustrating a timing of an initiation of a control signal to selectively output a frame of image data plotted against frame capture times.

FIG. 9 is a view of an inspection apparatus having a graphical user interface allowing selection of various frame selection algorithms that can be executed by an inspection apparatus.

FIG. 10 is a plot illustrating application of non-uniform digital gain in one embodiment.

DETAILED DESCRIPTION OF THE INVENTION

A simplified flow diagram illustrating a process for selectively outputting a high quality frame of image data is shown in FIG. 1. At block 10, a control signal to selectively output a frame of image data can be initiated. At block 20, apparatus 100 can process image data of one or more frames and can determine a motion parameter. In one embodiment, apparatus 100 can determine a motion parameter for each frame of image data processed. At block 30 apparatus 100 can selectively output a frame of image data responsively to processing at block 20. By selectively outputting a frame of image data subsequently to a frame retention control signal being initiated, the quality of the frame of image data that can be output can be improved. A technical effect of the processing described with reference to the flow diagram of FIG. 1 is to improve the quality of an output frame of image data. Accordingly, there is described herein, in one embodiment, a method for operating an inspection apparatus having an elongated inspection tube and an image sensor for generating image signals, said method comprising the steps of: initiating a control signal to selectively output a frame of image data; processing image data of one or more frames to determine a motion parameter; and subsequent to initiation of a control signal to selectively output a frame, outputting a frame of image data responsively to said processing to determine a motion parameter. In outputting a frame, an inspection apparatus can output to a display and/or an addressable memory location for later use a frame retained in a frame buffer that is continually written over during the course of operation of the apparatus. The frame buffer can be, e.g., an input frame buffer, an output frame buffer, or an accumulator frame buffer. The memory location to which the buffered frame can be written to can be a memory location of a memory device, e.g., a volatile memory device, a non-volatile memory device, or a storage device.

In some embodiments, image data that is subject to processing is frame image data captured subsequent to the time of initiation of a control signal to selectively output a frame of image data. In other embodiments, image data that is subject to processing is frame image data captured prior to the time of initiation of a control signal to selectively output a frame of image data. In other embodiments, frame image data subject to processing can comprise both frame image data of single frames having capture initiation times of subsequent to and prior to the time of initiation of a control signal to selectively output a frame of image data. Frame image data that is subject to processing for determining a motion parameter can comprise a full frame of image data or less than a full frame of image data. For example, in one embodiment a limited number of rows of image data can be processed. In another embodiment, a limited number of columns can be processed. In another embodiment, where image sensor 132 is provided to be capable of an interlaced readout mode, a single field (e.g., an odd-field or an even field) can be processed. When capturing a frame of image data, apparatus 100 need not simultaneously retain each image data element of a frame of image data. For example, when capturing a frame of image data subject to processing, apparatus 100 may buffer a limited amount of frame image data (e.g., pixel values corresponding to a few rows of pixels) at a given instant in time.

A motion parameter can be developed for each frame subject to processing. A motion parameter that can be developed for each frame having image data subject to processing can comprise a binary two-state parameter, i.e., a frame can be designated as being “in motion” or “motion free.” A motion parameter that is developed for each frame of image data subject to processing at block 20 can, in addition or in the alternative, comprise a qualitative measurement of motion. For example, a motion score can be ascribed to each frame of image data corresponding to the degree of motion determined to be present in the frame of image data.

A block diagram of an exemplary apparatus capable of supporting the above described processing is shown and described in connection with FIG. 2. Inspection apparatus 100 can include an elongated inspection tube 112 and a head assembly 114 disposed at a distal end of the elongated inspection tube. Head assembly 114 can include solid state image sensor 132 and imaging lens 140. Imaging lens 140 can focus an image onto an active surface of solid state image sensor 132. Imaging lens 140 can comprise, e.g., a lens singlet or a lens having multiple components, e.g., a lens doublet, a lens triplet. Solid state image sensor 132 can be, e.g., a CCD or CMOS image sensor. Solid state image sensor 132 can include a plurality of pixels formed in a plurality of rows and columns. Solid state image sensor 132 can provided on an integrated circuit. Image sensor 132 can generate image signals in the form of analog voltages representative of light incident on each pixel of the image sensor. Referring to further aspects of head assembly 114, image sensor 132 can be controlled so that image signals are clocked out from image sensor 132. Analog voltages representative of light incident on the various pixels of image sensor 132 can be propagated through signal conditioning circuit 136 along a cable, e.g., a coaxial cable disposed within elongated inspection tube 112. Head assembly 114 can include signal conditioning circuit 136 when conditions analog image signals for input to cable 138 and receives timing and control signals for control of image sensor 132. In one embodiment, image sensor 132 and signal conditioning circuit 136 can be mounted on a common circuit board 137. In the embodiment of FIG. 2 an imaging axis 250 of apparatus 100 extends through head assembly 114.

In the embodiment of FIG. 2, head assembly 114 of apparatus 100 at a distal end of inspection tube 112 comprises image sensor 132. Image sensor 132 of inspection apparatus 100 can, in one alternative embodiment be located at a position spaced apart from head assembly 114, and disposed at a position rearward of a proximal end of inspection tube 112. For example, image sensor 132 can be disposed in base assembly 105 interfaced to elongated inspection tube 112 as shown in FIG. 2. An imaging system fiber optic bundle (not shown) can be disposed in elongated inspection tube 112, and can terminate in head assembly 114. The apparatus can be configured so that such a fiber optic bundle relays image forming light rays from head assembly 114 to the spaced apart image sensor spaced apart from head assembly 114.

Various circuits disposed at a position spaced apart from head assembly 114 can receive and process image signals generated by image sensor 132. In one embodiment, various circuits receiving and processing image signals generated by image sensor 132 can be disposed in base assembly 105 interfaced to elongated inspection tube 112 as shown in FIG. 2. In the exemplary embodiment of FIG. 2, analog front end circuit 150 can include an analog gain circuit, an analog-to-digital converter, and a correlated double sampler and can receive analog image signals, digitize such signals, and transmit digitized image signals to digital signal processor 152 (DSP). DSP 152, in the embodiment shown, can be configured to perform such processing tasks as color matrix processing, gamma processing, and can process digital image signals into a standardized video format, wherein video signals are expressed in a standardized date format. By way of example, video signals output by DSP 152 can be in a BT656 video format and data carried in the video signal can have a 422YCRCB data format. DSP 152 can be in communication with a random access memory 160 through system bus 158. Referring to further aspects of an electrical circuit for inspection apparatus 100, apparatus 100 can include timing generator circuit 156 which can send timing and control signals to signal conditioning circuit 136 for input to image sensor 132 as well as to analog front end circuit 150 and DSP 152. As indicated by communication line labeled “to 136,” timing generator circuit 136 can send control signals such as exposure timing signals frame rate timing signals to signal conditioning circuit 136 for input to image sensor 132. In one embodiment, analog circuit front end 150, DSP 152, and timing generator circuit 156 can be provided on separate integrated circuits (ICs). In one embodiment, analog front end circuit 150, DSP 152, and tinting generator circuit 156 are provided as part of commercially available chips, e.g., an SS2 DSP chipset of the type available from SONY. While an analog to digital converter for converting analog image signals into digital form is described as being incorporated into front end circuit 150, such an analog to digital converter can be incorporated into an image sensor integrated circuit which commonly carries pixels of an image sensor and an analog to digital converter for digitizing analog image signals.

Referring to further aspects of apparatus 100, apparatus 100 can include DSP 180. DSP 180 can receive the formatted video output from DSP 152 for further processing. DSP 180 can be configured to perform a variety of processing tasks such as frame averaging, scaling, zoom, overlaying, merging, image capture, flipping, image enhancement, and distortion correction. DSP 180 can also be configured to perform motion detection as will be described more fully herein. In one embodiment, DSP 180 can be provided by a TMS32ODM642 Video/Imaging Fixed-Point Digital Signal Processor of the type available from TEXAN INSTRUMENTS. DSP 180 can be in communication with a volatile memory 161, e.g., RAM, a non-volatile memory 162, and storage memory device 164. Non-volatile memory 162 can be provided, e.g., by a flash memory device, an EEPROM memory device, or an EPROM memory device. Software for operating apparatus 100 can be retained in non-volatile memory 162 when apparatus 100 is not operating and such software can be loaded into RAM 161 when apparatus 100 is driven into an operating state. Apparatus 100 can include other types of storage memory. For example, a USB “thumb drive” can be plugged into serial I/O interface 172. A CompactFlash memory card can be plugged into parallel I/O interface 173. A memory of apparatus 100 can be regarded as including memory 161, 162, and 164, other storage memory, as well as internal buffer memories of DSP 152 and 180. Storage memory device 164 can be, e g., a hard drive or removable disk. RAM 161, non volatile memory 162, and storage device 164 can be in communication with DSP 180 via system bus 159. While DSP 152 and DSP 180 are shown as being provided on separate integrated circuits, the circuits of DSP 152 and DSP 180 could be provided on a single integrated circuit. Also, the functionalities provided by DSP 152 and DSP 180 could be provided by one or more general purpose microprocessor IC.

Apparatus 100 can be configured so that image signals are read out of image sensor 132 row by row until a frame of image signal including image signals corresponding to multiple pixels of image sensor 132 have been read out. Analog image signals read out from image sensor 132 am be converted into digital form by front end circuit 150. Front end circuit 150, in turn, can feed digitized frame image signals into DSP 152. DSP 152 can format the image signals into a specific format before feeding the digitized image signals for further processing to DSP 180. Digitized frame image signals can be referred to as frame image data.

Referring to further circuit components of the block diagram of FIG. 2, apparatus 100 can further include display 210, keyboard 214, and joystick 218. Keyboard 214 enables a user to initiate various control signals for the control of apparatus 100. Display 210 enables display of live video streaming images and other images to an inspector. For example, apparatus 100 can be controlled to switch from a live streaming video mode in which a live streaming video is being displayed on display 210 to a mode in which a still image is displayed on display 210. Apparatus 100 can be configured so that apparatus 100 can generate control signals to selectively output a frame responsively to an action by an inspector. Apparatus 100 can be configured so that an inspector can initiate a control signal to selectively output a frame of image data by actuating a designated button of keyboard 214. Control signals for selective output of a frame of image data can include, e.g., a freeze frame control signal, and a save frame control signal. Apparatus 100 can be configured so that when a freeze frame control signal is initiated, apparatus 100 can repeatedly (continuously) output a frame from a frame buffer to display 210. The frame buffer can be continuously overwritten during the course of operation of the apparatus. The frame buffer can be a buffer of RAM 161, and can be, e.g., an input frame buffer, an output frame buffer, or an accumulator frame buffer. Apparatus 100 can be configured so that when a “save frame” control signal is initiated, apparatus 100 can output a frame from a frame buffer to an addressable memory location for future access, e.g., a memory location of RAM 161, non-volatile memory 162 and/or storage device 164. A frame of image data saved responsively to initiation of a save frame control signal can be formatted into a standardized or known proprietary file format. During performance of an inspection procedure, an inspector may initiate a save frame control signal several times to save numerous frames relating to a work subject (e.g., an equipment article) being subject to an inspection. A sensor interface of apparatus 100 can include keyboard 214, joystick 218, and display 210.

In a further aspect, DSP 180 can be coupled to a serial I/O interface 172, e.g., an ETHERNET or USB interface and a parallel data interface, e.g., a CompactFlash interface or PCMCIA interface. DSP 180 can also be coupled to a wireless data communication interface 174, e.g., an IEEE 802.11 interface. Apparatus 100 can be configured to send frames of image data saved in a memory thereof to an external computer and can further be configured to be responsive to requests for frames of image data saved in a memory device of apparatus 100. Apparatus 100 can incorporate a TCP/IP communication protocol suite and can be incorporated in a wide area network including a plurality of local and remote computers, each of the computers also incorporating a TCP/IP communication protocol suite. With incorporation of TCP/IP protocol suite, apparatus 100 incorporates several transport layer protocols including TCP and UDP and several different layer protocols including HTTP and FTP.

Referring to further aspects of apparatus 100, apparatus 100 can include joystick 218 for controlling a positioning of head assembly 114. In one embodiment, articulation cables 222 can be incorporated in elongated inspection tube 112 to enable movement of head assembly 114 into a desired position so that a field of view of apparatus 100 can be changed. Joystick 218 can be in communication with DSP 180. Apparatus 100 can be configured so that control signals for controlling movement (articulation) of head assembly 114 are initiated by manipulating joystick 218. Apparatus 100 can be configured so that when joystick 218 is moved, DSP 180 receives a control signal from joystick 218 and sends corresponding motor control signals to articulation motor 220 to produce a desired movement of head assembly 114. Apparatus 100 can also be configured so that joystick 218 operates as a pointer controller for controlling a pointer displayed on display 210.

In another aspect, inspection apparatus 100 can include a light source 230, (e.g., an arc lamp or a bank of one or more LEDs), which, like circuits 150, 152, 156, and 180 can be disposed at a position spaced apart from head assembly 114. Apparatus 100 can also include an illumination fiber optic handle 232 receiving light emitted from light source 230. Fiber optic bundle 232 can be disposed in elongated inspection tube 112 so that fiber optic bundle 232 can relay light emitted from light source 230 through inspection tube 112 and to head assembly 114. A distal end of fiber optic bundle 232 can be interfaced to diffuser 234 for diffusing illumination light. Fiber optic bundle 232 and diffuser 234 can be arranged to project light over an area approximately corresponding to a field of view of image sensor 132. In a further aspect, light source 230 can be powered by a regulator 248 coupled to a power supply circuit 250. Power supply circuit 250 can be arranged to power circuit board 252 receiving various integrated circuits of apparatus 100 as well as buses 158, 159. Power supply circuit 250 can be interfaced to various alternative power sources, e.g., serial I/O power source 254, AC/DC transformer source 256 and rechargeable battery 258.

During operation to output a live streaming video image on display 210, incoming frames may be input into an input frame buffer of RAM 161, subject to processing by DSP 180 and output to an output frame buffer of RAM 161. Apparatus 100 can be configured so that when a freeze frame control signal is initiated, a frame of an output frame buffer is continually output to display 210. Apparatus 100 can also be configured so that when a save frame control signal is initiated, a frame of an input frame buffer is output to an addressable memory location of a memory device, e.g., RAM 161, non-volatile memory 162, or long term storage device 164.

Exemplary physical form views of the apparatus 100 shown in an electrical block view of FIG. 2 are shown in FIGS. 3 and 4. In the view of FIG. 3, apparatus 100 includes elongated inspection tube 112, and handset 101 incorporating housing 102, display 210, keyboard 214, and joystick 218. Circuits 150, 152, 156, 158, 160, 162, 164, 172 and 180 can be incorporated in housing 102. In the embodiment of FIG. 4, apparatus 100 includes a base unit 103 having a housing 104 incorporating a subset of the circuits shown in FIG. 2. For example, housing 104 can incorporate circuits 162, 164, 180, and 172. Handset 101 of FIGS. 3 and 4 can be a hand held handset sized and shaped to be held in a human hand. Skilled artisans will recognize that modifications to the circuit of FIG. 2 may be required if the circuits therein are described between a plurality of housings. For example, serial-deserializer circuits and twisted pair couplings as are explained in U.S. Provisional Patent Application No. 60/786,829 filed Mar. 27, 2006, incorporated herein by reference can be employed to transmit required video and control signals over distances of several feet at a high data rate. Additional circuits might be employed for communicating user initiated control signals generated at handset 101 to base unit 103. Additional circuits might also be employed for communicating image signals from base unit 103 to handset 101.

In one embodiment, apparatus 100 can have a base assembly 105, incorporating the components designated within dashed-in border 105 of FIG. 2. The components of base assembly 105 can be spread out into one or more housings. In the embodiment of FIG. 3, a single housing base assembly is provided. In the embodiment of FIG. 4, base assembly 105 comprises handset 101 and base unit 103. In another embodiment (not shown), base assembly 105 can include handset 101 and base unit. 103. However, rather than being interfaced to handset 101, elongated inspection tube 112 can be interfaced to base unit 103.

While methods described herein can be carried out utilizing an inspection apparatus having an elongated inspection tube, methods described herein can be carried out utilizing an inspection apparatus other than inspection apparatuses having an elongated inspection tube. In FIG. 5 there is shown an inspection apparatus 100 devoid of an elongated inspection tube. In the embodiment of FIG. 5, apparatus 100 is provisioned similarly to the embodiment of FIG. 2 except that imaging lens 140 as well as image sensor 132, signal conditioning circuit 136, and circuit board 137 are incorporated in base assembly 105. In the embodiment of FIG. 5, inspection apparatus 100 can include zoom lens motor 224 for varying a focal distance of inspection apparatus 100.

Base assembly 105, in the embodiment of FIG. 5, can take on a variety of forms. In the embodiment of FIG. 6 showing an inspection apparatus in the form of a hand held digital camera, base assembly 105 is provided by a hand held housing 102. The embodiment of FIG. 6 is similar to the embodiment of FIG. 3 except that whereas in the embodiment of FIG. 3 imaging lens 140 as well as image sensor 132, signal conditioning circuit 136 and circuit board 137 are incorporated in head assembly 114, imaging lens 140 as well as image sensor 132, signal conditioning circuit 136 and circuit board 137 in the embodiment of FIG. 6 are incorporated in base assembly 105 which in the embodiment of FIG. 6 is provided by a hand held housing 102. In the embodiment of FIG. 6, imaging axis 150 of apparatus 100 extends through hand held housing 102. In the embodiment of FIG. 7, inspection apparatus 100 is provided in the form of a pan-tilt-zoom (PTZ) camera. A PTZ camera as shown in FIG. 7 can be adapted to be mounted on a flat surface such as a ceiling, wall, table, or such as may be provided by a mounting platform of a robot. A PTZ camera as shown in FIG. 7 can be used in a variety of inspection applications such as robot inspections and surveillance monitoring. In the embodiment of FIG. 7, circuit components can be incorporated as shown in FIG. 5 such that imaging lens 140 as well as image sensor 132, signal conditioning circuit 136, and circuit board 137 are incorporated in base assembly 105 provided as shown in FIG. 7 by PTZ camera housing 106. In the embodiment of FIG. 7, imaging axis 250 can extend through a camera housing 106 as shown in the embodiment of FIG. 7. Referring still to the embodiment of FIG. 7 which can incorporate the circuit distribution of FIG. 5, inspection apparatus 100 can incorporate motor assembly 222 for controlling a pan and tilt of the inspection apparatus when provided by an inspection apparatus in the form of a PTZ camera. Keyboard 214, display 210, and joystick 218 (pointer controller) can be provided on board PTZ camera housing 106 as shown in FIG. 7, or else may be distributed into an inspection apparatus housing spaced apart from PTZ camera housing 106. As indicated by dashed-in laptop PC housing 107 of FIG. 7, circuits of FIG. 5 can be distributed into housings extraneous from housing 106. A PC incorporated in housing 107 can include various circuits such as DSP 180 and other circuits and can be configured to perform various image processing methods as described herein. A PC incorporated in housing 107 can be connected to the PTZ cameral incorporated in housing via IP network 109. Inspection apparatus 100 can also be provided by a camera of a machine vision system for use in an assembly process or other industrial process.

An inspection apparatus as described in connection with FIG. 2 can be configured to perform a method as described in connection with the flow diagram of FIG. 1. Accordingly, there is described herein, in one embodiment an inspection apparatus comprising: an elongated inspection tube; a two dimensional image sensor comprising a plurality of pixels, the two dimensional image sensor generating image signals corresponding to light incident on said pixels, wherein said inspection apparatus is configured to be capable of: (i) responding to a user-initiated control signal to selectively output a frame of image date; (ii) processing image data of one or more frames to determine a motion parameter; and (iii) outputting a frame of image data responsively to said processing.

Referring again to the flow diagram of FIG. 1, further details and variations of a method that can be performed with use of apparatus 100 are described. As described to with reference to the block diagram of FIG. 2, a user can initiate a control signal to selectively output a frame of image data by actuating a button of keyboard 214. Apparatus 100 can be configured to responsively generate selective frame output control signal responsively to an action of an inspector; e.g., actuating a button of keyboard 214.

At block 20, apparatus 100 can determine a motion parameter for each of at least one frame of image data. Apparatus 100, in one embodiment can be configured to execute block 20 subsequent to a time at which a control signal to selectively output a frame of image data at block 10 is initiated. Apparatus 100, in another embodiment can be configured so that apparatus 100 is executing processing block 20 at the time at which a control signal to selectively output a frame of image data is initiated. Apparatus 100 can be configured to execute processing block 20 in a number of different ways. For example, apparatus 100, in determining a motion parameter for each of one or more frames, can determine a binary (motion or motion free) motion parameter for a frame or can determine a qualitative motion parameter indicative of a degree of motion (e.g., can develop a motion scale from 0 to 9 where 0 is no motion and a 9 is maximum motion). Apparatus 100 can output a frame of image data responsively to initiation of a control signal to selectively output a frame of image data in a number of alternative possible ways. In one example, apparatus 100 can determine a motion parameter for a set of one to N frames. The set of one to N frames can be a set of successively captured frames of image data. Since the set of one to N frames can be captured within a certain time window, the set of one to N frames can be regarded as a “window.” In FIG. 8, there are shown timelines 270, 272 illustrating a time of initiation of a control signal to selectively output a frame of image data plotted against frame capture times. Timeline 270 shows a time of initiation of a frame control signal to selectively output a frame (e.g., a freeze frame or a same frame control signal). In the specific embodiment of FIG. 8, the time of initiation of a control signal is referenced by reference numeral 271. Timeline 272 shows times at which single frames (i.e., frames having image data corresponding to a specific frame readout period) of image data are captured. Apparatus 100 can be configured to continuously capture frames of image data as indicated by timeline 272, wherein the leading edges are the times where a first pixel value for a frame is buttered in DSP 180 and the falling edges are the times where a last pixel value of a frame is buffered by DSP 180. Referring to FIG. 8, apparatus 100 can determine a motion parameter for one or more frames after a time of initiation of a control signal to selectively output a frame, (i.e., frames within window 281). However, as indicated, apparatus 100 can be processing frames of image data to ascribe a motion parameter score to each frame of image data processed prior to time that a frame control signal to selectively output a frame is initiated. It should be noted that in buffering a frame of image data, a frame buffer of apparatus 100 such as a buffer of DSP 180 need not buffer each pixel value making up a frame of image data simultaneously. A frame buffer may be used to buffer only a subset of pixel values (e.g., a few rows) making up a frame of image data at a given time.

In one embodiment, each frame of image data that is captured by apparatus 100 prior to a time of initiation of a control signal to selectively output a frame can be subject to processing to determine a motion parameter. In one embodiment a set of frames subject to motion parameter processing comprises only frames (i.e., frames of window 282) captured prior to the time of initiation of a control signal to selectively output a frame. In another embodiment, a set of frames subject to motion parameter processing comprises both frames captured prior to the time of initiation of a control signal to selectively output a frame and frames captured subsequent to the time of initiation of a frame control signal to selectively output a frame, i.e., frames of window 283 in one example. In another embodiment, only frames of image data captured after initiation of a control signal to selectively output a frame are subject to processing.

Rules for determining which frame of N frames subject to processing is to be selectively output can be varied. Where apparatus 100 develops a binary motion parameter, apparatus 100 can selectively output the first frame subject to processing having a “motion free” designation. Where apparatus 100 develops a motion parameter score, apparatus 100 can, after processing a set of N frames, selectively output the frame having the lowest motion score. When processing image data of a set of N frames to selectively output a frame having a lowest motion score, apparatus 100 can buffer each incoming frame to a designated frame buffer location unless the incoming frame has a higher motion score that the currently buffered frame, in which case the incoming frame can be discarded. In such manner it is not necessary to buffer image data of N frames simultaneously when processing image data of N frames for purposes of determining the frame having the lowest motion score.

A number of possible configurations for processing image data of one or more frames and to selectively output frame responsively to the processing are summarized in Table A. As will be described further herein, apparatus 100 can be configured so that the exemplary configurations are user-selective. Apparatus 100 can be configured so that apparatus 100 activates a different algorithm for selectively outputting a frame of image data depending on which configuration is active.

TABLE A

Config-

Motion Parameter

uration
Window
Developed
Decision

1
Process one to N
Binary “motion”
Selectively output

frames captured after
or “motion free”
first frame

initiation of a

designated as

selective frame

motion free

output control signal

2
Process N frames
Qualitative motion
Selectively output

captured prior to
parameter
frame with lowest

initiation of a
developed
motion score

selective frame
(Score between

output control signal
0-9)

3
Process N frames
Binary “motion”
Selectively output

wherein some frames
or “motion free”
first frame

are captured before

designated as

initiation of a

motion free or Nth

selective frame

frame if no frame

output control signal

is designated as

and some frames are

being motion free

captured after

initiation of a

selective frame

output control signal

4
Process N frames.
Binary “motion”
Selectively output

The N frames may be
or “motion free”
first frame if F

captured before,

successive frames

during, or after

are designated as

initiation of a

being motion free

selective frame

output control signal

5
Process N frames.
Qualitative motion
Selectively output

The N frames may be
parameter
frame with lowest

captured before,
developed
motion score

during, or after
(Score between

initiation of a
0-9)

selective frame

output control signal

Methods for determining motion parameters are now described. Where image sensor 132 is of the type having an interlaced frame readout mode wherein an odd field of a frame is read out and then an even field, motion can be detected for by subtracting the even field from the odd field. The difference result can be scaled to yield a motion parameter score, e.g., between 0 and 9 wherein 0 is a score for no motion and 9 is a score for extensive motion. When head assembly 114 is not in motion, a motion parameter score can be expected to be about 0, though diagonal lines and/or horizontal edges may cause non-zero difference results. Even so, such analysis of frames including such diagonal lines and/or horizontal edges generally yields lower difference results for motion-free frames than for frames with motion. For converting a score to a binary motion parameter, i.e., “in motion” or “motion free” clarification, the score can be subject to thresholding (i.e, all scores below 2 are deemed to be motion free).

In another method for detecting motion, apparatus 100 can examine first and second successively captured frames. In examining first and second successively captured frames, apparatus 100 can locate one or more common edges in first and second frames, and can subtract pixel positions forming the common edge of the second frame from the first frame to derive a motion parameter scalable to scale, e.g., from 0 to 9. When head assembly 114 is not in motion, a motion parameter degree of motion score can be expected to be about 0. For converting a score to a binary motion parameter, i.e., “in motion” or “motion free” classification, the score can be subject to thresholding (i.e., all scores below 2 are deemed to be motion free).

In yet another method for detecting motion, apparatus 100 can examine image data of several frames in the form of first and second successively determined super frames. Each super frame can be determined by processing a set of M successively captured single frames. The processing can include, e.g., averaging or summing M successively captured frames, In one example, with a set of 10 successively captured frames, 0 to 9, a first super frame can be derived by averaging frames 0 through 4 and the second super frame can be derived by averaging frames 5 through 9. For conservation of processing requirements, accumulators may be employed for averaging. Super frames can be determined on a moving window basis. For example, during a first frame period, a first accumulator can retain the average or sum of frames N N . . . (N+4), and a second accumulator can retain the average or sum of frames (N+5) . . . (N+9). In a next frame period, the first accumulator can retain the average or sum of frames (N+1) . . . (N+5) and the second accumulator can retain the average or sum of frames (N+6) . . . (N+10). In examining first and second successively captured super frames, apparatus 100 can locate a common edge in first and second super frames, and subtract pixel positions forming the common edge of the second super frame from the first super frame to derive a motion parameter scalable to scale, e.g., from 0 to 9. When head assembly 114 is not in motion, a motion parameter score can be expected to be about 0. For converting a score to a binary motion parameter, i.e., “in motion” or “motion free” classification, the score can be subject to thresholding (i.e., all scores below 2 are deemed to be motion free). In another embodiment, apparatus 100 can be configured to subtract a super frame from a preceding super frame for purposes of developing a motion parameter. The inventor found that processing of super frames for motion detection is particularly advantageous under lower brightness (and, therefore, under higher expected noise) conditions. In one embodiment, apparatus 100 can be configured to process incoming image data for detection of brightness and can further be configured to automatically switch from a mode of operation in which single frames are processed for motion detection to a mode in which super frames are processed for motion detection when it is determined that brightness has fallen below a threshold brightness level.

In another embodiment, apparatus 100 when outputting a frame at block 30 responsively to a processing at block 20 can output a noise reduced frame. A noise reduced frame can be provided by processing a plurality of captured single frames as in a super frame. The plurality of frames that can be processed for providing a noise reduced frame can be successive frames or non-successive frames. For example, apparatus 100, as described in connection with FIG. 9 can be configured to process a set of 128 frames (F0 . . . F127) in determining a noise reduced frame. In processing the frames, apparatus 100 can determine if the frames are in motion, and can discard frames determined to be in motion. In discarding a frame, apparatus 100 can avoid inputting a frame determined to be in motion into an accumulator retaining a noise reduced frame. Apparatus 100 can also be configured to locate an edge in each frame and offset frames of the set of frames so that located edges are aligned. In offsetting a frame, apparatus 100 can offset a frame prior to accumulating the frame in an accumulator. Processing of a plurality of frames to determine a super frame can include averaging several frames or by otherwise utilizing image data from the plurality of frames to provide a noise reduced frame. A noise reduced frame provided by averaging a plurality of successively captured frames can be regarded as a frame averaged noise reduced frame. A noise reduced frame can also be regarded as a “filtered” frame. Where apparatus 100 is configured to output a noise reduced frame responsively to an initiation of a control signal to selectively output a frame, apparatus 100 can be configured to determine a binary “in motion” or “motion free” motion parameter classification for each frame of image data that is subject to processing. Apparatus 100 can further be configured so that when a first frame is determined to be motion free, an accumulator begins to maintain an accumulated average frame for a next set of F frames. Apparatus 100 can also be configured so that when F frames have been accumulated into the average frame accumulator, and with each frame being motion free, and with a selective frame output control signal being initiated within a predetermined time window of F motion free frames being accumulated, the accumulated average frame determined by averaging the F frames is saved into a memory of apparatus 100.

Accordingly, there is described herein, in one embodiment, a method for operating an inspection apparatus of the type having an elongated inspection tube and an image sensor generating image signals, said method comprising the steps of: configuring said inspection apparatus to process a plurality of frames to provide a noise reduced frame of image data; generating a control signal to selectively output a frame responsively to an action by a user to initiate said control signal to selectively output a frame, processing image data to determine a motion parameter; and, subsequent to generation of said control signal to selectively output a frame, outputting a noise reduced frame responsively to said processing to determine a motion parameter. The noise reduced frame can be output to a display and/or a memory device, e.g., device 161, 162 and/or 164.

An embodiment wherein a noise reduced frame (which can be regarded as a “filtered” frame) can be output responsively to an initiation of a control signal to selectively output a frame is summarized herein in Table A (See configuration 4 of Table A). In one embodiment, the candidate configurations summarized in Table A are user selectable.

As shown in FIG. 9, apparatus 100 can be configured to display designators (e.g., text or icons) corresponding to each configuration summarized in Table A. Apparatus 100 can also be configured so that an inspector can highlight a different designator 602, 604, 606, 608, 610 by moving joystick 218 and further so that an inspector can select a given configuration by actuating a button of keyboard 214 when designator corresponding to a designated configuration is highlighted. The user interface of FIG. 9 having displayed designators for each of several configurations can be regarded as a graphical user interface (GUI). Accordingly, there is described herein, in one embodiment, an inspection apparatus comprising: an elongated inspection tube; a two dimensional image sensor comprising a plurality of pixels, the two dimensional image sensor generating image signals corresponding to light incident on said pixels, a user interface enabling a user to activate first or second configurations, the apparatus being adapted so that when said first configuration is active, said apparatus selects a frame for outputting according to a first algorithm, said apparatus further being adapted so that when said second configuration is active said apparatus selects a frame for outputting according to a second algorithm, wherein said second algorithm is different than said first algorithm; wherein said inspection apparatus is further configured to allow a user to initiate a control signal to selectively output a frame of image data; and wherein said inspection apparatus subsequent to an initiation of a control signal to selectively output a frame of image data selects a frame of image data for outputting in a manner that varies depending on whether said first configuration or said second configuration has been selected.

In another aspect, inspection apparatus 100 can be configured to apply digital gain non-uniformly over a frame of image data. In one embodiment, apparatus 100 can be configured to determine position dependent digital gain parameters for pixel values of a frame of image data and to apply the determined position dependent digital gain parameters in determining pixel values of a frame of image data for outputting a display and/or a memory device. The frame of image data for which non-uniform digital gain parameters (non-uniform and offset parameters) can be determined can be a frame corresponding to a field of view of apparatus 100.

Inspection apparatuses are often used to capture frames of image data representing shiny surfaces. When a frame of image data representing a shiny surface is captured, illumination tends to reflect off the shiny surface causing what is often termed an over-bloomed bright spot in a frame of image data. In that bright spots will affect an overall brightness level used to determine applied digital gain and/or exposure parameters according to an imaging parameter determining algorithm, the presence of over-bloomed bright spots can lead to applied exposure period parameters and/or analog gain being too low, resulting in a frame of image data that is too dark in all but the area of an over-bloomed bright spot.

For addressing the problem of over-bloomed bright spots, inspection apparatus 100 can be configured to apply digital gain non-uniformly over a frame of image data in order to selectively brighten a frame of image data in areas other than a bright spot without substantial or without any brightening of a frame of image data in an area about a bright spot. Inspection apparatus 100 can also be configured to apply offsets non-uniformly over a frame of image data in order to reduce a washout effect of a frame of image data.

An exemplary method for outputting a frame of image data utilizing a set of position dependent non-linear digital gain values is as follows:

- 1. Add up luminance (e.g., gray scale) values for pixel positions within a region surrounding each pixel position (e.g., a 16×16 pixel position area) to obtain a regional brightness value for each pixel position.
- 2. Provide a lookup table mapping regional sum values to digital gain values (parameter). The lookup table, in one embodiment, can map larger digital gain values to smaller regional brightness values and zero or near-zero digital gain values to larger regional brightness values.
- 3. Determine a position dependent digital gain value utilizing the lookup table for each pixel position.
- 4. For each pixel position multiply the original pixel value by the determined digital gain value.

The result of applying non-uniform digital gain values determined responsively to the determination of regional brightness values is described in greater detail with reference to the plot of FIG. 10 showing pixel values through an arbitrary line of pixel positions within a bright region of pixel positions.

Referring to plot 702, line 704 indicates pixel brightness values for a row of pixels which is relatively dark at the left side and gradually brightens across the line. At the right side of the row, the brightness values clipped at the maximum possible value (e.g., 255 in an 8 bit pixel value frame), as shown by line 710. Dotted line 706 indicates a would-be pixel values if gain were applied uniformly, and bold line 708 indicates pixel values where non-uniform digital gain as described herein is applied. Referring to plot 702 it is seen with reference to the original image data 704 that several pixel values may be clipped at the peak 710 pixel value (indicating a possible over-bloomed bright spot). However, referring to image data 706 after application of a uniform digital gain parameter, several additional pixels can be clipped at the peak pixel value, resulting in loss of contrast information useful to an inspector. Referring to image data 708 after application of non-uniform digital gain, digital gain may be applied to increase pixel values in the non-clipped portions of the row; however, substantially no additional pixel values are clipped at the maximum pixel value. Accordingly, by application of the non-uniform gain parameters determined responsively to a determination of regional brightness values, clipping of additional pixel values is substantially avoided. According to the method described herein, wherein non-uniform digital gain parameters are determined responsively to a determination of regional brightness values, pixel positions of relatively darker pixel valued regions of a frame (darker regions) can have applied thereto digital gain parameters which would result in clipping of pixel values of pixel positions of relatively brighter pixel value regions (brighter regions) of a frame. Also, pixel positions of a relatively bright pixel value region of a frame can have applied thereto digital gain parameters smaller in value than the digital gain parameters applied to the relatively darker pixel position region. Application of the relatively smaller digital gain parameters determined responsively to a determination of a regional brightness value within a region will result in clipping of a fewer number of pixel values than would have been clipped by application of uniform gain parameter sufficient to render darker pixel values visibly brighter. A region can be regarded herein as a set of positionally adjacent pixel positions, e.g., a block of 16×16 positionally adjacent pixel positions.

In addition, another lookup table can be provided to provide mapping between regional sum values and a set of offset values (parameters). Such mapping can map larger offset values to smaller regional brightness values and little or no offset values to larger regional brightness values to reduce a “washout effect” when only digital gain is used. For example, when gain is applied to a frame, fine detail transitions might be amplified, but nevertheless, may not be rendered highly visible if the image data forming a transition has high white values (e.g., the human eye has difficulty in perceiving differences in differentiated but high gray scale values). For example, a human eye may have difficulty in perceiving an edge formed by an edge comprising 220 and 250 white level pixel values (the “washout effect”). The washout effect can be addressed by applying an offset, e.g., subtracting 100 from the area of the transition so that it is represented by pixel values having white levels of 120 and 150. For improving a quality of an image, offset can be applied non-uniformly by mapping pixel positions to offset parameters as indicated. For example, so that a white spot retains its appearance as a white spot in a frame of image data having offsets applied, it would not be desirable to have offsets applied to a white spot. By application of non-uniform offset pixel values of pixel positions that are in relatively dark regions prior to application of gain can be reduced by an offset so that transitions represented therein can be rendered more visible to an observer (e.g., an inspector). Pixel values of pixel positions of relatively bright regions prior to application of gain can have relatively little offset applied so that they are represented in accordance with their original brightness levels.

In one embodiment, the frame of image data to which non-uniform digital gain and/or offset parameters are applied can be a buffered frame of image data of an output frame buffer for output to display in a streaming video display.

In another embodiment, the frame of image data to which non-uniform digital gain and/or offset parameters are applied can be a frame of image data output to a memory device from a frame buffer (e.g., an input frame buffer) in response to an initiation of save frame control signal.

In another embodiment, the frame of image data to which non-uniform digital gain and/or offset parameters can be applied can be a frame of image data output to a display from a frame buffer (e.g., an output frame buffer) responsively to a processing of image data for determination of a motion parameter.

In another embodiment, the frame of image data to which non-uniform digital gain and/or offset parameters can be applied can be a noise reduced frame provided by processing of several frames and retained in art accumulator buffer as described herein. A noise reduced frame provided by processing of several single frames can be output to a display responsively to a processing of image data for determination of a motion parameter. Such a noise reduced frame to which non-uniform digital gain and/or offset parameters can be applied can also be frame that is output to a memory device in response to an initiation of a save frame control signal. By applying non-uniform digital gain and/or offset parameters to a frame provided by processing of several single frames, noise is reduced prior to the application of the digital gain. Thus, the applied digital gain tends to make image details more visible without creating a high noise level as may occur when digital gain is applied to a single potentially noisy frame. Additionally, where non-uniform digital gain and/or offset parameters are applied to a frame provided by processing several single frames, the accumulation of frames effectively increases the dynamic range available (such as from an 8-bit single frame to a 16-bit accumulator) allowing the application of digital gain without reducing the number of achievable output levels as described previously with uniform digital gain. It may further be desirable to use different digital gain and offset tables based on the number of accumulated frames such that lower gains are applied when few frames are accumulated and higher gains are applied when more frames are accumulated. This approach minimizes the amplification of image noise when few frames are accumulated while allowing significant enhancement with little noise once many frames have been accumulated. It also provides a gradual transition in the image appearance which is generally preferred over abrupt changes as would be seen if no enhancement were applied while in motion and full enhancement were applied when motion stops and frame accumulation begins.

A small sample of the methods of an apparatus described herein are as follows.

There is also described (A1) A method for operating an inspection apparatus having an elongated inspection tube and an image sensor for generating image signals, said method comprising the steps of: (a) initiating a control signal to selectively output a frame of image data; (b) processing image data of one or more frames to determine a motion parameter; and (c) subsequent to initiation of said control signal to selectively output a frame, outputting a frame of image data responsively to said processing referred to in step (b). There is also described (A2) The method of claim A1, wherein said processing step (b) comprises processing less than a full frame of image data. There is also described (A3) The method of claim A1, wherein said control signal is a freeze frame control signal. There is also described (A4) The method of claim A1, wherein said control signal is a save frame control signal. There is also described (A5) The method of claim A1, wherein said outputting step includes the step of outputting a single frame. There is also described (A6) The method of claim A1, wherein said processing includes processing of frames of image data captured prior to a time of initiation of said control signal. There is also described (A7) The method of claim A1, wherein said processing includes processing of frames of image data captured subsequent to a time of initiation of said control signal. There is also described (A8) The method of claim A1, wherein said motion parameter is a parameter classifying a frame of image data as an “in motion” frame or a “motion free” frame. There is also described (A9) The method of claim A1, wherein said motion parameter is a qualitative parameter indicating a degree of motion. There is also described (A10) The method of claim A1, wherein said frame of image data responsively output in step (c) is a noise reduced frame of image data provided by processing of several single frames. There is also described (A11) The method of claim A1, wherein said frame of image data responsively output in step (c) is a noise reduced frame having applied thereto at least one of a set of non-uniform gain parameters and a set of non-uniform offset parameters, the at least one of a set of non-uniform gain parameters and a set of non-uniform offset parameters being provided for brightening darker areas of said noise reduced frame without substantial or without any brightening of a bright spot of said noise reduced frame of image data. There is also described (A12) The method of claim A1, wherein said frame of image data responsively output in step (c) is a noise reduced frame having applied thereto a set of position dependent non-uniform gain parameters determined responsively to a determination of regional brightness values in a frame of image data. There is also described (A13) The method of claim A1, wherein said outputting step includes the step of outputting a noise reduced filtered frame. There is also described (A14) The method of claim A1, wherein said outputting step includes the step of outputting a noise reduced filtered frame, the noise reduced filtered frame being provided by processing of several frames. There is also described (A15) The method of claim A1, wherein said processing includes processing of a plurality of frames to determine a super frame. There is also described (A16) The method of claim A1, wherein said method includes the steps of detecting brightness of incoming image data and responsively to a determination that brightness has fallen below a threshold brightness processing super frames in step (b) for determination of a motion parameter. There is also described (A17) The method of claim A1, wherein said outputting step includes the step of repeatedly outputting a buffered frame buffered in a frame buffer to a display. There is also described (A18) The method of claim A1, wherein said outputting step includes the step of outputting a buffered frame to a memory device.

There is also described (B1) An inspection apparatus comprising: an elongated inspection tube; (a) a two dimensional image sensor comprising a plurality of pixels, the two dimensional image sensor generating image signals corresponding to light incident on said pixels; (b) a user interface enabling a user to activate first or second configurations, the apparatus being adapted so that when said first configuration is active, said apparatus selects a frame for outputting according to a first algorithm, said apparatus further being adapted so that when said second configuration is active said apparatus selects a frame for outputting according to a second algorithm, wherein said second algorithm is different than said first algorithm; (c) wherein said inspection apparatus is further configured to allow a user to initiate a control signal to selectively output a frame of image data; and (d) wherein said inspection apparatus subsequent to an initiation of a control signal to selectively output a frame selects a frame of image data for outputting in a manner that varies depending on whether said first configuration or said second configuration has been selected. There is also described (B2) The inspection apparatus of claim B1, wherein said inspection apparatus, when said first configuration is active, determines a binary motion parameter, and when said second configuration is active, determines a qualitative motion parameter. There is also described (B3) The inspection apparatus of claim B1, wherein said user interface is a graphical user interface displaying a designator for each of said first and second configurations.

There is also described (C2) An inspection apparatus comprising: (a) an elongated inspection tube; (b) a two dimensional image sensor comprising a plurality of pixels, the two dimensional image sensor generating image signals corresponding to light incident on said pixels; (c) wherein said inspection apparatus is configured to be capable of: (i) responding to a user-initiated control signal to selectively output a frame of image data; (ii) processing image data of one or more frames to determine a motion parameter; and (iii) selectively outputting a frame of image data responsively to said processing. There is also described (C2) The apparatus of claim C1, wherein said control signal is a freeze frame control signal. There is also described (C3) The apparatus of claim C1, wherein said control signal is a save frame control signal. There is also described (C4) The apparatus of claim C1, wherein said outputting step includes the step of outputting a single frame. There is also described (C5) The apparatus of claim C1, wherein said processing includes processing of frame image data captured prior to a time of initiation of said control signal to selectively output a frame of image data. There is also described (C6) The apparatus of claim C1, wherein said processing includes processing of frame image data captured subsequent to a time of initiation of said control signal to selectively output a frame of image data. There is also described (C7) The apparatus of claim C1, wherein said motion parameter is parameter classifying a frame of image data as a motion frame or a motion free frame. There is also described (C8) The apparatus of claim C1, wherein said motion parameter is a parameter indicating a qualitative level of motion. There is also described (C9) The apparatus of claim C1, wherein said frame is a noise reduced frame generated by processing multiple frames of image data. There is also described (C10) The apparatus of claim C1, wherein said processing comprises averaging multiple frames of image data. There is also described (C11) The apparatus of claim C1, wherein said apparatus in outputting said frame outputs a noise reduced frame provided by processing of several single frames.

There is also described (D1) A method for operating an inspection apparatus of the type having an elongated inspection tube and an image sensor generating image signals, said method comprising the steps of: (a) configuring said inspection apparatus to process a plurality of frames to provide a noise reduced frame of image data, (b) generating a control signal to selectively output a frame responsively to an action by a user to initiate said control signal; (c) processing image data to determine a motion parameter for a plurality of frames of image data; and (d) subsequent to generation of said control signal to selectively output a frame, outputting said noise reduced frame of image data responsively to said processing step (c). There is also described (D2) The method of claim D1, wherein said apparatus in executing said processing step (c) and said outputting step (d) determines whether several successive frames are motion free and outputs a said noise reduced frame if each of said several frames are determined to be motion free. There is also described (D3) The method of claim D1, wherein said configuring step comprises the step of configuring said inspection apparatus to average a plurality of frames to provide a noise reduced frame of image data.

This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to make and use the invention. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.

Claims

1. An inspection apparatus, comprising: an image sensor comprising a plurality of pixels, wherein the image sensor is configured to generate image signals based at least in part on light incident on each of the plurality of pixels;a processor communicatively coupled to the image sensor, the processor configured to receive the image signals and switch a mode of operation for processing the image signals to determine a motion parameter based on detecting a brightness level of the image signals, wherein the processor is programmed to: determine a first image frame corresponding to first image signals generated by the image sensor;determine a second image frame corresponding to second image signals generated by the image sensor;detect a first edge present in the first image frame and the second image frame;align the first edge in the first image frame and the second image frame by offsetting the first image frame, the second image frame, or both;determine a first super frame after aligning the first edge based at least in part on a sum of the first image frame and the second image frame to facilitate reducing noise in the first super frame compared to the first image frame and the second image frame;responsive to determining the detected brightness level of the first and second image signals equals or exceeds a threshold brightness level, determining the motion parameter based on processing the first and second image frames; andresponsive to determining the detected brightness level of the first or second image signals is less than the threshold brightness level, determining the motion parameter based on processing the first super frame.
2. The inspection apparatus of claim 1, comprising: a base assembly, wherein the processor is disposed in the base assembly; andan inspection tube coupled to the base assembly at a proximal end and the image sensor at a distal end.
3. The inspection apparatus of claim 1, wherein the processor is programmed to: determine a third image frame corresponding to third image signals generated by the image sensor;determine a second edge present in the third image frame and the first super frame;align the second edge in the third image frame and the first super frame by offsetting the first super frame; anddetermine a second super frame after aligning the second edge based at least in part on accumulation of the third image frame with the first super frame to facilitate reducing noise in the second super frame compared to the third image frame.
4. The inspection apparatus of claim 3, wherein, to align the second edge, the processor is programmed to: determine a first pixel position of the second edge in the first super frame;determine a second pixel position of the second edge in the third image frame; andoffset the first super frame based at least in part on difference between the first pixel position and the second pixel position.
5. The inspection apparatus of claim 1, comprising: a frame buffer communicatively coupled to the processor, wherein the frame buffer is configured to store image data corresponding with the first super frame; anda memory device communicatively coupled to the processor;wherein the processor is programmed to store the image data from the frame buffer to a memory location in the memory device when a save frame control signal is received.
6. The inspection apparatus of claim 1, comprising: a frame buffer communicatively coupled to the processor, wherein the frame buffer is configured to store first image data corresponding with an average image frame; anda display communicatively coupled to the frame buffer;wherein the processor is programmed to: determine the first image data by averaging the first image frame and the second image frame; andinstruct the frame buffer to continuously output the first image data to enable the display to continuously display the average image frame.
7. The inspection apparatus of claim 1, wherein, to align the first edge, the processor is programmed to: determine a first pixel position of the first edge in the first image frame;determine a second pixel position of the first edge in the second image frame; andoffset only the second image frame based at least in part on difference between the first pixel position and the second pixel position when the second image frame is captured after the first image frame.
8. The inspection apparatus of claim 1, wherein the processor is programmed to: determine a third image frame corresponding to third image signals generated by the image sensor, wherein the first edge is present in the third image frame;align the first edge in the first image frame, the second image frame, the third image frame by offsetting the first image frame, the second image frame, the third image frame, or any combination thereof; anddetermine the first super frame after aligning the first edge based at least in part on a sum of the first image frame, the second image frame, and the third image frame the second image frame, and the third image frame, or both to facilitate reducing noise in the first super frame compared to the third image frame.
9. The inspection apparatus of claim 1, wherein the processor is programmed to: determine a third image frame corresponding to third image signals generated by the image sensor, wherein the third image signals are generated after the first image signals and the second image signals;determine a fourth image frame corresponding to fourth image signals generated by the image sensor, wherein the fourth image signals are generated after the first image signals and the second image signals;determine a second edge present in the third image frame and the fourth image frame;align the second edge in the third image frame and the fourth image frame by offsetting the third image frame, the fourth image frame, or both; anddetermine a second super frame after aligning the second edge based at least in part on a sum of the third image frame and the fourth image frame to facilitate reducing noise in the second super frame compared to the third image frame and the fourth image frame.
10. The inspection apparatus of claim 9, wherein the processor is programmed to: determine a third edge present in the first super frame and the second super frame, wherein the second super frame is determined after the first super frame;align the third edge in the first super frame and the second super frame by offsetting the first super frame, the second super frame, or both; anddetermine a third super frame after aligning the third edge based at least in part on accumulation of the second super frame with the first super frame to facilitate reducing noise in the third super frame compared to the first super frame and the second super frame.
11. The inspection apparatus of claim 1, wherein the processor is programmed to: instruct the image sensor to generate the first image signals and the second image signals at a first brightness level;instruct the image sensor to generate third image signals at a second brightness level different from the first brightness level;determine a third image frame corresponding to the third image signals; andgenerate a high dynamic range image by utilizing at least a portion of the first super frame and at least a portion of the third image frame, wherein the high dynamic range image comprises more unsaturated pixels than the first super frame.
12. The apparatus of claim 1, wherein the motion parameter includes a binary motion parameter and a qualitative motion parameter.
13. A remote visual inspection system, comprising: an imager comprising an array of light-sensitive pixels, wherein the imager is configured to generate image data based at least in part on light level sensed by each pixel in the array of light-sensitive pixels;a processor communicatively coupled to the image sensor, the processor configured to receive the image data and switch a mode of operation for processing the image data to determine a motion parameter based on detecting a brightness level of the image data, wherein the processor is programmed to: determine a plurality of image frames each based on corresponding image data generated by the image sensor;determine a first baseline image by summing pixel values at a first one or more pixel locations in each of the plurality of image frames;determine a first image frame based on first image data generated by the imager after the first baseline image is determined;determine a first pixel location of a first sharp brightness transition in the first baseline image;determine a second pixel location of the first sharp brightness transition in the first image frame;align the first baseline image with the first image frame when the first pixel location and the second pixel location differ by applying a first rotation of the first baseline image, a first translation of the first baseline image, or both on the first image frame;determine a second baseline image by summing pixel values at a second one or more pixel locations in each of the plurality of image frames and the first image frame after the first baseline image is aligned to facilitate reducing noise in the second baseline image compared to the first image frame and the first baseline image;responsive to determining the detected brightness level of the first image data equals or exceeds a threshold brightness level, determining the motion parameter based on processing the first image frame; andresponsive to determining the detected brightness level of the first image data is less than the threshold brightness level, determining the motion parameter based on processing the second baseline image.
14. The remote visual inspection system of claim 13, wherein the processor is programmed to: determine a second image frame based on second image data generated by the imager after the second baseline image is determined;determine a third pixel location of a second sharp brightness transition in the second baseline image;determine a fourth pixel location of the second sharp brightness transition in the second image frame;align the second baseline image with the second image frame when the third pixel location and the fourth pixel location differ by applying a second rotation of the second baseline image, a second translation of the second baseline image, or both on the second image frame; anddetermine a third baseline image by summing pixel values at a third one or more pixel locations in each of the plurality of image frames, the first image frame, and the second image frame after the second baseline image is aligned to facilitate reducing noise in the third baseline image compared to the second image frame and the second baseline image.
15. The remote visual inspection system of claim 13, wherein the processor is programmed to: instruct the imager to capture the plurality of image frames and the first image frame at a first brightness level;instruct the imager to generate second image data at a second brightness level different from the first brightness level;determine a second image frame based on the second image data; andgenerate a high dynamic range image by utilizing at least a portion of the second baseline image and at least a portion of the second image frame, wherein the high dynamic range image comprises more unsaturated pixels than the second baseline image.
16. The remote visual inspection system of claim 13, wherein the remote visual inspection system comprises an endoscope, a borescope, a pan-tilt zoom camera, a push camera, or any combination thereof.
17. The system of claim 13, wherein the motion parameter includes a binary motion parameter and a qualitative motion parameter.
18. A method for operating a remote visual inspection system, comprising: instructing, using one or more processors, an image sensor of the remote visual inspection system to capture a plurality of image frames at a first brightness level and generate image signals corresponding to each image frame of the plurality of image frames, the one or more processors configured to receive the image signals and switch a mode of operation for processing the image signals to determine a motion parameter based on a brightness level of the image signals;determining, using the one or more processors, a first super frame by accumulating pixel values in each of the plurality of image frames;responsive to determining the first brightness level of the image signals corresponding to the plurality of image frames equals or exceeds a threshold brightness level, determining the motion parameter based on processing the plurality of image frames;responsive to determining the first brightness level of the image signals corresponding to the plurality of image frames is less than the threshold brightness level, determining the motion parameter based on processing the first super frame;instructing, using the one or more processors, the image sensor to capture a first image frame after the plurality of image frames at the first brightness level;detecting, using the one or more processors, whether first movement of the image sensor occurred between capture of the plurality of image frames and capture of the first image frame; andwhen the first movement of the image sensor is detected: adjusting, using the one or more processors, the first super frame by rotating, translating, or both the first super frame to offset the first movement; anddetermining, using the one or more processors, a second super frame by accumulating pixel values in each of the plurality of image frames with corresponding pixel values in the first image frame.
19. The method of claim 18, wherein: detecting whether the first movement of the image sensor occurred comprises: determining a first pixel location in the first super frame at which a physical feature is present;determining a second pixel location in the first image frame at which the physical feature is present; anddetermining that the first movement occurred when the first pixel location and the second pixel location are different; andadjusting the first image frame comprises rotating, translating, or both the first super frame based at least in part on difference between the first pixel location and the second pixel location.
20. The method of claim 18, comprising: instructing, using one or more processors, the image sensor of the remote visual inspection system to capture a second image frame;detecting, using the one or more processors, whether second movement of the image sensor occurred between capture of the plurality of image frames and capture of the second image frame; andwhen the second movement of the image sensor is detected: adjusting, using the one or more processors, the second super frame by rotating, translating, or both the second super frame to offset the second movement;determining, using the one or more processors, a third super frame by accumulating pixel values in each of the plurality of image frames with corresponding pixel values in the first image frame and corresponding pixel values in the second image frame when the second image frame is captured at the first brightness level; anddetermining, using the one or more processors, a high dynamic range image by utilizing at least a portion of the second super frame and at least a portion of the second image frame when the second image frame is captured at a second brightness level different from the first brightness level.
21. The method of claim 18, wherein the motion parameter includes a binary motion parameter and a qualitative motion parameter.

CROSS REFERENCE TO RELATED APPLICATIONS

Under 35 U.S.C. § 120, this application is a continuation of U.S. patent application Ser. No. 14/292,648 filed on May 30, 2014 and a continuation of U.S. patent application Ser. No. 14/331,084 filed on Jul. 14, 2014, which is a continuation of U.S. patent application Ser. No. 11/642,569 filed Dec. 20, 2006, now U.S. Pat. No. 8,810,636, which are all incorporated by reference herein in their entirety for all purposes.

US Referenced Citations (234)

Number	Name	Date	Kind
2929866	Melamed	Mar 1960	A
3188386	Byatt	Jun 1965	A
3493748	Tajima	Feb 1970	A
3515800	Ebihara et al.	Jun 1970	A
3623126	Newhouse	Nov 1971	A
3783190	Gaebele	Jan 1974	A
3845242	Richeson, Jr. et al.	Oct 1974	A
3934081	Schumacher	Jan 1976	A
3980819	Schwartz	Sep 1976	A
4115804	Morton et al.	Sep 1978	A
4354749	Hosoda	Oct 1982	A
4410914	Siau	Oct 1983	A
4433346	Stoffel et al.	Feb 1984	A
4442452	Kurata et al.	Apr 1984	A
4476494	Tugaye et al.	Oct 1984	A
4491865	Danna et al.	Jan 1985	A
4516153	Krull et al.	May 1985	A
4546381	Kurata et al.	Oct 1985	A
4547809	Southgate	Oct 1985	A
4608606	Levine	Aug 1986	A
4622954	Arakawa et al.	Nov 1986	A
4646723	Arakawa	Mar 1987	A
4658956	Takeda et al.	Apr 1987	A
4668978	Gokita	May 1987	A
4682219	Arakawa	Jul 1987	A
4692608	Coooer et al.	Sep 1987	A
4695878	Levine et al.	Sep 1987	A
4700693	Lia et al.	Oct 1987	A
4706654	Ogiu et al.	Nov 1987	A
4720178	Nishioka et al.	Jan 1988	A
4727859	Lia	Mar 1988	A
4733937	Lia et al.	Mar 1988	A
4735501	Ginsburgh et al.	Apr 1988	A
4741327	Yabe	May 1988	A
4745470	Yabe et al.	May 1988	A
4745471	Takamura et al.	May 1988	A
4747327	Itoh et al.	May 1988	A
4757805	Yabe	Jul 1988	A
4758891	Hitchcock et al.	Jul 1988	A
4777385	Hartmeier	Oct 1988	A
4777524	Nakajima et al.	Oct 1988	A
4779130	Yabe	Oct 1988	A
4786965	Yabe	Nov 1988	A
4787369	Allred, III et al.	Nov 1988	A
4790294	Allred, III et al.	Dec 1988	A
4794912	Lia	Jan 1989	A
4796607	Allred, III et al.	Jan 1989	A
4807026	Nishioka et al.	Feb 1989	A
4809680	Yabe	Mar 1989	A
4862253	English et al.	Aug 1989	A
4868644	Yabe et al.	Sep 1989	A
4868646	Tsuji	Sep 1989	A
4873572	Miyazaki et al.	Oct 1989	A
4887154	Wawro et al.	Dec 1989	A
4890159	Ogiu	Dec 1989	A
4895431	Tsujiuchi et al.	Jan 1990	A
4901143	Uehara et al.	Feb 1990	A
4909600	Ciarlei et al.	Mar 1990	A
4913369	Lia et al.	Apr 1990	A
4918521	Yabe et al.	Apr 1990	A
4933757	Kanno et al.	Jun 1990	A
4941454	Wood et al.	Jul 1990	A
4941456	Wood et al.	Jul 1990	A
4962751	Krauter	Oct 1990	A
4969034	Salvati	Nov 1990	A
4979035	Uehara et al.	Dec 1990	A
4980763	Lia	Dec 1990	A
4989581	Tamburrino et al.	Feb 1991	A
4993405	Takamura et al.	Feb 1991	A
4998182	Krauter et al.	Mar 1991	A
4998971	Fukunishi	Mar 1991	A
5014515	Krauter	May 1991	A
5014600	Krauter et al.	May 1991	A
5018436	Evangelista et al.	May 1991	A
5018506	Danna et al.	May 1991	A
5019121	Krauter	May 1991	A
5021888	Kondou et al.	Jun 1991	A
5023570	Matsumoto	Jun 1991	A
5032913	Hattori et al.	Jul 1991	A
5034888	Uehara et al.	Jul 1991	A
5047848	Krauter	Sep 1991	A
5052803	Krauter	Oct 1991	A
5061995	Lia et al.	Oct 1991	A
5066122	Krauter	Nov 1991	A
5070401	Salvati et al.	Dec 1991	A
5114636	Evangelista et al.	May 1992	A
5140975	Krauter	Aug 1992	A
5155585	Ishikawa	Oct 1992	A
5164824	Ieoka et al.	Nov 1992	A
5191879	Krauter	Mar 1993	A
5202758	Tamburrino	Apr 1993	A
5203319	Danna et al.	Apr 1993	A
5220198	Tsuji	Jun 1993	A
5262856	Lippman	Nov 1993	A
5270810	Nishimura et al.	Dec 1993	A
5275152	Krauter et al.	Jan 1994	A
5278642	Danna et al.	Jan 1994	A
5278656	Hynecek et al.	Jan 1994	A
5291151	Fujiwara et al.	Mar 1994	A
5305098	Matsunaka et al.	Apr 1994	A
5314070	Ciarlei	May 1994	A
5323899	Strom et al.	Jun 1994	A
5345339	Knieriem et al.	Sep 1994	A
5347989	Monroe et al.	Sep 1994	A
5365331	Tamburrino et al.	Nov 1994	A
5369446	Parker et al.	Nov 1994	A
5373317	Salvati et al.	Dec 1994	A
5377669	Schulz	Jan 1995	A
5387928	Nishimura et al.	Feb 1995	A
D358471	Cope et al.	May 1995	S
5418566	Kameishi	May 1995	A
5420644	Watanabe et al.	May 1995	A
5435296	Vivenzio et al.	Jul 1995	A
5441043	Wood et al.	Aug 1995	A
5469210	Noguchi et al.	Nov 1995	A
5515449	Tsuruoka et al.	May 1996	A
5568190	Noguchi et al.	Oct 1996	A
5587736	Walls	Dec 1996	A
5614943	Nakamura et al.	Mar 1997	A
5617136	Iso et al.	Apr 1997	A
5631695	Nakamura et al.	May 1997	A
5633675	Danna et al.	May 1997	A
5667474	Nishimura et al.	Sep 1997	A
5675378	Takasugi et al.	Oct 1997	A
5694530	Goto	Dec 1997	A
5696850	Parulski et al.	Dec 1997	A
5701155	Wood et al.	Dec 1997	A
5730129	Darrow et al.	Mar 1998	A
5733246	Forkey	Mar 1998	A
5734418	Danna	Mar 1998	A
5749362	Funda et al.	May 1998	A
5749827	Minami	May 1998	A
5754313	Pelchy et al.	May 1998	A
5764809	Nomami et al.	Jun 1998	A
5776049	Takahashi	Jul 1998	A
5779625	Suzuki et al.	Jul 1998	A
5788628	Matsuno et al.	Aug 1998	A
5796427	Suzuki et al.	Aug 1998	A
5821532	Beaman et al.	Oct 1998	A
5823958	Truppe	Oct 1998	A
5840014	Miyano et al.	Nov 1998	A
5857963	Pelchy et al.	Jan 1999	A
5860912	Chiba	Jan 1999	A
5877803	Wee et al.	Mar 1999	A
5883610	Jeon	Mar 1999	A
5946034	Cortiula	Aug 1999	A
5951464	Takahashi et al.	Sep 1999	A
5953013	Shimizu	Sep 1999	A
5956416	Tsuruoka et al.	Sep 1999	A
5964696	Mihalca et al.	Oct 1999	A
5983120	Groner et al.	Nov 1999	A
5990471	Watanabe et al.	Nov 1999	A
6002430	McCall et al.	Dec 1999	A
6008939	Hebert	Dec 1999	A
6046777	Patton	Apr 2000	A
6075555	Street	Jun 2000	A
6083152	Strong	Jul 2000	A
6084461	Colbeth et al.	Jul 2000	A
6088612	Blair	Jul 2000	A
6097848	Salvati	Aug 2000	A
6108005	Starks et al.	Aug 2000	A
6120435	Eino et al.	Sep 2000	A
6172361	Holberg et al.	Jan 2001	B1
6184514	Rezende et al.	Feb 2001	B1
6191809	Hori et al.	Feb 2001	B1
6211904	Adair et al.	Apr 2001	B1
6266430	Rhoads	Jul 2001	B1
6310647	Parulski et al.	Oct 2001	B1
6313456	Miyashita et al.	Nov 2001	B1
6373523	Jang	Apr 2002	B1
6428650	Chung	Aug 2002	B1
6461298	Fenster et al.	Oct 2002	B1
6468201	Burdick	Oct 2002	B1
6472247	Andoh et al.	Oct 2002	B1
6483535	Tamburrino et al.	Nov 2002	B1
6494739	Vivenzio et al.	Dec 2002	B1
6538732	Drost et al.	Mar 2003	B1
6547721	Higuma et al.	Apr 2003	B1
6590470	Burdick	Jul 2003	B1
6628713	Kojima	Sep 2003	B1
6721361	Covell et al.	Apr 2004	B1
6830545	Bendall	Dec 2004	B2
6842196	Swift et al.	Jan 2005	B1
6953432	Schiefer	Oct 2005	B2
7134993	Lia et al.	Nov 2006	B2
7239805	Uyttendaele et al.	Jul 2007	B2
7605854	Terzioglu	Oct 2009	B2
7626614	Marcu	Dec 2009	B1
7744528	Wallace et al.	Jun 2010	B2
8072507	Fuh et al.	Dec 2011	B2
8107083	Bendall et al.	Jan 2012	B2
8213676	Bendall	Jul 2012	B2
8368749	Lambdin et al.	Feb 2013	B2
9137454	Yang et al.	Sep 2015	B2
9204113	Kwok et al.	Dec 2015	B1
9424632	Patil et al.	Aug 2016	B2
20030067544	Wada	Apr 2003	A1
20040036775	Watson, Jr. et al.	Feb 2004	A1
20040183900	Karpen et al.	Sep 2004	A1
20040212808	Okawa et al.	Oct 2004	A1
20040215413	Weldum et al.	Oct 2004	A1
20050013501	Kang	Jan 2005	A1
20050050707	Scott et al.	Mar 2005	A1
20050063610	Wu	Mar 2005	A1
20050069207	Zakrzewski	Mar 2005	A1
20050099504	Nayar et al.	May 2005	A1
20050128355	Kang	Jun 2005	A1
20050129108	Bendall et al.	Jun 2005	A1
20050162643	Karpen	Jul 2005	A1
20050163402	Aiso	Jul 2005	A1
20050275725	Olsson	Dec 2005	A1
20050281520	Kehoskie et al.	Dec 2005	A1
20050286802	Clark	Dec 2005	A1
20060050983	Bendall et al.	Mar 2006	A1
20060072903	Weldum et al.	Apr 2006	A1
20060082646	Abe et al.	Apr 2006	A1
20060140600	Suda	Jun 2006	A1
20060158523	Estevez et al.	Jul 2006	A1
20060164511	Krupnik	Jul 2006	A1
20060187324	Lin	Aug 2006	A1
20060245618	Boregowda	Nov 2006	A1
20070014445	Lin	Jan 2007	A1
20070097266	Souchard	May 2007	A1
20080088711	Border	Apr 2008	A1
20090225333	Bendall et al.	Sep 2009	A1
20100034426	Takiguchi et al.	Feb 2010	A1
20120002899	Orr et al.	Jan 2012	A1
20120105673	Morales	May 2012	A1
20130293682	Zouda	Nov 2013	A1
20130308012	Fukutomi	Nov 2013	A1
20140232929	Ichikawa	Aug 2014	A1
20140307117	Feng et al.	Oct 2014	A1
20150051489	Caluser et al.	Feb 2015	A1
20150161797	Park et al.	Jun 2015	A1

Foreign Referenced Citations (19)

Number	Date	Country
4573101	Oct 2001	AU
2405434	Oct 2001	CA
1798264	Jul 2006	CN
4203630	Aug 1992	DE
602005003749	Dec 2008	DE
1290636	Mar 2003	EP
1649801	Apr 2006	EP
S-59215772	Dec 1984	JP
S-63292119	Nov 1988	JP
04-259442	Sep 1992	JP
07-246184	Sep 1995	JP
11-032986	Feb 1999	JP
11249031	Sep 1999	JP
2000107120	Apr 2000	JP
2001095757	Apr 2001	JP
2003534837	Nov 2003	JP
2006115964	May 2006	JP
2006186592	Jul 2006	JP
WO-0175798	Oct 2001	WO

Non-Patent Literature Citations (3)

Entry
Jill M. Boyce, “Noice Reduction of Image Sequences Using Adaptive Motion Compensated Frame Averaging”, 1992, IEEE, p. 1-4.
Teller et al., “Calibrated, registered images of an extended urban area”, Proceedings 2001 IEEE Conference on Computer Vision and Patiern Recognition. CVPR 2001. Los Alamitos, CA, IEEE Comp. Soc, US, vol. No. 1, pp. 813-828, Dec. 8, 2001.
Clancy et al., “Stroboscopic illumination scheme for seamless 3D endoscopy”, Advanced Biomedical and Clinical Diagnostic Systems X, Proc. of SPIE, vol. No. 8214, Issue No. 1, pp. 1-6, Mar. 8, 2012.

Related Publications (1)

	Number	Date	Country
	20170223273 A1	Aug 2017	US

Continuations (3)

	Number	Date	Country
Parent	14331084	Jul 2014	US
Child	15483830		US
Parent	14292648	May 2014	US
Child	14331084		US
Parent	11642569	Dec 2006	US
Child	14292648		US

Inspection apparatus method and apparatus comprising selective frame output

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

Field of Search

US

International Classifications

Abstract