Patent Application
20170061230
1. Field of the Invention
The invention relates to a signal process apparatus that processes an image input from an endoscope apparatus, and an endoscope system comprising the same.
2. Description of the Related Art
For an endoscope apparatus having an electronic endoscope apparatus, a function of generating and displaying a still image from a video endoscope image has been known. Such a still image is generated by using image data of a plurality of fields. Accordingly, if a subject or an endoscope apparatus moves, a still image with a large color-shift is generated. For example, Jpn. Pat. Appln. KOKAI Publication No. 2006-149483 provides a suggestion for avoiding color-shift when displaying a still image. Jpn. Pat. Appln. KOKAI Publication No. 2006-149483 discloses an electronic endoscope apparatus that determines a degree of color-shift, which is a shift between images of different colors, based on the symbol of difference in pixel values between adjacent pixels in the same horizontal line of image data, each having a different color obtained by the field sequential method, i.e., the change in pixel value. The electronic endoscope apparatus of Jpn. Pat. Appln. KOKAI Publication No. 2006-149483 corrects a color-shift by shifting image data so that the color-shift is minimized.
According to a first aspect of the invention, a signal process apparatus comprises: a contour extraction unit that acquires a plurality of image data items generated by sequential imaging of a subject, and extracts contour components from the acquired plurality of image data items; a shift amount detector that compares shift amounts of contour components extracted by the contour extraction unit between image data items included in each of a plurality of image sets, each of the image sets having a predetermined number of the image data items, and detects a shift amount between image data items included in each of the image sets; and an image selection unit that selects at least one image set among the image sets based on a detection result of the shift amount detector.
According to a second aspect of the invention, an endoscope system comprises: an imaging unit that acquires a plurality of image data items by sequential imaging of a subject; a contour extraction unit that extracts contour components from the plurality of image data items acquired at the imaging unit; a shift amount detector that compares shift amounts of contour components extracted by the contour extraction unit between image data items included in each of a plurality of image sets, each of the image sets having a predetermined number of the image data items, and detects a shift amount between image data items included in each of the image sets; and an image selection unit that selects at least one image set among the image sets based on a detection result of the shift amount detector.
Advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out hereinafter.
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention, and together with the general description given above and the detailed description of the embodiments given below, serve to explain the principles of the invention.
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
An endoscope system 1 includes an endoscope apparatus 2, a light source apparatus 3, a signal process apparatus 4 and a monitor 5. The endoscope apparatus 2 images inside a body of a subject and generates an image signal for the subject. The light source apparatus 3 is connected to the endoscope apparatus 2 through a light guide 36, and supplies illumination light to the endoscope apparatus 2. The signal process apparatus 4 processes an image signal generated at the endoscope apparatus 2. The monitor 5 displays an endoscope image which is an image of the inside of the body of the subject, for example, based on the image signal processed at the signal process apparatus 4. In the following, each block of the endoscope system 1 will be explained.
The endoscope apparatus 2 includes an insertion portion 21 and a cable 22. The insertion portion 21 is a portion of the endoscope apparatus 2 which is to be inserted inside the body of the subject. The insertion portion 21 is provided with an objective lens 211 and an imager 212 at a distal end. The objective lens 211 is an optical system to form an image from the subject at the imager 212. The imager 212 has two-dimensionally arranged pixels. Each pixel is a photo diode, for example, and generates an electrical signal (image signal) in accordance with the amount of applied light through the objective lens 211. As mentioned above, the endoscope system 1 generates an endoscope image by the field sequential method. Accordingly, the imager 212 performs imaging multiple times during the period of displaying one frame of an endoscope image. In each of the multiple times of imaging, an illumination light of a different color is applied to the subject. The imager 212 generates an image signal having only information of one of the different colors for each time of imaging. In the embodiment, a red (R) illumination light, a green (G) illumination light, and a blue (B) illumination light are sequentially applied. If the R illumination light is applied to the subject, an image signal having only information of R is generated, if the G illumination light is applied to the subject, an image signal having only information of G is generated, and if the B illumination light is applied to the subject, an image signal having only information of B is generated.
The cable 22 is connected to the signal process apparatus 4 through a connector not shown in the drawings. A drive signal line 212a and an output signal line 212b are provided inside of the cable 22. The drive signal line 212a is connected to each pixel of the imager 212. The drive signal line 212a is arranged to be connected to a drive controller 46 of the signal process apparatus 4 when the insertion portion 21 is connected to the signal process apparatus 4 through a connector not shown in the drawings. In this case, the imager 212 performs imaging of the subject in accordance with a drive signal from the drive controller 46 through the drive signal line 212a. The output signal line 212b is also connected to each pixel of the imager 212. The output signal line 212b is arranged to be connected to a preprocessor 41 of the signal process apparatus 4 when the insertion portion 21 is connected to the signal process apparatus 4 through the connector not shown in the drawings. In this case, the imager 212 outputs an image signal to the preprocessor 41 through the output signal line 212b.
The light source apparatus 3 includes a light source 31, a waveguide 32, a light control unit 34, a condenser lens 35, and the light guide 36.
The light source 31 includes a plurality of light sources such as RLED31R, GLED31G, and BLED31B. RLED31R is an LED that emits a red (R) light beam. GLED31G is an LED that emits a green (G) light beam. BLED31B is an LED that emits a blue (B) light beam. The waveguide 32 is, for example, an optical fiber which is optically coupled to each of RLED31R, GLED31G, and BLED31B. The waveguide 32 guides an illumination light beam emitted from each of RLED31R, GLED31G, and BLED31B to the condenser lens 35. The light control unit 34 controls the operation of RLED31R, GLED31G, and BLED31B synchronized with the imaging operation of the imager 212 controlled by the drive controller 46. In the field sequential method, the light control unit 34 sequentially drives RLED31R, GLED31G, and BLED31B for each field so that RLED31R, GLED31G, and BLED31B emit a light beam in a field-sequential manner. The condenser lens 35 condenses a light beam emitted from the waveguide 32 and applies the condensed light beam to the light guide 36. The light guide 36 applies the light beam condensed by the condenser lens 35 to the endoscope apparatus 2. The light beam is emitted from the endoscope apparatus 2 and applied to the subject. The subject is imaged in this manner, and each of an R image signal, a G image signal and a B image signal is obtained at the imager 212. The light source apparatus 3 may be provided with a white light source (a lamp or a white LED) and an RGB rotation filter so that the field sequential method is accomplished by sequentially inserting the RGB filter to the light path for each field, instead of providing with RLED, GLED, and BLED. The light source 31 may be capable of allowing the RLED31R, GLED31G, and BLED31B to emit light simultaneously, or emitting visible light directly from the white light source, to emit white light. This illumination method is called a simultaneous-type method, for example.
The signal process apparatus 4 includes the preprocessor 41, a storage 42, an image processor 43, a still image selection unit 44, a display controller 45, a display controller 46, a controller 47, and an operating unit 48.
The preprocessor 41 performs various kinds of analog processing such as analog gain adjustment, and analog noise elimination of an image signal. In addition, the preprocessor 41 also converts an image signal subjected to analog processing into image data which is a digital signal. In the following explanation, a set of image data items each having a different color obtained by multiple imagings by the imager 212 is referred to as an image set. The image set is a set of multiple items of image data used for generating one item of a color endoscope image. In the present embodiment, an image data set is composed of three items of image data. That is, the three items of image data are R image data having only red (R) information, G image data having only green (G) information, and B image data having only blue (B) information.
For the endoscope system, it is desirable that the endoscope apparatus 2 and the signal process apparatus 4 are insulated by an insulation circuit. The insulation circuit is provided, for example, at the preprocessor 41. The insulation circuit includes a circuit that wirelessly transmits a signal, such as a photocoupler, for example.
The storage 42 temporarily stores image data obtained by the preprocessor 41. The storage 42 has a storage capacity required for storing multiple image sets.
The image processor 43 performs image processing to each image data item included in an image set stored in the storage 42. The image processor 43 performs image processing for video display to an input image data when displaying video, and performs image processing for still image display to an input image data when displaying a still image. The image processor 43 includes a synchronization circuit, a luminance/color difference isolation circuit, a gradation correction circuit, a color correction circuit, a scaling circuit, and an enhancement circuit, for example. The synchronization circuit synchronizes three items of image data included in an image set obtained in accordance with the field sequential method to generate a synchronized image data. The synchronized image data is image data in which each pixel has information of multiple types of color components. The gradation correction circuit performs gradation correction to the synchronized image data. The color correction circuit performs color correction to the synchronized image data. The scaling circuit enlarges or reduces the size of the synchronized image data in accordance with the display size of the monitor 5 to generate a display image data. The enhancement circuit performs processing to enhance the contour of the subject in the display image data, or to enhance the structure of a blood vessel, for example, when the subject is a blood vessel.
The still image selection unit 44 performs image selection for still image display by using image data included in an image set stored in the storage 42. The still image selection unit 44 includes a contour extraction unit 441, a contour expansion processing unit 442, a shift amount detector 443, and an image selection unit 444.
The contour extraction unit 441 includes a contour extraction filter such as a Laplacian filter or a Sobel filter, and extracts a contour component (a high-frequency component having a predetermined value or higher) of each image data item included in an image set stored in the storage 42. The contour extraction processing by the contour extraction unit 441 may be performed by signal processing on the frequency domain using a Fourier transform (FFT), for example, instead of filter processing. The contour expansion processing unit 442 performs processing to expand the contour component extracted by the contour extraction unit 441. The contour expansion processing is performed by thickening an extracted contour line, or blurring using a smoothing filter such as a Gaussian filter. The contour extraction and smoothing may be performed by a single filter.
The shift amount detector 443 detects a shift amount of contour components extracted from image data items included in an image set as a shift amount between image data items included in the image set. The image selection unit 444 selects an image set to be used for still image display based on the shift amount detected by the shift amount detector 443. That is, the image selection unit 444 selects as an image set for still image display an image set having the smallest shift amount between image data items among multiple image sets obtained by imaging for video display. Image data included in the image set selected at the image selection unit 444 is obtained from the storage 42 by the image processor 43.
The display controller 45 displays on the monitor 5 an endoscope image colored based on the image data (display image data) subjected to the image processing at the image processor 43. The display controller 45 converts the display image data which is a digital signal to an analog video signal, for example. In addition, the display controller 45 inputs a video signal to the monitor 5 in response to an instruction from the controller 47 to display the endoscope image to the monitor 5. The drive controller 46 controls the imaging operation of the imager 212. That is, the drive controller 46 controls exposure initiation or termination of the imager 212 and controls reading of an image signal from the imager 212.
The controller 47 controls the entire operation of the endoscope system 1 including the operation of the signal process apparatus 4. For example, the controller 47 synchronizes control of the light source 31 by the light control unit 34 with imaging control of the imager 212 by the drive controller 46. The controller 47 also controls the display controller 45. In addition, the controller 47 causes the still image selection unit 44 to perform processing for still image display in response to a still image display instruction through the operating unit 48.
The operating unit 48 includes an operating member through which a user operates the endoscope system 1. In the present embodiment, the operating unit 48 includes an operating member by which the user instructs still image display of an endoscope image. The operating member may be a button, a switch, or a touch panel.
The monitor 5 is a monitor such as a liquid crystal display, and displays various types of images such as an endoscope image, in accordance with control by the display controller 45. Next, the operation of the endoscope system 1 according to the present embodiment will be explained.
In step S101, the controller 47 determines whether or not the power supply of the endoscope system 1 is turned off. In step S101, if it is determined that the power supply of the endoscope system 1 is turned off, the operation shown in
In step S101, if it is determined that the power supply of the endoscope system 1 is not turned off, the processing proceeds to step S102. In step S102, the controller 47 causes the drive controller 46 to initiate imaging control of the imager 212, and causes the light source apparatus 3 to start light emission so as to be synchronized with imaging of the imager 212. The light control unit 34 controls the light source 31 so that the light source apparatus 3 emits a different color of light for each predetermined field period, for example. On the other hand, the drive controller 46 controls the imager 212 so that imaging is performed in each predetermined field period. The image data is stored in the storage 42 in the aforementioned processing.
In step S103, the controller 47 causes the image processor 43 to perform image processing for video display of an endoscope image. The image processor 43 performs basic processing for video display such as synchronization, gradation correction, color correction, scaling, and enhancement of image data.
In step S104, the controller 47 causes the display controller 45 to display an endoscope image based on the image data processed at the image processor 43. That is, the display controller 45 converts display image data generated for every three fields (one frame) to a video signal and inputs the video signal to the monitor 5. The monitor 5 displays an endoscope image based on the video signal. The user can observe the inside of the body of the subject, for example, as a video by viewing the endoscope image displayed on the monitor 5.
In step S105, the controller 47 determines whether an instruction for still image display of an endoscope image which is being displayed as video is made by a user operation of the operating unit 48. When it is determined that no instruction for still image display is made in step S105, the processing returns to step S101. In this case, the video display of the endoscope image is continued.
When it is determined that an instruction for still image display is made in step S105, the processing proceeds to step S106. In step S106, the controller 47 causes the still image selection unit 44 to select an image set for still image display. The operation of selecting an image set will be explained below.
In
In addition, if, for example, a special light is emitted from the light source apparatus 3, the influence of G image data and B image data greatly affects display image data generated from the image set. Accordingly, the contour extraction unit 441 may be arranged to acquire only image data of two particular colors when it is anticipated that the particular colors greatly affect an image set. In the following explanation, it is assumed that the contour extraction unit 441 acquires only G image data and B image data, for example.
In step S202, the contour extraction unit 441 extracts contour components of G image data and B image data included in each of acquired image sets, for example, by a filtering processing. As shown in
In step S203, the contour expansion processing unit 442 performs contour expansion processing to the contour components of the G image data and B image data extracted by the contour extraction unit 441. The magnitude of contour expansion processing may vary in accordance with the allowed amount of shifts between image data items. That is, if the allowed amount of shifts between image data items is large, the magnitude of contour expansion processing can be increased, and if the allowed amount of shifts between image data items is small, the magnitude of contour expansion processing needs to be decreased.
In step S204, the shift amount detector 443 normalizes contour components of R image data, G image data and B image data subjected to contour expansion processing at the contour expansion processing unit 442. The shift amount detector 443 normalizes the contour components by a maximum value that each pixel data can take in image data, for example.
In step S205, the shift amount detector 443 calculates a shift amount between image data included in the same image set. If a shift amount is F, the shift amount F is calculated by equation (1), i.e., L2norm (sum of squared differences) of each pixel of image data. In the equation (1), “I1 (x, y)” represents a pixel value of a position (x, y) in image data I1, which is one of R image data, G image data, and B image data after normalization, “I2 (x, y)” represents a pixel value of a position (x, y) in image data I2, which is another one of R image data, G image data, and B image data after normalization, “x” represents a horizontal position, and “y” represents a vertical position. For example, the left end position of image data is x=1, and the right end position of the image data is x=N. In addition, the top end position of image data is y=1, and the bottom end position of the image data is y=M.
As shown in
The calculation of equation (1) is usually performed between R image data and G image data; G image data and B image data; and B image data and R image data. However, if the contour components are extracted only for image data items of two particular colors, equation (1) is applied to the image data items.
In step S206, the image selection unit 444 selects an image set to be used for still image display, in accordance with the shift amounts between image data items included in each of the image sets. When applying equation (1), as the value of F is smaller, the contour shift amount becomes smaller. Thus, the image selection unit 444 selects an image set having the smallest shift amount between image data items as an image set for still image display. In the example of
The explanation returns to
In step S108, the controller 47 causes the display controller 45 to display an endoscope image based on the image data processed at the image processor 43. That is, the display controller 45 converts display image data generated by image processing for still image display to a video signal, and inputs the video signal to the monitor 5. The monitor 5 displays an endoscope image based on the video signal. The user can observe a still image of the inside of the body at necessary timings by viewing the endoscope image displayed on the monitor 5. In step S109, the controller 47 determines whether an instruction for termination of still image display is made by user operation of the operating unit 48. When it is determined that no instruction for still image display termination is made in step S109, the processing returns to step S108. In this case, the still image display of the endoscope image is continued.
When it is determined that an instruction for still image display termination is made in step S109, the processing returns to step S101. In this case, the video display is resumed if the power supply of the endoscope system 1 is not turned off. As explained above, in the present embodiment, the shift amount between image data items included in an image set is detected based on the shift amount of contour components of image data items included in the image set. Since contour components are not extracted from image data having a uniformly high luminance such as an image data having halation or an overexposed image data, the possibility of selecting an image set that is not suitable for still image display can be decreased. In addition, by calculating the shift amount F by L2norm of pixel value, not only the way of change in luminance in pixel, but also the sum of a pixel change amount can be evaluated. This improves noise resistance.
Furthermore, when calculating the shift amount F, blurring is performed to the contour components. Accordingly the noise resistance can be further improved. Blurring is not an essential processing, and may be performed in accordance with the allowed shift amount.
In this embodiment, an image set includes three items of image data obtained by the field sequential method. However, the image set is not limited to include image data items obtained by the field sequential method. For example, the image set may include image data obtained for each field of simultaneous-type method (interlace method). In the interlace method, different rows of pixels are exposed for each field, and accordingly, a shift occurs between image data items. The shift amount between image data items is detected by the technique of the present embodiment.
According to the present embodiment, the shift amount F is calculated by L2norm of pixel values. The shift amount F is not limited to L2norm of pixel values. For example, the shift amount F may be L1norm (sum of absolute difference values) of pixel values.
According to the present embodiment, one image set is selected in accordance with the shift amount F. However, the image selection unit 444 may be arranged to select multiple image sets having small shift amounts F.
The variation example of the embodiment will be explained below.
The noise reduction unit 445 is provided as a preprocessing stage of the contour extraction unit 441, and reduces noise components (components regarded as noise of frequencies higher than contour components) of each image data item included in image sets stored in the storage 42. By reducing noise components prior to the contour extraction processing by the contour extraction unit 441, a contour component can be extracted with high accuracy in the contour extraction processing. The reduction processing unit 446 is provided as a preprocessing stage of the contour expansion processing unit, for example, and performs reduction processing to the contour component of an image data item extracted at the contour extraction unit 441. The reduction processing is performed, for example by thinning or pixel addition processing. By reduction processing, the filter size required for contour expansion processing at the contour expansion processing unit 442 can be reduced, and the load of the contour expansion processing also can be reduced. The reduction processing unit 446 may be provided as a preprocessing stage of the contour extraction unit 441. In this case, the filter size required for contour extraction processing at the contour extraction unit 441 can be reduced, and the load of the contour extraction processing also can be reduced.
In step S302, the noise reduction unit 445 reduces noise of G image data and B image data included in each of the acquired image sets.
In step S303, the contour extraction unit 441 extracts contour components of G image data and B image data subjected to noise reduction at the noise reduction unit 445, for example, by filtering processing.
In step S304, the reduction processing unit 446 performs reduction processing to the contour components of the G image data and B image data extracted by the contour extraction unit 441.
In step S305, the contour expansion processing unit 442 performs contour expansion processing to the contour components of the G image data and B image data subjected to reduction processing at the reduction processing unit 446.
In step S306, the shift amount detector 443 normalizes contour components of R image data, G image data, and B image data subjected to contour expansion processing at the contour expansion processing unit 442.
In step S307, the shift amount detector 443 calculates a shift amount between image data items included in the same image set.
In step S308, the image selection unit 444 selects an image set to be used for still image display, in accordance with the shift amounts between image data items included in each of the image sets.
According to the aforementioned variation example, the shift amount between image data items can be detected with high accuracy, and the possibility of an image set being selected that is not suitable for still image display can be reduced.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2014-211035 | Oct 2014 | JP | national |
This application is a Continuation application of PCT Application No. PCT/JP2015/077981, filed Oct. 1, 2015 and based upon and claiming the benefit of priority from the prior Japanese Patent Application No. 2014-211035, filed Oct. 15, 2014, the entire contents of both of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2015/077981 | Oct 2015 | US |
| Child | 15348600 | US |
