Patent Grant
11503990
The present disclosure relates to an imaging system including a light source that emits illumination light and an imaging element, a processing device, and an illumination control method.
In medical fields, an endoscope system is conventionally used to observe the inside of the body of a subject. In an endoscope system, generally, a flexible insertion portion having an elongated shape is inserted into the body of the subject such as a patient, illumination light is emitted from the distal end of the insertion portion, and the reflected light of the illumination light is received by an imaging unit at the distal end of the insertion portion to capture the in-vivo image. The thus captured biological image is presented on a display of the endoscope system.
When the observation target is vocal cords so as to observe the vocal-cord motion during voice production, the vocal cords are intermittently illuminated with strobe light, and the image of the vocal cords is captured (for example, see Japanese Laid-open Patent Publication No. 2002-172088).
In some embodiments, an imaging system includes: a light source configured to emit illumination light; an imager configured to store an electric charge corresponding to an amount of received light and read out the stored electric charge as a signal value by using a rolling shutter method; a frequency detector configured to detect a vibrational frequency of a predetermined site of a subject; and an illumination controller configured to control emission of the illumination light during a readout period of the signal value. The illumination controller is configured to set a phase for an emission timing of the illumination light in an exposure period during which the imager stores the electric charge to be an identical phase at the frequency in each frame, and refer to respective phases set in exposure periods of chronologically adjacent frames to set an emission timing of the illumination light in the readout period.
In some embodiments, provided is a processing device configured to be connected to an endoscope including an imager configured to read out an electric charge stored in a pixel by using a rolling shutter method. The processing device includes: a frequency detector configured to detect a frequency of a predetermined site of a subject; and an illumination controller configured to control emission of the illumination light during a readout period of the signal value. The illumination controller is configured to set a phase for an emission timing of the illumination light in an exposure period during which the imager stores the electric charge to be an identical phase at the frequency in each exposure period, and refer to respective phases set in exposure periods of chronologically adjacent frames to set an emission timing of the illumination light in the readout period.
In some embodiments, provided is an illumination control method implemented by an imaging system including: a light source configured to emit illumination light; and an imager configured to store an electric charge corresponding to an amount of received light and read out the stored electric charge as a signal value by using a rolling shutter method. The illumination control method includes: detecting a vibrational frequency of a predetermined site of a subject; setting a phase for an emission timing of the illumination light in an exposure period during which the imager stores the electric charge to be an identical phase at the frequency in each frame; and referring to respective phases set in exposure periods of chronologically adjacent frames to set an emission timing of the illumination light in a readout period of the signal value.
The above and other features, advantages and technical and industrial significance of this disclosure will be better understood by reading the following detailed description of presently preferred embodiments of the disclosure, when considered in connection with the accompanying drawings.
Modes (hereinafter referred to as “embodiments”) for carrying out the disclosure are described below. In the embodiments, a medical endoscope system that captures and displays an image inside the body cavity of a subject, such as a patient, is described as an example of an imaging system according to the disclosure. The disclosure is not limited to the embodiments. In the description of the drawings, the same components are denoted by the same reference numeral.
The endoscope 2 includes: an insertion portion 21 that is inserted into the subject; an operating unit 22 that is provided on the proximal end side of the insertion portion 21 so as to be grasped by the operator; and a flexible universal cord 23 that extends from the operating unit 22.
The insertion portion 21 is implemented by using an illumination fiber (light guide cable), an electric cable, etc. The insertion portion 21 includes: a distal end portion 211 including an imaging unit having a built-in imaging element that captures the inside of the subject; a bendable curved portion 212 including a plurality of bendable pieces; and a flexible tube portion 213 having flexibility and provided on the proximal end side of the curved portion 212. The distal end portion 211 includes an illumination unit that illuminates the inside of the subject via an illumination lens; an observation unit that captures the inside of the subject; an opening that communicates with a treatment tool channel; and an air/water supply nozzle (not illustrated).
The operating unit 22 includes: a curving knob 221 that curves the curved portion 212 in a vertical direction and a horizontal direction; a treatment tool insertion portion 222 through which a treatment tool, such as biopsy forceps or a laser scalpel, is inserted into the body cavity of the subject; and a plurality of switch portions 223 that operate peripheral devices such as the processing device 4, the light source device 5, an air supply device, a water supply device, or a gas supply device. A treatment tool inserted through the treatment tool insertion portion 222 is exposed from an opening at the distal end of the insertion portion 21 via the treatment tool channel provided inside.
The universal cord 23 is configured by using an illumination fiber, an electric cable, etc. The universal cord 23 branches at the proximal end thereof so that the end of a branch cord 231, which is one of the branches, is a connector 232 and the proximal end of the other one is a connector 233. The connector 232 is attachable to and detachable from the processing device 4, and the connector 233 is attachable to and detachable from the light source device 5. The universal cord 23 propagates the illumination light emitted from the light source device 5 to the distal end portion 211 via the connector 232, the operating unit 22, and the flexible tube portion 213. The universal cord 23 transmits the imaging signal captured by the imaging unit provided in the distal end portion 211 to the processing device 4.
An illumination fiber 214 (see
The voice input device 3 receives the input of a voice generated by vocal cords when the object of capturing is the vocal cords. The distal end of a cord 31 is coupled to the voice input device 3, and a connector 311 at the proximal end is attachable to and detachable from the processing device 4. The voice input device 3 outputs the input voice to the processing device 4 via the cord 31 and the connector 311. The voice input device 3 corresponds to a vocal-cord vibration input unit.
The processing device 4 performs predetermined image processing on the imaging signal on the inside of the subject's body captured by the imaging unit in the distal end portion 211 of the endoscope 2 and input via the universal cord 23. The processing device 4 controls each unit of the endoscope system 1 based on various instruction signals transmitted from the switch portion 223 in the operating unit 22 of the endoscope 2 via the universal cord 23.
The light source device 5 is configured by using a light source that emits pulsed white light as illumination light (hereinafter referred to as pulsed light), a condenser lens, etc. The light source device 5 receives a light control signal from the processing device 4 and performs PWM (Pulse Width Modulation) control on the driving timing (light emission period) for driving the light source based on the light control signal. Accordingly, the light source device 5 emits pulsed light by pulse driving under the control of an illumination controller 408. The light source device 5 feeds the pulsed light from the light source to the endoscope 2 coupled thereto via the connector 232 and the universal cord 23 (the illumination fiber) as illumination light for illuminating (intermittently illuminating) the inside of the subject's body that is the object of capturing.
The display device 6 is configured by using a display, or the like, using a liquid crystal or an organic EL (Electro Luminescence). The display device 6 presents various types of information including the image corresponding to the display image signal generated by the processing device 4 via a video cable 61. This allows the operator to observe the desired position inside the subject and determine the properties while viewing the image (in-vivo image) presented by the display device 6 and operating the endoscope 2.
Next, configurations of the endoscope 2, the voice input device 3, the processing device 4, and the light source device 5 illustrated in
The endoscope 2 includes an imaging unit 24 at the distal end portion 211. The imaging unit 24 includes: an optical system 241 such as an objective lens disposed on the light receiving surface side of a light receiving unit 242a which is described later; and an imaging element 242 that is provided at the imaging position of the optical system 241 to receive the light condensed by the optical system 241 and execute photoelectric conversion to obtain an electric signal.
The imaging element 242 is implemented by using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging element 242 includes the light receiving unit 242a and a readout unit 242b.
The light receiving unit 242a receives, with the light receiving surface, the light from the object illuminated with the pulsed light from the light source device 5 and executes the photoelectric conversion on the received light to generate an electric signal. Specifically, the light receiving unit 242a includes a plurality of pixels arranged in a matrix and each having, for example, a photodiode that stores the electric charge corresponding to the amount of light and a capacitor that converts the electric charge transferred from the photodiode into a voltage level (signal value), and each of the pixels executes the photoelectric conversion on the light from the optical system 241 to generate an electric signal. In the light receiving unit 242a, two or more pixels are arranged in a pixel row (horizontal line) in a horizontal direction, and the pixel rows are arranged in a vertical direction.
The readout unit 242b sequentially reads the electric signal (signal value) generated by any pixel that is set as the readout target among the pixels of the light receiving unit 242a and outputs the electric signal as an imaging signal. The readout unit 242b executes the exposure on the pixels in the light receiving unit 242a and the readout of electric signals from the pixels. The readout unit 242b sequentially reads the electric signal generated by each of the pixels arranged in a matrix in each horizontal line (pixel row). The readout unit 242b generates an imaging signal by using a rolling shutter method, that is, performs the imaging operation including the exposure and the readout, starting from the horizontal line at the beginning, shifts the timing in each horizontal line, and executes the electric charge reset (the reset of the capacitor), the exposure, and the readout.
Thus, in the imaging unit 24, the exposure timing and the readout timing are different in each horizontal line even during one imaging period (frame). The readout unit 242b outputs the electric signal (imaging signal) read from the pixels of the light receiving unit 242a to the processing device 4 via a cable (not illustrated) and the connector 232.
Next, the processing device 4 is described. The processing device 4 includes an AGC (Auto Gain Control) 401, a strobe processing unit 402, an image processing unit 403, a memory 404, a display controller 405, an input unit 406, a vocal-cord frequency detecting unit 407, the illumination controller 408, and a control unit 409. According to the present embodiment, the processing device 4, the endoscope 2, and the light source device 5 operate in accordance with a clock signal generated by a clock generator (not illustrated) provided in the processing device 4.
The AGC 401 adjusts the amplification factor (gain) of an electric signal to maintain a constant output level. The AGC 401 is configured by using a general-purpose processor such as a CPU (Central Processing Unit) or a dedicated processor such as various arithmetic circuitries that execute a specific function, e.g., an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) that is a programmable logic device capable of rewriting the processing contents.
The strobe processing unit 402 acquires a PWM signal for controlling the intermittent illumination with pulsed light from the light source device 5 and outputs an imaging signal input from the AGC 401 in association with the PWM signal to the image processing unit 403. The strobe processing unit 402 is configured by using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuitries that execute a specific function, e.g., an ASIC or an FPGA.
The image processing unit 403 executes predetermined signal processing on the electric signals (imaging signals) of the pixels read by the readout unit 242b of the imaging unit 24 to generate image signals. For example, the image processing unit 403 performs, on an imaging signal, image processing including at least an optical black subtraction process, a white balance (WB) adjustment process, a synchronization process in the case of an imaging element having the Bayer arrangement, a color matrix calculation process, a gamma correction process, a color reproduction process, an edge enhancement process, and the like. The image processing unit 403 is configured by using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuitries that execute a specific function, e.g., an ASIC or an FPGA.
The memory 404 stores various programs for operating the processing device 4 and the light source device 5. The memory 404 temporarily stores the information that is being processed by the processing device 4. The memory 404 stores the imaging signals, read by the readout unit 242b, corresponding to the matrix arrangement of the pixels in the light receiving unit 242a in units of frame. The memory 404 stores the image signals generated by the image processing unit 403 in units of frame. The memory 404 is configured by using a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The memory 404 may be configured by using a memory card, or the like, attached at the outside of the processing device 4.
The display controller 405 selects a display image signal from the image signals of frames generated by the image processing unit 403 in accordance with the display cycle of the display device 6 and outputs the selected image signal as the image signal to be displayed on the display device 6. Alternatively, the display controller 405 combines the image signals of frames generated by the image processing unit 403 for each display period of the display device 6 to generate a display image signal and outputs it to the display device 6. The display controller 405 converts the display image signal from a digital signal into an analog signal, changes the format of the analog image signal after the conversion into a format such as a high-definition format, and outputs it to the display device 6. The display controller 405 is configured by using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuitries that execute a specific function, e.g., an ASIC or an FPGA.
The input unit 406 is implemented by using an operating device such as a mouse, a keyboard, or a touch panel to receive the input of various types of instruction information (instruction signals) of the endoscope system 1. Specifically, the input unit 406 receives the input of various types of instruction information such as subject information (e.g., the ID, the date of birth, or the name), the identification information (e.g., the ID or examination corresponding item) on the endoscope 2, or examination content.
The vocal-cord frequency detecting unit 407 detects the frequency (vocal-cord frequency) of the voice (the vibration of vocal cords) input to the voice input device 3 and input to the processing device 4 via the cord 31 and the connector 311. According to the first embodiment, the voice is generated from the vocal cords of the subject. The vocal-cord frequency detecting unit 407 outputs the detected frequency of the voice to the control unit 409. The vocal-cord frequency detecting unit 407 is configured by using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuitries that execute a specific function, e.g., an ASIC or an FPGA.
The illumination controller 408 controls the operation of the light source device 5. Specifically, the illumination controller 408 controls the emission timing and the illumination period of pulsed light from a light source 51 in synchronization with the frequency of the voice detected by the vocal-cord frequency detecting unit 407. The illumination controller 408 generates the pulse for driving the light source 51 based on the vocal-cord frequency detected by the vocal-cord frequency detecting unit 407 and the preset light emission period (the pulse width or the duty ratio), generates the light-source control PWM signal including the pulse, and outputs it to a pulse generating unit 53. The illumination controller 408 outputs the generated PWM signal to the control unit 409. The illumination controller 408 is configured by using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuitries that execute a specific function, e.g., as an ASIC or an FPGA.
The illumination controller 408 includes a phase setting unit 408a. The phase setting unit 408a sets the phase for emitting the strobe light with regard to the vocal-cord vibration (vocal-cord frequency). The phase setting of the strobe emission by the phase setting unit 408a is described later.
The control unit 409 controls the processing operation of each unit of the processing device 4. The control unit 409 executes, for example, the transfer of instruction information and data to each component of the processing device 4 so as to control the operation of the processing device 4. The control unit 409 is coupled to the imaging unit 24 and the light source device 5 via each cable. The control unit 409 also controls the operation of the imaging unit 24. In the description according to the present embodiment, the imaging element 242 and the light source 51 are driven at the synchronized imaging timing and illumination timing under the control of the control unit 409. The control unit 409 is configured by using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuitries that execute a specific function, e.g., an ASIC or an FPGA.
Next, the light source device 5 is described. The light source device 5 includes the light source 51, a light source driver 52, and a pulse generating unit 53.
The light source 51 is configured by using a light source such as a white LED that emits pulsed white light (pulsed light) and an optical system such as a condenser lens. The light source 51 generates the illumination light to be supplied to the endoscope 2 and guides the illumination light to the endoscope 2 via an illumination fiber, etc.
The light source driver 52 supplies a predetermined electric power to the light source 51 based on the PWM signal generated by the pulse generating unit 53. Accordingly, the subject is illuminated with the light (pulsed light) that is emitted from the light source 51 through the distal end portion 211 of the insertion portion 21 via the connector 233 and the universal cord 23.
The pulse generating unit 53 generates the pulse for driving the light source 51 based on the PWM signal acquired from the illumination controller 408, generates the light source control signal including the pulse, and outputs it to the light source driver 52.
Next, the phase setting of the strobe emission by the phase setting unit 408a is described with reference to
The phase setting unit 408a refers to the readout timing of the readout unit 242b to set the readout period and the full exposure period of the imaging operation performed by the imaging unit 24. For example, as illustrated in
Then, with regard to the full exposure periods TE1, TE2, TE3, TE4, . . . , the phase setting unit 408a sets a phase for the strobe emission in each full exposure period. According to the first embodiment, the phase setting unit 408a sets a first phase in each full exposure period as each of emission phases (phases for the strobe emission) in the full exposure period. As illustrated in for example
After setting the phases of the full exposure periods TE1, TE2, TE3, TE4, . . . , the phase setting unit 408a sets the phase for the emission in each of the total readout periods TR1, TR2, TR3, . . . . The phase setting unit 408a calculates the average of the phases set in the frames previous to and subsequent to the readout period and sets the averaged phase as the emission phase in the total readout period.
As illustrated in for example
The illumination controller 408 generates a PWM signal based on the phase set by the phase setting unit 408a. The pulse generating unit 53 generates a pulse based on the acquired PWM signal so as to drive the light source driver 52 and cause the light source 51 to emit the strobe. All of the strobe emissions have the same emission intensity. The emission intensities in the full exposure period and the total readout period may be set to be different from each other.
According to the first embodiment described above, with regard to the full exposure periods TE1, TE2, TE3, TE4, . . . , each of the phases for the strobe emission in the frame is set to be the identical phase and, with regard to the total readout periods TF1, TR2, TR3, . . . , the average of the phases set in the adjacent frames is calculated and the averaged phase is set as the emission phase in the total readout period. According to the first embodiment, even in the case of sequential imaging using the rolling shutter method, it is possible to suppress blurring of a vocal-cord image.
Next, a second embodiment of the disclosure is described. An endoscope system according to the second embodiment has the same configuration as that of the above-described endoscope system 1. The second embodiment is different from the first embodiment described above in the phase setting during the readout period. A setting process, which is different from that in the first embodiment described above, is described with reference to
The phase setting unit 408a sets a phase for the strobe emission in each of the full exposure periods TE1, TE2, TE3, TE4, . . . , in the same manner as in the first embodiment. Specifically, the phase setting unit 408a sets a first phase in each full exposure period as each of emission phases in the full exposure period. As illustrated in for example
After setting the phases in the full exposure periods TE1, TE2, TE3, TE4, . . . , the phase setting unit 408a sets the respective phases for the emission in the total readout periods TR1, TR2, TR3, . . . . According to the second embodiment, the phase setting unit 408a sets the phase in the chronologically previous frame out of the phases set in the frames previous to and subsequent to the readout period as the emission phase in the total readout period. As illustrated in for example
The illumination controller 408 generates a PWM signal based on the phase set by the phase setting unit 408a. The pulse generating unit 53 generates a pulse based on the acquired PWM signal so as to drive the light source driver 52 and cause the light source 51 to emit the strobe. According to the second embodiment, the emission intensity in the total readout period is set to be lower than the emission intensity in the full exposure period.
According to the second embodiment described above, with regard to the full exposure period, each of the phases for the strobe emission in the frame is set to be the identical phase, with regard to the total readout period, the phase set in the frame before the total readout period is set as the emission phase in the total readout period, and the emission intensity in the total readout period is set to be lower than the emission intensity in the full exposure period. Thus, even though the phase in the frame after the readout period is different from the phase in the full exposure period, the effect of blurring of a vocal-cord image due to the exposure in the total readout period may be reduced. According to the second embodiment, even in the case of sequential imaging using the rolling shutter method, it is possible to suppress the blurring of a vocal-cord image.
Next, a modification of the second embodiment of the disclosure is described. An endoscope system according to the present modification has the configuration similar to that of the above-described endoscope system 1. The present modification is different from the above-described second embodiment in the phase setting during the readout period. A setting process different from that in the above-described second embodiment is described below with reference to
The phase setting unit 408a sets a phase for the strobe emission in each of the full exposure periods TE1, TE2, TE3, TE4, . . . , in the same manner as in the first embodiment and the second embodiment. Specifically, the phase setting unit 408a sets a first phase in each full exposure period as each of emission phases in the full exposure period.
After setting the phases in the full exposure periods TE1, TE2, TE3, TE4, . . . , the phase setting unit 408a sets the phase for the emission in each of the total readout periods TR1, TR2, TR3, . . . . According to the present modification, the phase setting unit 408a sets the phase in the chronologically subsequent frame out of the phases set in the frames previous to and subsequent to the readout period as the emission phase in the total readout period. As illustrated in for example
The illumination controller 408 generates a PWM signal based on the phase set by the phase setting unit 408a. The pulse generating unit 53 generates a pulse based on the acquired PWM signal so as to drive the light source driver 52 and cause the light source 51 to emit the strobe. According to the present modification, the emission intensity in the total readout period is set to a value lower than the emission intensity in the full exposure period.
According to the modification described above, with regard to the full exposure period, each of the phases for the strobe emission in the frame is set to be the identical phase, with regard to the total readout period, the phase set in the frame after the total readout period is set as the emission phase in the total readout period, and the emission intensity in the total readout period is set to be a value lower than the emission intensity in the full exposure period. Thus, even though the phase in the frame before the readout period is different from the phase in the full exposure period, the effect of blurring of a vocal-cord image due to the exposure in the total readout period may be reduced. According to the present modification, even in the case of sequential imaging using the rolling shutter method, it is possible to suppress the blurring of a vocal-cord image.
In addition to the phase setting example according to the above-described modification, the average of the phases in the adjacent frames as described above in the first embodiment may be set as the phase in the total readout period, or the conventionally set phase illustrated in
In the first embodiment and the second embodiment described above, it is possible to provide a rotary filter including a plurality of filters that is arranged on the optical path of white light (illumination light) emitted from the light source 51 to rotate so as to exclusively allow the passage of light having a predetermined wavelength band included in the white light. The provided rotary filter allows the light having an individual wavelength band, such as red light (R), green light (G), and blue light (B), to be sequentially transmitted and emitted. By controlling the rotation of the rotary filter in accordance with the emission timing of pulsed light, any of red light (R illumination), green light (G illumination), and blue light (B illumination) having a narrow band, included in the white light emitted from the light source 51, may be sequentially emitted to the endoscope 2 (sequential lighting). Instead of the rotary filter, it is possible to use a light source (e.g., an LED light source) that emits the light having a wavelength band for each color.
In the description according to the first embodiment and the second embodiment described above, the imaging element 242 operates under the control of the control unit 409; however, the illumination controller 408 (the phase setting unit 408a) may be provided at the side of the endoscope 2 so that the strobe emission is controlled at the side of the endoscope 2. In the description, the imaging element 242 operates based on the clock signal generated by the processing device 4; however, a clock generator may be provided in the endoscope 2 so that the imaging element 242 operates based on the clock signal generated by the clock generator (the clock signal generated by the endoscope 2), or the imaging element 242 may operate based on a clock signal generated by an external clock generator.
In the description according to the first embodiment and the second embodiment described above, the AGC 401 is provided in the processing device 4; however, the AGC 401 may be provided in the endoscope 2 (e.g., the imaging unit 24).
In the description according to the first embodiment and the second embodiment described above, the object is a vocal cord; however, any object other than a vocal cord is applicable as long as the object vibrates at a high speed and the frequency thereof is detectable by the vocal-cord frequency detecting unit 407.
As described above, the imaging system according to the disclosure is useful in suppressing the blurring of an image even in the case of application of a rolling shutter method.
According to the disclosure, it is possible to produce an advantage such that the blurring of an image may be suppressed even in the case of application of a rolling shutter method.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the disclosure in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2018-021867 | Feb 2018 | JP | national |
This application is a continuation of PCT International Application No. PCT/JP2019/002378 filed on Jan. 24, 2019, which designates the United States, incorporated herein by reference, and which claims the benefit of priority from Japanese Patent Application No. 2018-021867, filed on Feb. 9, 2018, incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20040095464 | Miyagi et al. | May 2004 | A1 |
| 20140316196 | Wichern | Oct 2014 | A1 |
| 20160073865 | Takahashi | Mar 2016 | A1 |
| 20170064178 | Kagawa et al. | Mar 2017 | A1 |
| 20190142264 | Bos | May 2019 | A1 |
| 20190142265 | Bos | May 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 2002-172088 | Jun 2002 | JP |
| 2004-166761 | Jun 2004 | JP |
| 2016-052453 | Apr 2016 | JP |
| 2016-202380 | Dec 2016 | JP |
| 2015194415 | Dec 2015 | WO |
| Entry |
|---|
| International Search Report dated Apr. 9, 2019 issued in PCT/JP2019/002378. |
| Number | Date | Country | |
|---|---|---|---|
| 20200345222 A1 | Nov 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2019/002378 | Jan 2019 | US |
| Child | 16932891 | US |
