Patent Application
20220360700
This application claims the benefit under 35 U.S.C. 119(a) to European Patent Application No. 20166380.4, filed Mar. 27, 2020, the disclosure of which is incorporated herein by reference in its entirety.
The present invention relates to a medical imaging device for spatially resolved recording of multispectral video data of an examination area of a patient, comprising a light source for illuminating the examination area, having a plurality of optical emitters with different wavelengths distributed over the visible and NIR spectral range, an RGB camera sensor for recording the examination area, which generates red, green and blue colour signals using filters with red, green and blue filter curves, an acquisition controller arranged for synchronized control of the light source and the RGB camera sensor to repeatedly activate one or more emitters, which together produce a spectral exposure pattern having one or more wavelengths, in a predetermined activation sequence of a plurality of successive spectral exposure patterns, each taken as an image by the RGB camera sensor, a data processing device connected to the exposure control device and the RGB camera sensor for recording the RGB camera sensor signals and for evaluating them for spatially resolved multispectral analysis to derive a physiological parameter and for spatially resolved display thereof as a video of the examination area.
The invention relates to a medical imaging device which may be integrated in an endoscope, laparoscope, microscope or exoscope. A video sequence of images is recorded using one or two (stereoscope) high-resolution colour camera sensors (RGB camera sensors) with colour filters with red, green and blue filter curves based on CMOS or CCD technology for the actual sensor.
Various optical devices are used in medicine to assist doctors in diagnosis and therapy. These devices often use high-resolution colour cameras to record the area under examination. One example is the medical video endoscope. A colour camera sensor is located at the proximal or distal exit of the endoscope. In endoscopes, the “chip in the tip” (CIT) technology is becoming increasingly popular here, in which the miniaturized colour camera sensor with imaging optics is placed directly in the tip of the endoscope (distal). The device and method described here can be used with this technology. The RGB colour camera sensors are colour filter sensors with red, green, blue colour filters arranged in a Bayer matrix. The pixels behind the red, green and blue colour filters generate red, green and blue colour signals. Illumination is provided by a light source (e.g. cold light sources with halogen lamps or xenon light sources, lasers or preferably LED light sources). The light from the light source is usually coupled into the endoscope or other optical device via fibre optics and illuminates the examination area. Such endoscopes can be designed with very high resolution (Ultra-HD) and/or three-dimensional imaging (stereoscopes).
WO 2017/118735 A2 describes a multispectral analysis method to be used on the human body, in which a light source with several LEDs of different wavelengths in the visible and in the NIR spectral range and a camera sensor are used; however, the method is not an imaging method, but serves to determine individual physiological parameters of a subject by multispectral analysis. The light source is repeatedly exposed in activation sequences with successive spectral exposure patterns when LEDs with different wavelengths are activated. However, the parameters obtained from the multispectral analysis are not displayed in a spatially resolved manner. Rather, the camera sensor is placed directly on the subject's skin and captures light transflected by the subject's tissue as a function of the distance from the light source to the capture location in order to derive physiological parameters of the subject during physical exertion, which include oxygen saturation of the arterial blood, tissue oxygen saturation, water content, pulse rate, water content of the tissue, etc.
Outside of medical image analysis, there are a number of image analysis techniques that use LEDs of different wavelengths with exposure at different wavelengths in synchronized multiplexed operation with a colour sensor to determine spectral information—see for example US 2016/0187199 A1, US10 161 796 B1 and US 2010/0073504 A1.
Classical video endoscopes are used for colour video imaging of an examination area inside the body. It is now known from physiological imaging that spectral imaging, in this case multi- or hyperspectral imaging, can provide users of endoscopes with a lot of additional information that can be used during operations or in diagnostics. A very important area in physiological imaging is perfusion imaging, which can be performed, for example, via multispectral analysis of colour signals or by fluorescence imaging.
In medical technology, physiological imaging with multispectral or hyperspectral methods is known, among other things, for the fact that physiological parameters, such as haemoglobin content and the oxygenation of haemoglobin in the examination area, which are displayed spatially resolved by false colours, contain very important clinical information for the physician. Furthermore, spatially resolved spectral imaging is also known as molecular imaging using chemometric techniques, as chemical information can be displayed spatially resolved. For clinical application, the spatially resolved representation of the haemoglobin content, especially the water content and the fat content in the tissue, are important variables.
The derivation of systemic physiological parameters such as heart rate, respiratory rate and especially the properties of the arterial blood is also of great importance and provides information with high significance about the patient's condition. There are already various approaches to combine spatially resolved imaging of a physiological parameter with the usual white light imaging (colour video).
Outside the medical field, there are a number of imaging techniques using multispectral analysis, wherein, to obtain spatially resolved multispectral information, LED light sources repeatedly illuminate an object or subject with LED emitters of different wavelengths in multiplexed operation with activation sequences of successively different spectral exposure patterns and record reflected light with a colour camera sensor, wherein the light source is timed and synchronized with the operation of the colour camera sensor such that successive exposure patterns fall into successively captured images (video frames) of the colour camera sensor. The different spectral exposure patterns of an activation sequence can thus be captured separately and subjected to multispectral analysis.
From WO 2019/051591 Al a medical imaging device with the features of the generic term of patent claim is known. The light source used comprises a plurality of LEDs with different wavelengths in the visible and NIR spectral range. Since an activation sequence of successive spectral exposure patterns with a corresponding number of successive images is recorded with the colour camera sensor, after the multispectral analysis the frame rate, at which the physiological information obtained from the multispectral analysis is displayed, is reduced compared to the frame rate of the colour camera sensor for a normal colour video corresponding to the number of successive exposure patterns in the activation sequence. In order to nevertheless enable a sufficient time resolution of the physiological information from the multispectral analysis, which is represented as a video in false colours, exposure patterns are used in the described method in which several LEDs are activated simultaneously at different wavelengths. The intensity information associated with the different wavelengths can be separated from each other from the three colour signals of the colour camera sensor, such techniques being referred to as spectral unmixing. This makes it possible to capture spectral information at two or more wavelengths per image captured by the colour camera sensor. In this way, activation sequences with, for example, two or three successive spectral exposure patterns, each with simultaneous activation at multiple wavelengths, may be sufficient to perform meaningful multispectral analyses with a sufficient number of spectral support points at different wavelengths.
If activation sequences are repeated, e.g. with three successive spectral exposure patterns, one after the other, and after the multispectral analysis based on this, one image of the derived physiological parameter is displayed in each case in false colours, the video of the physiological parameter, e.g. the oxygen saturation of the arterial blood, has a frame rate that is reduced to ⅓ compared to a normal colour video.
For high-quality colour video, a frame rate of 60 Hz is standard today. Accordingly, the RGB camera sensors in widespread use today are designed for such a frame rate of 60 Hz, although some camera sensors can also achieve refresh rates of up to 120 Hz. For physiological imaging of physiological parameters derived from multispectral analysis, a frame rate reduced to ⅓, i.e. typically 20 Hz, is acceptable.
It is the object of the present invention to provide a device and a method for medical imaging of a physiological parameter derived from a multispectral analysis, wherein for a given number of spectral exposure patterns per activation sequence, the largest possible number of spectral support sites (number of different wavelengths) are to be provided for the multispectral analysis.
The medical imaging device with the features of patent claim 1 and the method with the features of patent claim 14 serve to solve this problem. Advantageous embodiments are indicated in the subclaims.
The medical imaging device has a light source with an emitter, typically an LED, with a wavelength that is within ±50% of the half-width of the emitter around the intersection of the blue and green filter curves or the green and red filter curves. Wavelength here refers to the peak or peak wavelength of the emission spectrum of the emitter. The first intersection point mentioned is the intersection point of the blue filter curve sloping after its maximum with the green filter curve rising before its maximum, and the second intersection point mentioned is the intersection point of the green filter curve sloping after its maximum with the red filter curve rising before its maximum, i.e. the intersection points in the respective transition region of the said filter curves are meant. In other words, the peak wavelength of the emitter is shifted by no more than half the half-width of the emitter with respect to the wavelength of the intersection of the filter curves concerned. Said another way, the wavelength of the intersection of the affected two filter curves lies in the wavelength interval swept by the half-width of the emitter. The fact that the wavelength of the emitter is sufficiently close, namely not further than 50% of the half-width, to the intersection of the affected two filter curves ensures that both colour signals belonging to the affected filter curves account for a significant proportion of the emitter's radiated intensity.
The recording control device and the data processing device are arranged to separately record the affected two of the red and green or the green and blue colour signals in a spectral exposure pattern with activation of an emitter at one of the intersections and to evaluate them in the multispectral analysis with wavelengths shifted against each other and different from each other by the two affected filter curves two supporting point wavelengths. In this way, two colour signals with peak wavelengths shifted against each other can be recorded by an emitter due to the shift of the signals of an emitter in the two affected colour signals caused by the two affected filter curves. This makes it possible to obtain two interpolation point wavelengths for multispectral analysis by activating a single emitter.
In a preferred embodiment, the light source has a first emitter whose wavelength is within ±50% of its half-width about the intersection of the blue and green filter curves and a second emitter whose wavelength is within ±50% of its half-width about the intersection of the green and red filter curves. The exposure control and data processing means are arranged to activate the first and second emitters in different exposure patterns of the activation sequence and, when the first emitter is activated, to detect the green and red colour signals separately and to analyse them in the multispectral analysis with the blue and green filter curves shifted with respect to each other, and, when the second emitter is activated, to detect the green and red colour signals separately and to evaluate them in the multispectral analysis with wavelengths that are shifted by the green and red filter curves and different from each other as two supporting wavelengths. In this way, the first and second emitters detect a total of four different wavelengths for the multispectral analysis.
In another embodiment, the light source comprises a first emitter whose wavelength is less than ±50% of its half-width less than the wavelength at the intersection of the blue and green filter curves, and a second emitter whose wavelength is less than ±50% of its half-width greater than the wavelength at the intersection of the blue and green filter curves. The recording control and data processing device are set up to activate the first and second emitters in different exposure patterns of the activation sequence and to detect the green and blue colour signals separately both when the first emitter is activated and when the second emitter is activated, and to evaluate these in the multispectral analysis in each case with wavelengths that are shifted from one another by the blue and green filter curves, as a total of four interpolation point wavelengths.
Also in this way, four interpolation point wavelengths in the blue-green transition range can be generated for multispectral analysis.
In a preferred embodiment, when the light source is controlled with an exposure pattern with one or more emitters with wavelengths in the visible range, the exposure control and data processing devices are arranged to simultaneously activate an emitter in the NIR spectral range and to subtract that one of the blue, green and red colour signals on which the exposure with the emitters in the visible range has the least influence from the other two colour signals in order to determine the effect of the exposure in the NIR spectral range, green and red colour signals on which the exposure with the emitters in the visible range has the least effect, from the other two colour signals in order to compensate for the effect of the exposure in the NIR spectral range on the other two colour signals. The red, green and blue filter curves each have, in addition to their main filter curve in the red, green and blue spectral range respectively, a filter curve in the NIR spectral range that rises again in the same way. This has the consequence that in the case of an exposure with a wavelength in the NIR spectral range, all three colour signals have increased colour signals of approximately the same strength. The colour signal that is least affected, ideally negligibly affected, by the simultaneous exposure in the visible range can then be subtracted from the other two remaining colour signals as a correction to compensate for the contribution of the exposure in the NIR range.
In such an embodiment, the exposure control and data processing means may be arranged, when controlling the light source to produce an exposure pattern with wavelengths above 500 nm, to simultaneously activate a first emitter with a wavelength in the range of 500 to 600 nm, a second emitter having a wavelength in the range of 600 to 700 nm and an emitter having a wavelength in the NIR spectral range, and then subtracting the blue colour signal, which is at least affected by the exposures in the visible range, from the green and red colour signals to compensate for the effect of the exposure in the NIR spectral range on the green and red colour signals.
In medical operation, in addition to the pictorial representation of a physiological parameter derived from the multispectral analysis, it is important to also display the examination area in the form of a colour video parallel to the video images of the physiological parameter. For this reason, in a preferred embodiment, the acquisition control device is arranged to be switchable between a white light mode of operation and a multispectral mode of operation. In the white light mode of operation, a plurality of emitters with wavelengths distributed over the visible spectral range are continuously activated simultaneously to provide continuous illumination with a spectrum approximating white light, thereby capturing the red, green and blue colour signals from the RGB camera sensor and displaying them as colour video. In the multispectral mode of operation, the acquisition control device is arranged to select exposure patterns with activated emitters in the activation sequence which, when summed, result in an approximate white light spectrum, and to display the associated summed red, green and blue colour signals as colour video with a reduced frame rate compared to the colour video in the white light mode of operation, in parallel with the spatially resolved video of the physiological parameter derived by the multispectral analysis. In this way, exposure patterns with wavelengths in the visible range are used both to generate a colour video and to generate the video representation of the physiological parameter derived from the multi spectral analysis.
In a preferred embodiment, the recording control device and the data processing device are set up to combine the intensities Iλ1(t) determined in the course of the multispectral analysis for several wavelengths in each case over the exposure patterns of several successive activation sequences to form a time average
In a preferred embodiment, the acquisition control device and the data processing device are arranged to record the spectral intensities determined in the course of the multispectral analysis for several wavelengths λ1 for a first activation sequence as It2(λ1) and to record them for a later, second activation sequence as It2(λ1) and to evaluate the graph λ1 of the difference of the intensities It1(λ1) and It2(λ1) formed at the support wavelengths as a function of λ1 in order to determine a pulsatile parameter as a physiological parameter.
The temporal distance between the first and second activation sequences can be selected so that they lie within a period of the heart pulse; e.g. a difference between an activation sequence near the high point and an activation sequence near the low point of the pulse signal can be formed and evaluated. To determine physiological parameters that change slowly relative to the pulse beat, such as the oxygen saturation of the arterial blood, differences of intensity spectra It1(λ1) from different periods of the pulsation signal can also be formed and evaluated across periods. Two or more differences between intensity spectra It1(λ1) in different successive activation sequences from one or more periods of the pulsation signal can also be formed and evaluated.
As pulsatile parameters, e.g. the oxygen saturation of the arterial blood, the heart rate, the pulsation index or the heart rate variability can be determined.
In a preferred embodiment, two RGB camera sensors are present and arranged relative to each other to enable stereoscopic images.
The data processing device is set up to evaluate the signals from the two RGB camera sensors in such a way that a stereoscopic colour video is displayed and a stereoscopic representation of the physiological parameter derived by the multispectral analysis in a spatially resolved manner is displayed.
In a preferred embodiment, the medical imaging device has an excitation light source for illuminating the examination area with excitation light in the blue or violet spectral region. The excitation light source is designed in such a way that its spectrum of excitation light is blocked by one of the red and green filter curves so that, in contrast, longer wavelength fluorescent light can be detected by the RGB camera sensor undisturbed by the excitation light.
In a preferred embodiment, the medical imaging device comprises an endoscope carrying with its distal end the light source and the RGB camera sensor, which are in data exchange communication with the acquisition control device and the data processing device via leads passing through the endoscope and emerging from its proximal end.
Finally, the invention provides a method for recording multispectral video data of an examination area of a patient, for multispectral analysis for spatially resolved derivation of a physiological parameter and for spatially resolved display thereof as a video of the examination area, in which method the examination area is illuminated with a light source, which has a plurality of optical emitters with different wavelengths distributed over the visible and NIR spectral range, the examination area is recorded with an RGB camera sensor which generates red, green and blue colour signals using filters with red, green and blue filter curves, the light source and the RGB camera sensor are operated in a synchronized controlled manner by a recording control device, to activate one or more emitters of the light source each to produce a spectral exposure pattern having one or more wavelengths, and to produce, in a predetermined activation sequence, a plurality of successive predetermined exposure patterns, the reflection of which in the examination area is respectively recorded as one image per exposure pattern by the RGB camera sensor, and the colour signals of the RGB camera sensor are subjected to a spatially resolved multispectral analysis in a data processing device which is connected to the recording control device and via the latter to the RGB camera sensor, and the colour signals of the RGB camera sensor are subjected to spatially resolved multispectral analysis in a data processing device which is connected to the recording control device and via the latter to the RGB camera sensor, in order to derive a physiological parameter, and to generate video data for spatially resolved representation of the parameter as a video of the examination area, characterized in that an emitter of the light source, whose wavelength lies in the range of ±50% of its half-value width around the wavelength of the intersection point of the blue and green filter curve or the green and red filter curve is activated in an exposure pattern, and the two colour signals concerned, which are assigned to the two filter curves of the intersection point in whose range the emitter lies, are evaluated in such an exposure pattern by the data processing device in the multispectral analysis with wavelengths which are shifted with respect to one another by the two filter curves concerned and differ from one another as two supporting point wavelengths.
By normalizing the intensity signals of the different emitters to a reference (white balance), the data points of the reflection profile and the absorption profile of the considered examination area can be calculated at each image point from the measured intensities. Thus, the methods from spectroscopy and multispectral/hyperspectral camera technology are applicable for the generation of the physiological parameter images.
Another aspect that can be used with the medical imaging device is the time-resolved evaluation of the medical signals. In this context, the technology of pulse oximetry should be particularly emphasized, in which a pulsatile evaluation of the signals can be used to record the oxygenation of the arterial blood. Such evaluations with pulsatile temporal resolution can be used to visualize the pulse and respiration-dependent physiological parameters with high spatial resolution.
The pulsation of the blood by the heartbeat and the resulting temporal change of the haemoglobin concentration at a local measuring point is the basis of pulse oximetry. Pulse oximetry conventionally evaluates the pulsatile change in the absorption of light in the tissue at 660 nm and 910 nm. The light source for illumination is preferably equipped with emitters in the form of LEDs that can be clocked quickly in time. Due to the fast clocking, the pulsating signal of the arterial blood inside the tissue can be divided into a variable and a constant part. The analysis of the variable portion with the two wavelengths mentioned, at which the intensities of the saturated and unsaturated haemoglobin behave in opposite directions, allows the oxygen saturation of the arterial blood (SpO2 ) to be calculated.
In such an evaluation, further spectral interpolation points can also be used in addition to the two wavelengths mentioned.
The invention is described below with reference to examples of embodiments in connection with the drawings, in which:
The light from the light source 3 is directed onto the examination area via an imaging lens 1. The light reflected from the examination area is directed via a lens 2 onto an RGB colour camera sensor 4. The red, green and blue colour filters designated R, G and B are arranged in the so-called Bayer pattern directly in front of the pixels of the camera sensor, which may be a CMOS or CCD sensor. Each pixel therefore directly measures only one of the three colour signals, the two missing colours are usually estimated (interpolated) from the colour values of the eight neighboring pixels.
A recording control device 5 communicates with the light source 3 and the RGB colour camera sensor 4. The recording control device 5 is arranged to synchronize and control the operation of the light source 3 and the operation of the colour camera sensor 4. The recording control device 5 is also in data exchange connection with a data processing unit 6, 7, represented here by two EDP system modules 6 and 7. The first module 6 of the data processing unit receives and processes the signals from the colour camera sensor and generates successive images (video frames) therefrom. Each successive frame represents a spectral exposure pattern from an activation sequence with several successive, different spectral exposure patterns. The activation sequences are in turn repeated sequentially in time by the recording control device. Each image of the RGB colour camera sensor taken by the module 6 thus corresponds to a specific spectral exposure pattern from the activation sequence. The successive images of the spectral illumination patterns of an activation sequence are subjected to multispectral analysis in the analysis module 7 of the data processing device using predetermined algorithms in order to derive desired information and physiological parameters, as described further below. The analysis module 7 of the data processing device may also be associated with other functions such as image storage, image analysis and other functions. The analysis module 7 of the data processing device may be based on a PC, for example. The data acquisition module 6 can in principle also be implemented in this PC; as a rule, the data acquisition module 6 is implemented in a real-time processor system.
The light source 3 comprises several LED emitters in the visible and NIR spectral range. LEDs are fast-switching, variable emitters (light pulses with a length in the range of 10 μs to 10,000 μs are typical). LEDs operate without thermal problems at high light powers that are not critical for the tissue. Preferably, the light source comprises more than 10 LEDs so that more than 10 supporting wavelengths distributed over the covered spectral range are available. In the visible spectral range, support sites should be present at local absorption maxima of oxygenated haemoglobin. This allows an additional check whether a periodic signal supposedly detected as a pulse signal is actually the pulse signal of the circulation, since the real pulse signal is accompanied by a corresponding pulse signal for oxygenated haemoglobin.
It is further preferred that the light source comprises at least one emitter with emission maximum in the range of 500 nm to 540 nm and one in the range of 570 nm to 600 nm. In particular, an emitter with a wavelength of about 520 nm and an emitter with a wavelength of 585 nm may be present. These wavelengths allow good discrimination between oxygenated and deoxygenated haemoglobin, since 520 nm is an isosbestic point of haemoglobin, i.e. the absorption coefficients of oxygenated and deoxygenated haemoglobin are equal. Another isosbestic point of haemoglobin is at 810 nm. In contrast, the absorption of deoxygenated haemoglobin at 575 nm and 880 nm is much smaller than that of oxygenated haemoglobin. At 600 nm and 760 nm, the absorption of oxygenated haemoglobin is again smaller, and therefore these four wavelengths can also be used to calculate the oxygenation of haemoglobin
By using short wavelength emitters (violet), the imaging device can be used to increase the contrast between organic structures such as blood vessels, structures with different water contents or differences in concentration of other chromophores. Due to the high absorption coefficients of haemoglobin in this spectral range with constant scattering coefficients of the tissue, very sharp images are obtained, as the light penetrates and scatters little into the tissue.
A preferred selection of wavelengths as spectral interpolation points are listed in Table 1 and shown in
The recording control device is arranged, e.g. by programming, to generate a sequence of spectral exposure patterns in a predetermined activation sequence, whereby in each individual spectral exposure pattern one or more emitters with different wavelengths are activated, and to repeat this activation sequence successively in time. Each spectral exposure pattern of the activation sequence is recorded with an image of the colour camera sensor. This synchronization between the light source 3 and the colour camera sensor 4 is provided by the recording control device 5.
The number of spectral exposure patterns and the selection of emitters to be activated in the respective exposure patterns can be set during operation of the exposure control unit. control device. For this purpose, an input device can be provided in which the user enters the number of spectral exposure patterns in the activation sequence and then, for each spectral exposure pattern, the respective emitters to be activated. Thus, if required, a higher number of spectral intercept wavelengths can be recorded at a lower frame rate or, conversely, a lower number of spectral intercept wavelengths can be recorded at a higher frame rate.
The activation sequence illustrated in
For the third spectral exposure pattern over time period S3, emitter 2 (455 nm), emitter 4 (520 nm) and emitter 6 (620 nm) are switched on. The camera sensor is switched on during the phases of all three spectral exposure patterns for image recording.
The activation sequence in
The second spectral exposure pattern uses two emitters at 490 nm and 660 nm, whereby the signals resulting from the first emitter at 490 nm are decomposed by the blue and green filter curves into two neighboring signal maxima in the blue and green colour signals, so that the second spectral exposure pattern also provides three supporting wavelengths.
The third spectral exposure pattern of the activation sequence switches on three emitters at 455 nm, 520 nm and 620 nm, so that the third spectral exposure pattern also provides three supporting point wavelengths. Thus, the activation sequence described in connection with
The right column shows the time-averaged signals (i.e. the time-averaged signals at the respective wavelengths over the seven activation phases shown in the graphs in the middle column). From these average signals at the individual wavelengths, the graph Ī(λ) with all five spectral interpolation points can be assembled (bottom right in
The graph I(t), which is shown second from the bottom in the middle column of
| Number | Date | Country | Kind |
|---|---|---|---|
| 20166380.4 | Mar 2020 | EP | regional |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17194860 | Mar 2021 | US |
| Child | 17874588 | US |
