Patent Grant
11083368
This application claims priority of European patent application number 18214911.2 filed Dec. 20, 2018, the entire disclosure of which is incorporated by reference herein.
The invention relates to a system and a method for creating High Dynamic Range (HDR) monochrome images of a fluorescing fluorophore.
During surgery or during a pathological examination of a biopsy, fluorophores are used to mark specific types of tissue, cells or parts of a cell. The fluorescence of a fluorophore is triggered by light covering the fluorescence excitation spectrum of the fluorophore. Upon excitation, the fluorophore emits fluorescence in a fluorescence spectrum. Light intensities in the fluorescence spectrum are typically low, resulting in low-contrast fluorescence images even if all other wavelengths not falling within the fluorescence spectrum are blocked.
When within the same image areas of very low and very high intensity coexist, then the dynamic range of the imaging sensor might not be sufficient to capture both high and low intensities. This results in losing the intensity information for the brightest, or/and darkest intensities. In that case all areas with intensity below the lowest level will be presented as black and all intensities above the highest level will be presented as white.
The problem of limited dynamic range does not only occur in cases that high and low intensities are exhibited simultaneously, but also sequentially. In such cases the adjustment of the camera sensitivity could mitigate the problem, but rapid intensity changes might not be able to be compensated fast enough by adjustment of the sensitivity. For example, when fluorescence dye, such as indocyanine green (ICG), is injected intravenously for fluorescence angiography, then the fluorescence signal will be start as very dark, and then upon the arrival of the dye bolus, the signal will become rapidly very high, resulting in a few frames with saturated areas due to the delay of the camera sensitivity auto-adjustment.
Even when the imaging sensor (e.g. CCD, CMOS) has high nominal dynamic range, this often refers to digitization circuit and does not reflect the overall dynamic range of the sensor. For example, even if the output signal is 14 bit, i.e. ˜16K different signal levels, the sensor can only distinguish 1000 different light intensity levels. Typically, the dynamic range is increased by capturing multiple images sequentially with a single sensor at different exposure levels so that each subsequent image captures a different dynamic range, e.g. one image for the low intensities, and another for the high intensities. The images are then merged in a way that the dynamic ranges are combined and both the brightest and darkest areas are visualized in a single image. This approach, however, cannot be used for video capture.
The presence of high and low intensity of the fluorescence images results in a low contrast and the loss of valuable information in the process of recording the fluorescence image. To retain this information, it is an object of the system and method according to the invention to create fluorescence images which have an increased dynamic range without the use of expensive additional equipment and which can be used for video.
To solve this problem, the invention provides an image acquisition system, in particular for a microscope or endoscope, for processing digital input images of a fluorescing fluorophore, and a method for creating a digital monochrome HDR fluorescence image from at least two digital monochrome input images of at least one fluorophore.
According to the invention, the image acquisition system comprises a color camera, the color camera having at least two different types of color sensor, a plurality of input pixels, each type of color sensor generating a different digital monochrome input image comprising the input pixels and having a different spectral responsivity, the color camera further comprising at least one co-sensing wavelength band in which the respective spectral responsivities of at least two sensors at a pixel overlap; and an HDR image generator coupled to the color camera; wherein the HDR image generator is configured to compute a digital monochrome HDR fluorescence output image from the at least two digital monochrome input images.
Further, the method comprises the steps of acquiring at least two digital monochrome input images of the fluorescing fluorophore using at least two different types of color sensor, the at least two different types of color sensor having a different spectral responsivity and comprising at least one co-sensing wavelength band in which the spectral responsivities of the at least two types of sensor overlap, selecting the fluorophore to have a fluorescence spectrum which overlaps with the co-sensing wavelength band; and computing a digital monochrome HDR fluorescence image from the at least two digital monochrome input images.
Thus, the system and method according to the invention are able to provide an HDR image of the fluorescing fluorophore by using e.g. a standard color camera, such as an RGB camera that has three different types of color sensor for each pixel, namely an R (red), a G (green) and a B (blue) color sensor. The HDR images are generated by acquiring a separate monochrome image from each type of color sensor. By using the different responsivities of the different types of color sensor in the co-sensing wavelength band, the different exposure levels necessary for an HDR image are generated automatically.
The above system and method can be further improved by adding one or more of the following additional features, of which each can be added independently of the other, and each is advantageous on its own. The additional features described below may each be applied to both the method and the system.
For example, it is preferred that the digital monochrome images are acquired only in the at least one co-sensing wavelength band. This makes sure that the digital monochrome input images are recorded at different responsivities, i.e. at a different mean intensity level.
Further, it is preferred that the spectral responsivities, in particular the mean spectral responsivities, of at least two, preferably at least three, different types of color sensor differ in the at least one co-sensing wavelength band.
The system may, according to another embodiment, comprise an optical band-pass filter having at least one pass band, the at least one pass band overlapping both the at least one co-sensing wavelength band and the fluorescence wavelength band. Use of such a band-pass filters allows the proper co-sensing wavelength band to be selected, where the respective spectral responsivities of the at least color sensors are sufficiently different to enhance the dynamic range of the resulting HDR image. The optical band-pass filter is preferably located in front of the color camera for filtering light reaching the color camera, or the different types of color sensor, respectively.
Preferably, the pass band of the optical band-pass filter is restricted to a co-sensing wavelength band in which the spectral responsivity of each of the at least two, preferably three, color sensors is different. Most preferably, the spectral responsivities of the at least two or three color sensors have their maximum difference in the pass band of the band-pass filter. The maximum difference may be computed from the mean spectral responsivity of the color sensor in the pass band. The mean spectral responsivity can be computed by using the geometric, arithmetic and/or harmonic mean, and/or the median. The maximum difference may be computed by maximizing the sum of the absolute values of the differences between each of the spectral responsivities in the pass band, and/or by maximizing the product of the differences and/or the mean difference. Only those spectral responsivities that are located above and/or below a corresponding predetermined threshold—e.g. above 40% and/or below 90% of the dynamic range—may be considered in the pass band for computing the maximum difference.
Further, it may be advantageous if the spectral responsivities of the color sensors, in particular their mean values as calculated above, are staggered at least approximately equidistantly in the pass band. The distance between the spectral responsivities may be computed linearly or in a log scale. Variation of +/−20% may still be considered as equidistant.
Each of the above features allows the dynamic range covered by the digital monochrome input images to be maximized.
The pass band may, in one embodiment, be adjusted depending on the fluorescence spectrum of the at least one fluorophore and the spectral responsivities in the co-sensing wavelength band by exchanging the band-pass filter manually.
According to another preferred embodiment of the invention, the pass band may be modified without the need to exchange the band-pass filter. For this, the band-pass filter may be a tunable band-pass filter. The tunable band-pass filter may comprise a tunable pass band which is tunable with respect to at least one of a center frequency of the pass band, at least one cut-off frequency of the pass band, a roll-off at at least one cut-off frequency, and a Q-factor. For example, the tunable optical band-pass filter may be a liquid crystal filter, a fiber-optic filter or a tunable band-pass filter based on interferometry.
The pass band may be adjusted during operation of the image acquisition system, preferably automatically. For such an automatic adjustment of the pass band, the image acquisition system may comprise a filter controller for changing the pass band, preferably depending on an image characteristic of the digital monochrome input images and/or the digital monochrome HDR output image. The image characteristic may comprise at least one of a mean intensity, an intensity histogram, a variance of the intensity and a dynamic range of the fluorescing fluorophore.
According to one embodiment, the pass band may changed automatically based on a feedback loop. As a correcting variable in such a feedback loop, an image characteristic of the at least two monochrome input images and/or the digital monochrome HDR output image may be used. For example, the pass band may be automatically modified to maximize fluorescence intensity in the pass band. The fluorescence intensity in the pass band may be computed by determining an average or mean intensity or brightness of at least one of the at least two digital monochrome input images and/or the digital monochrome HDR output image.
Alternatively or cumulatively, the pass band may be automatically modified, e.g. by the feedback loop, to increase the difference between the mean spectral responsivities in the pass band. Such a difference may be determined by comparing the mean intensity of the at least two digital monochrome input images with one another. Further, alternatively or cumulatively, the pass band may be modified to include the fluorescence wavelength at which the fluorescence intensity is at its maximum level. Such wavelengths may be stored for a variety of different fluorophores.
In another embodiment, the pass band may be changed automatically depending on the fluorophore. For example, the type of fluorophore may be entered manually and the pass band may be adjusted automatically. Such an adjustment may be made using e.g. values for fluorophore-dependent pass bands that have been determined empirically and are stored in the image acquisition apparatus.
In another embodiment, which may be used separately or in addition to the above embodiments, the pass band may be modified for the digital monochrome HDR fluorescence output image to have a mean maximum dynamic range across at least some output pixels of the monochrome input images.
Alternatively or cumulatively, the pass band may be modified to include at least one spectral responsivity located in the middle of a dynamic range of a color sensor. The middle dynamic range is at 50%+/−15% of the overall dynamic range. If e.g. the maximum intensity value at one pixel of an 8-bit color sensor is 255, the middle dynamic range is at an intensity value of 128+/−38.
For performing such an automatic adjustment of the pass band, the filter controller of the image acquisition system may be configured to compute a mean intensity of at least some pixels of at least one monochrome input image, and to modify the pass band depending on that intensity. Alternatively or additionally, the spectral responsivities of the different types of color sensor may be stored in the image acquisition system and used to automatically adjust the pass band. The image acquisition system, in particular the HDR image generator, may be configured to adjust the pass band depending on the spectral responsivities of the different types of color sensor. The image acquisition system, in particular the HDR image generator, may be configured to automatically compute an overlap of the spectral responsivities and to adjust the pass band to at least partly overlap the overlap of the spectral responsivities.
The limiting wavelengths, i.e. the ends, of the co-sensing wavelength band may be specified manually and the automatic adjustment of the pass band may be limited to fall within these limits.
The color camera may be an RGB camera, a CYGM camera or an RGBE camera. The color camera may include a mosaic filter arrangement, such as a Bayer filter, a CYGM filter, an RGB filter and/or it may comprise a Foveon sensor architecture, a color co-site sampling architecture, dichroic mirrors and/or a transparent diffractive filter array. The color camera may also be a multi-spectral or hyper-spectral camera. At least two different types of color sensor are required, preferably at least three. The number of co-sensing bands is preferably one lower than the number of different sensor types. Each type of sensor records a different color band.
The HDR image generator and the filter controller may each be implemented as a hardware device, as a software device, or as a combination of both a hardware and a software device. For example, the HDR image generator and/or the filter controller may comprise at least one CPU, at least one GPU and/or at least one ASIC, and/or a software module which in operation carries out the required function. It is to be noted that, in operation, the software may cause structural changes to the hardware device so that the latter may operate as an HDR image generator and/or a filter controller. These operational changes may e.g. relate to changes of electronic component such as transistors and storage elements.
According to one embodiment, the at least two digital monochrome input images are part of a single frame. The at least two monochrome input images are thus recorded at substantially the same time, so that there is no motion blurring in the resulting HDR output image.
The invention in particular may relate to a medical observation apparatus, such as a microscope or an endoscope, comprising an image acquisition system in one of the above embodiments and/or adapted to carry out the above-described image processing method.
Finally, the invention relates to a non-transitory computer-readable medium storing a program causing a computer to execute the image processing method according to any of the above embodiments.
In the following, an exemplary embodiment of the invention is described with reference to the drawings. In the drawings, elements that correspond to one another with respect to at least one of function and structure are provided with the same reference numerals.
The combination of features shown and described in the figures is used for explanation only and does not limit the invention to the combination shown and described. In particular, one or more of the above-described optional features may be omitted from the embodiment if the technical effect obtained is not needed for a particular application; and vice versa: one or more additional features described as optional above may be added to the shown embodiment if the technical effect of that one or more features is needed for a particular application.
In the figures:
An image acquisition system 1 according to the invention may comprise a color camera 2, such as an RGB camera, a multispectral camera or a hyperspectral camera. An RGB camera 2 typically records a digital color input image 4 comprising three different monochrome images 6. Each of the digital monochrome input images 6 has been recorded by a different type of color sensor 8, as shown in
The different digital monochrome input images 6 may be generated by e.g. providing a mosaic filter arrangement 12 such as a Bayer filter, a CYGM filter or an RGBE filter. Only for exemplary purposes,
The invention works for any of these configurations having at least two different types 16, 17, 18 of color sensor 8. However, three or more different types of color sensor are preferred. Preferably, the different digital monochrome input images 6 are part of the same frame 21, i.e. have been taken simultaneously.
The image acquisition system 1 is adapted to record images 4, 6 of a fluorescing fluorophore 22 (
The image acquisition system 1 may comprise a light source 26 which generates light at wavelengths that trigger fluorescence in the fluorophore 22. The light 28 from the light source 26 is directed onto an object 30 containing the fluorophore 22, such as the live tissue 24. The fluorescence emitted from the fluorophore 22 is recorded by the color camera 2 in the at least two digital monochrome input images 6.
The light source 26 may also include white light so that a visible-light image can be recorded at the same time as the image containing the fluorescence light. To acquire a visible-light image, another camera (not shown) may be used which is decoupled by an optical filter arrangement (not shown) from the fluorescence excitation and emission bands, so that these wavelengths are not recorded in the visible-light images.
In this case, the color camera 2 may be solely devoted to recording fluorescence images and may be decoupled from both the excitation wavelengths of the light source 26 and any white light by a band-pass filter 32. The band-pass filter 32 may comprise one of more optical filters which, in combination, result in only wavelengths that fall within a pass band 34 being able to pass through the band-pass filter, as shown schematically in
The band-pass filter 32 is located in front of the camera 2 in an optical path 36 of an optical acquisition system 37 of the image acquisition system 1, which may also comprise the camera 2. If an endoscope is used, the optical path is directed through an optical fiber (not shown) to the object 30.
For computing digital monochrome HDR output images 38, the image acquisition system 1 comprises an HDR image generator 40, which may be a hardware device 41, such as a CPU of a general-purpose computer 42, and/or may comprise a GPU, FPU and/or an ASIC. Alternatively or cumulatively, the HDR image generator 40 may consist of or comprise a software module. The digital color input image 4 or the digital monochrome input image 6, respectively, may be input into the HDR image generator 40 via an input interface 44, which may be configured to accept e.g. RGB, HDMI, DVI or other input formats suitable for transmitting image or video data.
The digital monochrome output image 38 may be output in the same or in a different image format using an output interface 46.
The HDR image generator 40 is adapted to compute the digital monochrome HDR output image 38 from at least two of the digital monochrome input images 6. The digital monochrome HDR output image 38 is displayed on at least one display device 48, such as an eyepiece 50, AR goggles 52 or a monitor 54. As any of the digital color input image, the digital monochrome input image 6 and the digital monochrome HDR output images 38 may be stereoscopic, one or more of the display devices 48 may also be stereoscopic.
The HDR image generator 40 may be adapted to blend the digital monochrome HDR output image 38 with a visible-light image and/or to assign a pseudo-color to the digital monochrome HDR output image 38 depending on the intensity value at an output pixel 56.
It is to be noted that the digital input images 4, 6 and the digital monochrome HDR output image 38 may be part of a time-series of input and output images which result in a video sequence. Each digital monochrome HDR output image 38 is preferably computed and output in real-time by the HDR image generator 40, and displayed in real-time by any of the display devices 48. Moreover, if it is stated in this description that the digital monochrome HDR output image 38 is based on the digital monochrome input images 6, this also includes cases in which two or more monochrome input images 6 of one or more types of color sensor 8 may be averaged to result in a single digital monochrome input image 6 which is then used to compute the digital monochrome HDR output image 38.
The computation of the digital monochrome HDR output image 38 is described in closer detail with reference to
In the topmost part of
Also shown in the topmost part of
Moreover, in co-sensing wavelength band 64, the mean spectral responsivity 70 of the one type 16 of color sensor is different from the mean spectral responsivity 74 of the other type 18 of color sensor. The mean spectral responsivity 70 can be calculated by any procedure for averaging, such as the geometric, arithmetic and/or harmonic mean value or the median of the respective spectral responsivity 58, 60, 62 within the respective co-sensing wavelength band 64, 66, 68. Preferably, the mean spectral responsivities 70, 72, 74 of at least two types of color sensor, preferably of at least three types of color sensor, are different.
In co-sensing wavelength band 66, the spectral responsivities 58, 60, 62 overlap so that each color sensor 8 will yield a signal if illuminated with wavelengths that fall within the co-sensing wavelength band 66. Again, reference numerals 70, 72, 74 denote the mean spectral responsivities. In co-sensing wavelength band 68, the relative mean spectral responsivities 70, 72 and 74 have shifted compared to co-sensing wavelength band 66. From the top of
For example, the co-sensing wavelength band may be selected such that the mean spectral responsivities 70, 72, 74 are, at least approximately, equidistantly spaced in a linear or log scale. Hereby, variations of +/−20% of the difference between subsequent mean spectral responsivities 70, 72, 74 still count as equidistant.
In another approach, which can be used alternatively or cumulatively, a co-sensing wavelength band is selected where there is at least one mean spectral responsivity 70, 72, 74 at approximately half the normalized spectral responsivity R0.5. Again, a variation of +/−20% relative to R0.5 is still considered to correspond to R0.5.
Further, again alternatively or cumulatively, the co-sensing wavelength band may be shifted so that the differences between the mean spectral responsivities 70, 72, 74 are maximized in the chosen co-sensing wavelength band. Such a maximization of the differences may be performed by maximizing the sum, product or any other norm of the differences between the mean spectral responsivities.
As used in this application, the spectral responsivity of a type of color sensor is representative of its response to incident light at a specific wavelength λ. The spectral responsivity may specifically relate to the ratio of the electrical output per optical input, i.e. not only include the optical wavelength-dependent properties of the type of color sensor, but also its electrical and electronic properties. At the very least, however, the spectral responsivity relates to the optical properties of the respective type of color sensor.
For creating and/or adjusting the co-sensing wavelength band 64, 66, 68, the band-pass filter 32 (
For example, a co-sensing wavelength band 64, 66, 68, or any other co-sensing wavelength band not shown in
It is, however, preferred that the band-pass filter 32 is a tunable band-pass filter 32. This means that at least one of the following filter characteristics 77 may be modified: the upper cut-off frequency or, equivalently, the lower cut-off wavelength λl; the upper cut-off frequency or lower cut-off wavelength λu; the center frequency or center wavelength λc; the bandwidth 75 (
Using a tunable band-pass filter 32 allows the pass band 34 to be quickly adjusted in order to select a co-sensing wavelength band 64, 66, 68 which yields the best results, in particular maximum contrast, for the digital monochrome HDR output image 38 (
Moreover, if a tunable band-pass filter 32 is used, an automatic adjustment, in particular a feedback control, of any of the filter characteristics 77 may be implemented using e.g. a filter controller 78 (
As an example of such a feedback control, the HDR image generator 40 may be configured to compute the dynamic range within each of the digital monochrome input images 6 and/or of the digital monochrome HDR output image 38 and adjust at least one of the filter characteristics 77 to maximize this dynamic range. For this, the HDR image generator 40 may comprise a memory section 80 (
In the bottom part of
In the case of the fluorophore having fluorescence spectrum 82, both the co-sensing wavelength bands 64 and 66 may lead to good results. However, although co-sensing wavelength band 64 includes λF from fluorescence spectrum 82, the signal of only two color sensors may be used, as the mean responsivity 60 of the red color sensor 16 is zero or negligible in this co-sensing wavelength band 64. Therefore, depending on the image characteristics of the digital monochrome input images 6 and/or the digital monochrome HDR output image 38, shifting the pass band 34 to a co-sensing wavelength band 66 may be preferable.
This demonstrates that the fluorophore may also be selected depending on the spectral responsivities of the different types of color sensor to achieve optimum results. In particular, the fluorophore should have a strong fluorescence in the available co-sensing wavelength bands and, at the same time, that the different types of color sensor should have sufficiently different spectral responsivities.
In optional step 90, a fluorophore 22 is selected, of which the fluorescence spectrum 82, 84 overlaps with a co-sensing wavelength band. This step ensures that there will be sufficient fluorescence intensity generated in a wavelength band, in which at least two color sensors 8 will produce a sufficiently large output signal.
Further, in step 90, a band-pass filter 32 may be selected, of which the pass band 34 defines a co-sensing wavelength band which is preferably narrower than the fluorescence spectrum of the fluorophore.
In the next step 92, the fluorescence excitation light is provided to trigger the fluorescence in the fluorophore 22.
Next, in step 94, the fluorescence emitted by the fluorophore, preferably along with any reflected light, is filtered by the band-pass filter 32 in order to restrict it to the co-sensing wavelength band.
In step 96, digital monochrome input images 6 are acquired, preferably for at least two, most preferably for each color channel of color camera 2. The digital monochrome input images 6 are preferably acquired at the same time, i.e. as part of the same frame 21 or the same digital color input image 4. The images 4, 6 may be stereoscopic.
In step 98, the digital monochrome HDR output image 38 is computed from the digital monochrome input images 6. The digital monochrome HDR output image 38 may be three-dimensional or stereoscopic.
In step 100, the digital monochrome HDR output image 38 is displayed on at least one display device 48.
If the digital monochrome HDR output image 38 does not exhibit satisfactory contrast, a new band-pass filter 32, a new pass band 34 and/or a new fluorophore 22 may be chosen in step 102. If a tunable band-pass filter 32 is used, computation of a new pass band 34 may be manually triggered at this stage. Step 102 may be carried out at any time once a digital monochrome HDR image 38 is displayed.
In step 104, an image parameter, such as the dynamic range in at least one of the digital monochrome input images 6 and/or of the digital monochrome HDR output image 38 may be computed. In addition or alternatively, the location of the mean spectral responsivities in the current co-sensing wavelength band may be determined. From this, and by e.g. using a look-up table representing the spectral responsivities of the different types of color sensor used in the camera, a pass band 34 having filter characteristics 77 that maximize the contrast in the digital monochrome HDR output image 38 is computed. The required filter characteristics 77 may be displayed for a user to manually select a band-pass filter 32 or manually adjust a tunable band-pass filter 32. The filter characteristics may also be communicated electronically and automatically for the filter controller 78 to adjust the pass band 34.
In step 106, the band-pass filter 32 is automatically adjusted or exchanged manually to exhibit to the new filter characteristics 77. Steps 104 and 106 establish a feedback control 108 for automatically adjusting the optimum pass band 34.
The above steps can be performed in real-time so that the computation of the digital monochrome HDR output image 38 can be performed at the frame rate of a video sequence or stream.
| Number | Date | Country | Kind |
|---|---|---|---|
| 18214911 | Dec 2018 | EP | regional |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20090173892 | Courtney et al. | Jul 2009 | A1 |
| 20110009163 | Fletcher et al. | Jan 2011 | A1 |
| 20110064296 | Dixon | Mar 2011 | A1 |
| 20120150045 | Matsumoto et al. | Jun 2012 | A1 |
| 20150022647 | Takei | Jan 2015 | A1 |
| 20150297073 | Nguyen et al. | Oct 2015 | A1 |
| 20170112354 | DiCarlo | Apr 2017 | A1 |
| 20180049644 | Themelis | Feb 2018 | A1 |
| 20180373017 | Dixon | Dec 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 2014125724 | Aug 2014 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20200196846 A1 | Jun 2020 | US |
