Patent Application
20070115376
This application claims the benefit of foreign priority of JP 2005-335251 filed Nov. 21, 2005, the contents of which are hereby incorporated by reference.
The present invention relates to an image pickup apparatus having two image sensors, particularly an image pickup apparatus having two image sensors that is suitable for mounting at the end of a medical endoscope.
The image quality of an endoscope, more particularly a so-called videoscope, wherein a solid state image pickup device is arranged at the end of an insertion part, strongly depends on the design of the solid state image pickup device. Previously, high image quality has been sought by reducing the size of the individual detecting unit cells of the solid state image pickup devices, thereby increasing the number of pixels. However, the physical limit of miniaturization of the unit cell size has nearly been reached. Therefore, it will soon be necessary to adopt a different approach to improving image quality. One such approach is to use a plurality of image sensors, more particularly, a plurality of solid state image pickup devices. However, increasing the number of image pickup devices as done previously markedly increases the size of the image pickup apparatus as compared to using a single image pickup device. Therefore, it had been commonly believed that this approach was not practical in the case of videoscopes. Although with today's technology it appears to be impossible to mount a three-sensor image detecting device in a videoscope in order to provide high quality imaging, there is currently a need for an improved two-sensor image detecting device that can be made sufficiently small for use in a videoscope.
The following remarks discuss various prior art two-sensor image detecting devices. Japanese Laid-Open Patent Applications S59-127492 and 2005-210359 disclose different light paths and detector configurations using two image sensors. Japanese Laid-Open Patent Application H5-122710 and Japanese Patent No. 2,929,655 disclose two-sensor image detecting devices with particular color shading. Japanese Laid-Open Patent Applications 2004-258497, H10-304388, H10-341449, H8-68904, and H5-244610 disclose two-sensor image detecting devices using a variety of prism sizes and shapes for splitting light and detection by two solid state image sensors or detectors.
Japanese Laid-Open Patent Application H5-122710, mentioned above, discloses a composition that takes into account the fluctuations of spectral characteristics due to the fluctuations of the angle of incidence by using a dichroic mirror made from a thin plate. However, the obliquely arranged thin plate mirror has the problem that eccentric aberrations may occur in the transmission optical path. Moreover, it is difficult for the thin plate mirror to adopt a frame structure with high reliability compared to a prism, and deformation or displacement due to changes of temperature easily occur, making it difficult to maintain relative position accuracy between the two detectors over time. In addition, in an endoscope exposed to moisture in various situations, moisture easily invades into the space at the front side of the solid state image pickup device, and a deterioration in image quality easily occurs.
Japanese Patent No. 2,929,655, mentioned above, discloses a device with a filter arrangement for the purpose of color shading reduction. The configuration arranges an interference film having a so-called trimming function under specified conditions. In this configuration, the prism configuration becomes complicated because of the arrangement of an inclined interference film, and providing a small size mounting becomes difficult.
Japanese Laid-Open Patent Application 2004-258497, mentioned above, discloses a device designed for miniaturization of the prism structure. However, the prism structure has the problem of being radially oversized for an endoscope.
A prism configuration whose size is radially small and thus suitable for use in an endoscope is disclosed in S57-5537. However, the prism configuration disclosed in S57-5537 tends to make the prism configuration be long in the optical axis direction, which is disadvantageous in that it results in increased length of the rigid end in an endoscope having a curved mechanism at the end. Japanese Laid-Open Patent Applications H10-304388 and H10-341449, both mentioned above, disclose prism configurations wherein two triangular prisms are adhered to form a cube, and Japanese Laid-Open Patent Application H8-68904, also mentioned above, discloses a modification of a triangular, prism-type device for the purpose of preventing ghost light. The prism configuration of Japanese Laid-Open Patent Application H8-68904 is complicated. The prism configurations of Japanese Laid-Open Patent Applications H10-304388 and H10-341449 are simpler and small in size; however, they create problems from the standpoint of securing desirable optical properties in the imaging.
In an endoscope, in order to respond to size and tolerance requirements, a color shading countermeasure must be adopted and assembly properties must be secured after designing a simple prism configuration. However, the two-sensor image detecting device configurations shown in the patent literature discussed above lack consideration of these technical problems. In addition, as a problem unique to the two sensor configurations, the investment needed to resolve these technical problems is great. In the case of a configuration designed to separate green light and magenta light (which is the configuration that can improve the image quality the most), because two separately designed types of solid state image pickup devices are normally required, development expenses and production time are high as compared to using a single image sensor or even three image sensors.
The present invention relates to an image pickup apparatus having two image sensors with a configuration that enables realizing a size suitable for mounting in the insertion end of an endoscope, that provides high quality imaging without using a complex structure or special elements, that enables the problem of color shading to be solved through proper spectral design, that resolves problems of changing environments and assembly problems, and that improves the return on investment related to solid state image pickup devices.
The present invention will become more fully understood from the detailed description given below and the accompanying drawings, which are given by way of illustration only and thus are not limitative of the present invention, wherein:
For convenience herein, at times, an image pickup apparatus having two image sensors will be referred to simply as a “two-sensor image detecting device”, and an image pickup apparatus having three image sensors will be referred to simply as a “three-sensor image detecting device”. The two-sensor image detecting device of the present invention addresses the problem of color shading as follows. The two-sensor image detecting device of the present invention includes, arranged along an optical axis, a prism group that includes a color separation coating that reflects green light, a first solid state image pickup device that detects images formed by the light reflected by the color separation coating, and a second solid state image pickup device that detects images formed by light transmitted through the color separation coating. Additionally, the second solid state image pickup device directly supports a filter for transmitting red light and a filter for transmitting blue light that has been transmitted through the color separation coating. Furthermore, the two-sensor image detecting device of the present invention satisfies the following conditions:
5 nM≦λiR−λcR≦35 nm Condition (1)
−20 nm≦λcB−λiB≦20 nm Condition (2)
λiR−λcR>λcB−λiB Condition (3)
where
Hereinafter, in order to simplify explanations, each color band of the three primary colors, red, green, and blue, may be simply described as R, G, and B, respectively. Similarly, the filter for transmitting red light and the filter for transmitting blue light supported on the second solid state image pickup device may be simply described as an “R filter” and a “B filter”, respectively.
In the two-sensor image detecting device of the present invention, a G band with a large contribution to luminance information is imaged by one solid state image pickup device. In order to achieve this, as the color separation coating, two types, a G reflection type and a G transmission type, can be considered. The G reflection type, which provides more degrees of freedom in an interference film design, is used in the present invention. Moreover, in order to separate magenta light, which has been transmitted through the color separation coating, into R and B bands, primary color filtering into the R and B bands is used for the other solid state image pickup device. In general, although the R filter and the B filter have been widely adopted for a solid state image pickup device for a digital still camera having Bayer's arrangement of primary colors, because the optical absorption property of organic pigments is used, the degree of freedom in the spectral characteristic selection is not as high as when using multilayer optical films. However, because this is an absorption-type filter, it has the advantage that the risk of flare and ghost lights, which easily occur with interference filters, is low. Therefore, in the present invention, color shading can be greatly reduced without using an interference trimming filter where flare and ghost lights easily occur and without using an absorption trimming filter that complicates the shape and size of the optical unit outside the solid state image pickup devices. In the present invention, easily produced R filters and B filters themselves can achieve a trimming function that advantageously cooperates with the spectral characteristics of the color separation coating.
The R filter pixel spectral sensitivity curve 51 is standard for a primary color Bayer's type solid state image pickup device, and it has the above defined wavelength λiR on the G band boundary. The B filter pixel spectral sensitivity curve 52 is also standard for a primary color Bayer's type solid state image pickup device, and it has the above defined wavelength λiB on the G band boundary. For the purpose of reflecting G light with a color separation coating, the spectral transmittance curve 53 of such a coating will inherently transmit magenta light, and it has the above defined wavelength λcR at the R band boundary side and the above defined wavelength λcB on the B band boundary side.
Hereinafter, in order to simplify explanations, the light energy per R, G or B band is expressed as ER, EG or EB, respectively. Wavelength shifts generated by fluctuations of the angle of incidence of light onto the color separation coating are conceptually equivalent to a shifting of the spectral transmittance curve 53 of
By satisfying Conditions (1), (2), and (3) above, color shading is controlled without needing to use a complex configuration. The present invention not only achieves this goal, as it also results in providing bright images.
Condition (1) above limits the color difference fluctuation on the R side when the color separation coating spectral transmittance curve 53 shifts towards longer wavelengths. It does this by setting the 50% relative transmittance wavelength of the color separation coating at a shorter wavelength as compared to that of the R filter. In general, since the spectral sensitivity curve 51 of the R pixel filters changes rapidly on the G side, if an overlap occurs due to wavelength shifts, the color difference fluctuation will be large. Because the λiR to λcR wavelength range can be established as a low sensitivity wavelength range on the boundary between the R band and the B band, the ER fluctuation is controlled by setting the quantity λiR−λcR at a positive predetermined value and appropriately establishing this low sensitivity wavelength band. Furthermore, during critical observations that relate to the main uses in medical endoscopes, there is no purpose in having a specific spectral characteristic in the wavelength band from λiR through λcR. Therefore, there is no problem with this band being established in the low sensitivity wavelength range. However, in order to have as great an EG as possible and not to decrease the brightness, it is desirable that the quantity λiR−λcR not be too large. Condition (1) above specifies an appropriate value for the quantity λiR−λcR. If the lower limit of Condition (1) is not satisfied, the ER fluctuation becomes too large, and if the upper limit of Condition (1) is not satisfied, EG becomes too small and the brightness decreases. Therefore, is desirable that Condition (1) above be satisfied.
Additionally, it is even more preferable that the following Condition (4) is satisfied:
10 nM≦λiR−λcR≦25 nm Condition (4)
where λiR and λcR are defined as set forth previously.
Condition (2) above relates to the boundary of the B band and the G band, and the circumstances of transmittance and spectral sensitivity are slightly different from those related to Condition (1). In general, since the spectral sensitivity curve 52 of the B filter pixel changes less rapidly on the G side, color difference fluctuations in the case of the occurrence of overlap due to wavelength shifts will be comparatively small. Therefore, specifying λcB and λiB to be nearby wavelengths enables the achievement of a balance between color shading control and securing EG for bright imaging. Moreover, the wavelength range between λcB and λiB is a range where light absorption by hemoglobin is high, and imaging in this wavelength range is vital for emphasizing blood vessel contrast at the time of critical observations. Consequently, it is not desirable to place a condition (related to the low sensitivity wavelength region at the boundary of the B band and G band) similar to that of Condition (1) above. Instead, Condition (2) specifies an appropriate value of the quantity λcB−λiB. If the lower limit of Condition (2) is not satisfied, the EB fluctuation will become excessive. On the other hand, if the upper limit of Condition (2) is not satisfied EG becomes insufficient and the brightness decreases. Therefore, it is desirable to satisfy Condition (2).
Condition (3) above directly relates to relative magnitude of the quantities λiR−λcR and λcB−λiB used in Conditions (1) and (2) above. If Condition (3) is not satisfied, the balance between the color shading control and securing brightness, specifically related to EG, cannot be achieved, and this of course is undesirable. In addition, it is preferable that the following Condition (5) be satisfied:
λiR−λcR>λcB−λiB+10 nm Condition (5)
where
λiR, λcR, λcB, and λiB are defined as set forth previously.
With particular regard to resolving assembly problems and problems arising from changing environmental conditions, the two-sensor image detecting device of the present invention uses: a first prism that includes, arranged in order along an optical axis, a first plane of incidence, a first interface inclined between forty and fifty degrees relative to the first plane of incidence, and a first output plane; a second prism that includes a second plane of incidence joined to the first interface and a second output plane; a color separation coating arranged on the first interface that reflects green light; a first solid state image pickup device that is joined with the first output plane via a first sealing glass plate; and a second solid state image pickup device that is joined with the second output plane via a second sealing glass plate, that detects images formed by light transmitted through the color separation coating, and that directly supports a filter for transmitting red light and a filter for transmitting blue light that has been transmitted through the color separation coating. Additionally, the first prism and the second prism are formed of the same glass material Gp having a refractive index of 1.76 or less and an Abbe number of 52 or greater, the first sealing glass plate and the second sealing glass plate are made of the same glass material Gi having a lower ultraviolet transmittance than the glass material Gp of the first prism and the second prism, and the following items, namely, the first prism and the second prism, the first prism and the first solid state image pickup device, and the second prism and the second solid state image pickup device are joined using ultraviolet hardening optical adhesives having a refractive index that differs from the refractive index of the glass material Gp by 0.15 or less.
Furthermore, the refractive indices and the Abbe numbers used in the explanations herein of the present invention are all measured relative to the d-line.
In
In the present invention, where the prisms and the solid state image pickup devices are all adhered using optical adhesives, consideration of registration shifts in the adhering and hardening process is also important. In order to adjust the registration before hardening and to adhere the prisms and the solid state image pickup devices and in order to harden the optical adhesives in a short time while proper registration is maintained, it is necessary to use adhesives having ultraviolet hardening properties. However, there is a trade-off between where excessive irradiation of ultraviolet rays to the solid state image pickup devices damages organic raw materials that may be used in microlenses or spectral filters integrated with the solid state image pickup devices and where insufficient irradiation of ultraviolet rays may result in inadequate hardening of the optical adhesives. Therefore, design choices and specifics of the manufacturing processes should be considered together as affecting one another.
Accordingly, in the present invention, the design of the two-sensor image detecting device pays particular attention to the fact that optical adhesives hardened by ultraviolet rays are used, as set forth in the following remarks. The choices of the refractive index being 1.76 or less and the Abbe number being 52 or greater in the glass material Gp for the prisms enables assuring high ultraviolet transmittance up to 300 nm. With this design, when hardening the optical adhesive between the two prisms and between the prisms and the solid state image pickup devices by ultraviolet rays, the optical path of irradiation by transmission through the prism becomes usable, increasing the degrees of freedom for the irradiation direction in the ultraviolet ray irradiation device so that choices in the setting processes become greater. Moreover, there is another advantage of using low-dispersion glass with a high Abbe number, namely, enabling the reduction of focal displacements between different color bands generated due to the axial chromatic aberration of the prisms. In addition, setting the ultraviolet transmittance of the glass material Gi for the sealing glass plates lower than that of the glass material Gp for the prisms enables setting the ultraviolet irradiation conditions to simultaneously prevent damage to the solid state image pickup devices by ultraviolet rays and to assure complete hardening of the adhesive. The ultraviolet wavelength range where the transmittance of the glass material Gp is high and that of the glass material Gi is low is a wavelength range where light may be transmitted through the prism and absorbed by the sealing glass plate. Therefore, setting this wavelength range to be the main wavelength range of the ultraviolet ray irradiation enables complete hardening of the optical adhesives without damaging the solid state image pickup devices even if the irradiation energy and time periods are increased. The reduction of the difference of the refractive indices between the adhesive and the glass material Gp contributes to reducing the eccentric aberrations generated when transmitting light through the adhesive layer inclined at 40° to 50°.
With particular regard to the problem of improving the return on investment related to solid state image pickup devices, the two-sensor image detecting device of the present invention includes a prism group that includes a color separation coating that reflects green light, a first solid state image pickup device that detects images formed by the light reflected by the color separation coating, and a second solid state image pickup device that detects images formed by light transmitted through the color separation coating, with the second solid state image pickup device supporting a filter for transmitting red light and a filter for transmitting blue light transmitted through the color separation coating.
The filter for transmitting red light and the filter for transmitting blue light are “on-chip”, vertical stripe filters. Additionally, the first solid state image pickup device and the second solid state image pickup device read from a field of pixels having the same number of pixels of the same size for each of the first solid state image pickup device and the second solid state image pickup device and each of the first solid state image pickup device and the second solid state image pickup device provides readouts based on combining two vertically adjacent pixels.
In the above configuration, the return on investment related to the solid state image pickup devices is improved by using the same solid state image pickup device normally used in a single-sensor image pickup device in a two-sensor configuration. Single-sensor image detecting devices for complementary viewing currently use color image pickup devices in diffusion-type endoscopes where size, cost, and brightness are emphasized more than image quality, and it appears this situation will continue in the future. In these color solid state image pickup devices, because it is conventional during image readout to combine information from two vertically adjacent pixels of the image field, as discussed, for example, in Japanese Laid-Open Patent Application H5-244610, if the number of pixels and the pixel dimensions can be standardized with a similar image reading configuration for a two-sensor configuration, a semiconductor wafer structure, which excludes the optical structure in front of the photoelectric conversion structures, can be standardized. Furthermore, in order to combine two vertical pixels in the second solid state image pickup device that directly supports the spectral filter arrangement “on chip”, it is necessary to use a vertical-stripe-type filter wherein color filters of the same color are aligned vertically.
If the techniques described above are adopted, it becomes possible to adapt the color solid state image pickup devices of single-sensor image pickups that have been developed for diffusion-type endoscopes to a two-sensor arrangement by making relatively minor changes in, for example, the design of the “on chip” micro lenses and color filters that are supported on the wafer structure. Thus, conversions can be made to diffusion-type endoscopes based on a good return on investment and a shortening of development time. On the other hand, if the techniques described above are not adopted, achieving the desired structures becomes difficult due to a greater investment being required and the development time being increased.
As described above, the two-sensor image pickup apparatus of the present invention has a spectral design that enables color shading to be controlled using a simple prism configuration of appropriate size and quality that is mountable to the end of an endoscope. Moreover, the two-sensor image pickup apparatus has excellent environmental resistance and assembly properties, and provides an excellent return on investment due to only small modifications being needed to the conventional solid state image pickup devices used.
Four embodiments will now be discussed in detail with further reference to the drawings and with reference to various tables.
The primary elements of the two-sensor image detecting device 1 are the first solid state image pickup device 2, the second solid state image pickup device 3, the first prism 4, and the second prism 5. A two-dimensional detector having a CCD or CMOS structure is used for the first solid state image pickup device 2 and the second solid state image pickup device 3. The image plane of device 2 is protected by a first sealing glass plate 8, and the image plane of device 3 is protected by a second sealing glass plate 9. The first prism 4 has three optical mirror surfaces, a first incident plane surface 13, a first interface 14, and a first output plane 15. The first interface 14 is inclined at 45° relative to the first incident plane surface 13, and the first interface 14 has a color separation coating that reflects green light. The second prism 5 has two optical mirror surfaces, a second incident plane surface 16, and a second output plane 17. For the first prism 4 and the second prism 5, the same glass with moderate or low refractive index and low dispersion is used. For example, S-BSL7 (refractive index: 1.516, Abbe number: 64.1) by Ohara, Inc. may be used. The above four basic elements are integrally combined by adhesive layers 10, 11 and 12 made of the same adhesive material with a refractive index of 1.51 and having ultraviolet hardening properties. The size of the first solid state image pickup device 2 and the second solid state image pickup device 3 is one-sixth inch; the pixel pitch is approximately 2 μm; and the registration displacement tolerance is 1 μm or less. Moreover, the lengths of the matching surfaces of the first prism 4 and the second prism 5 are approximately 2 mm to 3.5 mm, which is considerably smaller than prisms generally used in multiple detector imaging devices for commercial use. The color separation coating is formed by alternately laminating Y2O3 (refractive index: 1.86) and Ta2O5 (refractive index: 2.21), and a coating having 24 layers or more is used.
The first solid state image pickup device 2 images a green light (G optical path in
The second solid state image pickup device 3 images a magenta light (R/B optical path in
The first solid state image pickup device 2 and the second solid state image pickup device 3 read pixels from the image field by combining two vertical pixels. Associated with this, the second solid state image pickup device 3 enables the combination of the two vertical pixels as a vertical stripe type arrangement of two colors, R and B, as shown in the arrangement of the directly supported filter in
In all embodiments of the present invention, for the purpose of sealing the components, the prism group in the two-sensor image detecting device 1 and the solid state image pickup devices are integrated without any air gaps. Furthermore, with regard to the objective lens 6 being formed from a circular lens and/or filter, it is possible to adopt measures to exclude moisture by secure sealing properties of the frame structure at the objective lens 6 that is comparatively resistant to thermal expansion or shrinkage, so that there is no problem with air gaps in this portion.
The prism configuration of Embodiment 1 is constructed with two triangular prisms, each prism having a slant face of 45°, which is convenient for miniaturization. This configuration is the most useful for both radial and longitudinal miniaturization, and it is also the simplest.
The following processes are performed in the following order in the assembly of the two-sensor image detecting device 1 of Embodiment 1:
In Embodiment 1, because the color separation coating is located on the first interface 14, the image formation position of the first prism 4 and the first solid state image pickup device 2 does not depend upon the second prism 5, and the image position and the optical path length at the G image side are determined at the time of the completion of Process (1). Therefore, the result of Process (1) will not be affected by Process (2) that follows Process (1). Furthermore, it is possible to arrange that the color separation coating be on the second incident plane surface 16 of the second prism 5. However, in this case, because the image position and the optical path length vary according to the thickness of the adhesive 10, the result of Process (1) is affected by Process (2), complicating the operation of Process (2). Consequently, it is desirable that the color separation coating be arranged on the first interface 14. In Process (2), adjustment is possible using the G image determined in Process (1) as the standard, and the optical path difference between the two detectors can be adjusted by sliding the second prism 5 parallel to the slant face that extends at 45°. Furthermore, in Process (2), thickness variations in the sealing glass plates 8 and 9 are considered at the time of adjusting the optical path difference, and it is desirable that the second solid state image pickup device 3 be temporarily secured in the vicinity of the second prism 5 and the two solid state image pickup devices 2 and 3 are in the imaging state, so that adjustment errors related to image processing are detected. Moreover, registration shift detection related to image processing is effective even in Process (3). Processes (1), (2), and (3) in the configurations described above are effective in resolving assembly problems and problems of changing environments in terms of setting manufacturing processes that take into account the exposure of the solid state image pickup devices to ultraviolet light.
A conventionally known problem in imaging devices is the color shading caused by the characteristics of angle and polarized light related to the color separation coating being arranged on the slant face of 45°. However, in the present invention, the configuration overcomes this color shading problem. In the spectral characteristics of the elements of Embodiment 1 shown in
λcB=500 nm, λcR=562 nm
λiB=500 nm, λiR=577 nm
λiR−λcR=15 nm
λcB−λiB=0 nM
The light energies ER, EG and EB, corresponding to R, G or B bands, respectively (as defined previously) can be calculated as the integrated values of the spectral sensitivities, shown by the corresponding curves of
The data in the parentheses in Table 1 above indicate the percentage variation from the quantities at the design value with no wavelength shift. In Embodiment 1, because the value of the quantity λiR−λcR is set at a relatively large positive value, the percentage variation of ER/EG from the design value is very small and the color shading appears primarily as a variation of EB/EG. However, the total variation of the color shading is approximately 15% according to the absolute value simple sum of the percentage variation of the design values of ER/EG and the percentage variation of the design values of EB/EG. In the case of Embodiment 1, the absolute values simple sum with −15 nm of wavelength shift is 12% and the absolute values simple sum with +15 nm of wavelength shift is 14%, and since both are within 15%, approximately ±15 nm of wavelength shift can be allowed in Embodiment 1.
Next, as an example in the case of not simultaneously satisfying Conditions (1)-(3) of the present invention, an example where λcB and λcR shift by +20 nm compared to those in Embodiment 1 is considered. In this new example, λiB is still 500 nm. This results in the following numerical values related to Conditions (1)-(5) above:
λcB=520 nm, λcB=582 nm
λiR−λcR=−5 nm
λcB−λiB=20 nm
For this new example, various values of ER/EG, EB/EG and the wavelength shift that relate to color shading are shown in Table 2 below.
In this case, compared to Embodiment 1, the percentage variation of ER/EG is greater, but the percentage variation of EB/EG is smaller. However, the absolute value simple sum with −15 nm of wavelength shift is 14% and the absolute value simple sum with +15 nm of wavelength shift is 19%, so that the color shading generation is greater than that in Embodiment 1. Consequently, the wavelength shift of +15 nm, which is tolerable in Embodiment 1, is exceeded in this new example, which is undesirable.
As described above, the angle of the first interface 14 to the first incident plane surface 13 is allowed to vary from 45°. However, if the angle becomes less than 40° or greater than 50°, the inclination of the first solid state image pickup device 2 becomes greater, which is associated with an increasing radial dimension. Therefore, this is not preferable.
The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention. Rather, the scope of the invention shall be defined as set forth in the following claims and their legal equivalents. All such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2005-335251 | Nov 2005 | JP | national |
