Patent Grant
11146723
The present application is based on PCT filing PCT/EP2017/082891, filed Dec. 14, 2017 which claims priority to EP 16204872.2, filed Dec. 16, 2016, the entire contents of each are incorporated herein by reference.
The present disclosure relates to a capturing an image of a scene. In particular, the present disclosure relates to capturing an image in an optical scope system.
The “background” description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in the background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present disclosure.
A problem with certain image capture devices such as those used in industrial instruments (such as industrial endoscopes) or medical instruments (such as medical endoscopes) is the limited depth of field at which high spatial frequencies can be obtained in order to capture a sufficiently sharp image. In order to improve the depth of field, the size of the aperture through which light travels to form the image to be captured can be reduced (increasing the so-called F number (F#) of the image capture device). This leads to a larger depth of field, but, in turn, reduces the resolution/in-focus sharpness (due to diffraction) and increases the noise of the captured image (due to there being less received light and thus a reduced signal to noise ratio). In other words, there is a trade off between having a larger depth of field and having greater in-focus sharpness and low noise images.
Furthermore, as the form factor of such image capture devices is reduced (for example, to allow smaller form factor endoscopes), thus requiring image capture sensors with smaller pixel sizes, the problems of this approach are set to get worse.
Moreover, in many endoscope applications such as surgical endoscopy or industrial endoscopy, high resolution images such as 4K, 8K or the like are also desired. This means that the imager becomes larger and so there is a trade-off required between the size of imager and the depth of focus, In other words, a problem exists of how to provide an extended depth of field when high resolution images (such as those provided using larger imagers) are required.
It is an aim of the present disclosure to address at least one of these problems.
The present technique provides an optical scope system for capturing an image of a scene, the optical device comprising: a plurality of image sensors each operable to capture a respective initial image of the scene; a lens arrangement operable to receive light from the scene and to form each initial image on each respective image sensor, each image sensor being located at a different respective distance from the lens arrangement; and an image processor operable to generate the captured image of the scene on the basis of image data from one or more of the captured initial images.
This arrangement is advantageous because an image is produced that has an improved depth of field for a given resolution/in-focus sharpness of image.
The present technique also provides a method of capturing an image of a scene, the method comprising: capturing a plurality of initial images of the scene, each initial image of the scene being captured using a respective one of a plurality of image sensors, wherein each image sensor is located at a different respective distance from a lens arrangement operable to receive light from the scene and to form each initial image on each respective image sensor; and generating the captured image of the scene on the basis of image data from one or more of the captured initial images.
The foregoing paragraphs have been provided by way of general introduction, and are not intended to limit the scope of the following claims. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings.
A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views.
In operation, the lens arrangement 104 receives light from the scene and forms a plurality of initial images of the scene at the image sensor 802 using the received light. The image sensor 802 then captures each initial image of the scene (that is, it captures each initial image of the scene as an electronic image). These electronic images are then processed by the image processor 900 so as to form a final image of the scene on the basis of image data from one or more of the captured initial images. The final image of the scene is then output for display and/or storage by the output unit 904.
In this particular embodiment, the light from the scene used to form the captured image is received from a medical instrument (in this example, a medical endoscope 114, such as a surgical endoscope). That is, the lens arrangement 104 receives light from the scene captured by the medical instrument and forms the initial images of the scene at the image sensor using this received light. It will be appreciated, however, that light from the scene may be received from any other type of instrument, as long as the light is focused by the lens arrangement 104 at the image sensor 802. An example of another type of instrument is an industrial instrument such as an industrial endoscope. In the example of
The image sensor 802 of the endoscope system 800 comprises a plurality of image sensors 102A, 102B and 102C (each image sensor being a charged coupled device (CCD), complementary metal oxide semiconductor (CMOS) or Organic CMOS image sensor, for example). Each of the plurality of image sensors is operable to capture a respective one of the initial images of the scene. That is, the lens arrangement 104 operable to receive light from the scene and to form each initial image on a respective one of the image sensors 102A, 102B and 102C. Each image sensor is located at a different respective distance from the lens arrangement so that the in-focus position in the scene is different for each initial image. In the example of
Once each initial image has been captured, the image processor 900 determines an image sharpness level for corresponding portions of each of the captured initial images. The image processor 900 then determines the highest one of the determined image sharpness levels and generates, on the basis of image data from each of the captured initial images, the final image of the scene so that a portion of the final image of the scene corresponding to the corresponding portions of each of the captured initial images has a sharpness level equal to the determined highest image sharpness level.
Such an arrangement is exemplified with reference to
Each of the initial images and final image is divided into 9 portions as indicated by the grids 1001A, 1001B, 1001C and 1001 marked on each image. Corresponding portions of the images are those portions appearing at the same location in each image. Thus, for example, portion 1002A in image 1000A, portion 1002B in image 1000B, portion 1002C in image 1000C and portion 1002 in image 1000 are corresponding portions because each of them is the top right portion for its respective image. Similarly, portion 1006A in image 1000A, portion 1006B in image 1000B, portion 1006C in image 1000C and portion 1006 in image 1000 are corresponding portions because each of them is the bottom right portion for its respective image.
Due to the different in-focus positions of each initial image, a first object appearing to be in focus in a first portion of one of the initial images will be out of focus in the corresponding portions of the other initial images. On the other hand, a second object appearing to be out of focus in a second portion of the one of the initial images will be in focus in the corresponding portion of one of the other initial images. A final image in which both the first and second objects appear to be in focus can therefore be generated on the basis of the first portion of the initial image in which the first object appears to be in focus and on the basis of the second portion of the initial image in which the second object appears to be in focus. Using this principle, a final image in which all objects in the image (which appear to be in focus in one of the initial images) appear to be in focus may be generated by the image processor 900. It is noted that, in this description, an object which appears to be more in focus may be described as appearing sharper, where as an object which appears to be less in focus may be described as appearing less sharp.
This is exemplified in
In the example of
This determination of sharpness levels can be performed by partial image block (comprised of a block of pixels which form a sub-region of the image) or based on pixel by pixel basis. If the system applies pixel by pixel basis transportation, the quality of image is improved on a pixel by pixel basis. However, if the system applies determining sharpness level for each sub-region, calculation cost will be reduced compared to the pixel by pixel basis
In a first variant of the present technique, each of the image sensors captures light using primary colours, namely each of red, green and blue (RGB) colour channels (for example, using an RGB Bayer array). The image processor 900 then generates each portion of the final image of the scene on the basis of the one of the corresponding portions the initial images which has the highest image sharpness level. In one example, the measurement of the sharpness of each portion of each initial image occurs before demosaicing of the image. The demosaicing may then be carried out after the appropriate portions of the initial images have been combined in order to generate the final image.
In a second variant of the present technique, each of the image sensors captures light using a different respective one of red, green and blue (RGB) colour channels (so that, for example, image sensor 102A captures red light, image sensor 102B captures green light and image sensor 102C captures blue light). The image processor 900 then generates each portion of the final image of the scene by transporting the image sharpness level of the corresponding portion of the initial image with the highest measured image sharpness level to the corresponding portions of the other initial images, and combining the corresponding portion of the initial image which has the highest image sharpness level with the corresponding portions of the other initial images to which the measured highest image sharpness level has been transported.
This is advantageous because as the sharpness levels are different among RGB, this system can obtain the best sharpness values and provide this value to the other colour signals. This provides a higher quality resultant image. Also, this arrangement is easy to arrange using 3CMOS sensor systems.
As a further variant of the present technique, the system may have a plurality of image sensors (for example, more than one, two or even three). In addition to RGB colour, further sensor for infrared (IR) or Near-Infrared (NIR) wavelength may be arranged. This NIR wavelength band is useful for fluorescence biomedical imaging or imaging vessels with more perceivable colour.
It is noted that, with the above-mentioned variants, the determination of the sharpness level of each portion of the initial images may be determined using a suitable sharpness measurement technique, as is known in the art. Furthermore, the transport of a measured sharpness level of a portion of a first initial image to a corresponding portion of a second initial image may be carried out using a suitable sharpness transport technique, as is known in the art.
As shown in
The controller 902 may control the position of the image sensors to be adjusted on the basis of, for example, information received directly from the instrument (such as endoscope 114) attached to the optical device 800 via instrument interface 910 or from information received from a user of the optical device 800 via user interface 912. It is noted that the optical device 800 may have one or both of the instrument interface 910 and user interface 912.
This instrument interface arrangement is advantageous it this is used for an optical scope system, such as an endoscope system as the endoscope system is generally compatible for various types of scopes (different diameters, direct-view or oblique view endoscopes).
The instrument interface 910 may be any suitable interface via which the optical device 800 can receive information with the instrument (such as endoscope 114) to which it is attached. In one variant, the instrument (such as endoscope 114) attached to the endoscope system 800 has a corresponding instrument interface together with a suitable controller and storage unit for storing information which, when received by the controller 902, allows the controller 902 to determine the image sensor distances to be used with the instrument. Such an arrangement is exemplified in in
The user interface 912 may be any suitable interface for allowing a user to provide information to the optical device 800 for use by the controller 902 in determining the image sensor distances to be used with the instrument. For example, the user interface 912 may comprise one or more of a touch screen, keyboard, various control buttons, a gesture recognition system, a speech recognition system, or the like.
The information provided to the controller 902 via the instrument interface 910 or user interface 912 may be any suitable information which allows the controller 902 to determine the image sensor distances to be used with the instrument. For example, the information may comprise the image sensor distances D1, D2 and D3 themselves or may comprise one or more optical properties of the instrument so as to allow the controller 902 to calculate the image sensor distances D1, D2 and D3. Alternatively, the information may simply identify the particular instrument (for example, via a model number or serial number of the like), and the controller 902 may then look up the instrument in a suitable database in order to obtain the image sensor distances or one or more optical properties for calculating the image sensor distances associated with that instrument. The database may be stored in a storage unit 914 of the optical device 800 in advance. Alternatively, the database may be stored at a remote location on a network (such as the internet), which the controller 902 is able to access via network interface 916. The network interface 916 may be any suitable interface such as a Wi-Fi® or Ethernet interface, for example. The database, whether stored in the storage unit 914 or at a remote location may be updatable. This allows information to be added to the database for newly available instruments which are compatible with the endoscope system 800, thus allowing suitable image sensor distances to be determined for such newly available instruments. This allows the endoscope to be changed more quickly which is important in an industrial or medical setting.
Although the foregoing has described the optical device in the context of instruments (either medical or industrial, for example), the disclosure is not so limited. For example, the optical device may be used in other devices such as cameras (either still cameras or video cameras) or the like. Also the scope described here is not limited to be inserted into a body (i.e. endoscope), and can be used for microscope or exoscope or the other types of optical scope.
Various embodiments of the present disclosure are defined by the following numbered clauses:
1. An optical scope system (800) for capturing an image of a scene, the optical device comprising:
2. The optical scope system according to clause 1, wherein the image processor is operable to:
3. The optical scope system according to clause 2, wherein:
This is advantageous because as the sharpness levels are different among RGB, this system can obtain the best sharpness values and provide this value to the other colour signals. This provides a higher quality resultant image. Also, this arrangement is easy to arrange using 3CMOS sensor systems.
4. The optical scope system according to clause 2, wherein:
5. The optical scope system according to any preceding clause, wherein each of the plurality of image sensors is located along a different respective optical path with respect to the lens arrangement.
This allows that the system to capture multiple differently focused images simultaneously.
6. The optical scope system according to any preceding clause, wherein the lens arrangement is operable to receive light from the scene captured by an optical instrument and to form the initial image of the scene on each image sensor using the received light, wherein the distance of each respective image sensor from the lens arrangement is determined on the basis of one or more characteristics of the optical instrument.
7. The optical scope system according to clause 6, comprising a plurality of image sensor adjustment devices (901A-901C) each operable to adjust the distance of a respective one of the image sensors from the lens arrangement.
This allows the distances between sensors and object to be adjustable.
8. The optical scope system according to clause 7, comprising:
9. The optical scope system according to clause 7, comprising:
10. The optical scope system according to clause 8 or 9, wherein the information for determining the distance of each respective image sensor from the lens arrangement comprises (a) the distance of each respective image sensor from the lens arrangement, (b) one or more optical properties of the optical instrument on the basis of which the controller is configured to calculate the distance of each respective image sensor from the lens arrangement, or (c) an identifier of the optical instrument on the basis of which the controller is configured to consult a database in order to obtain the distance of each respective image sensor from the lens arrangement or the one or more optical properties of the optical instrument for calculating the distance of each respective image sensor from the lens arrangement.
11. The optical scope system according to any one of clauses 6 to 10, wherein the optical instrument is an endoscope.
12. The optical scope system according to clause 11, wherein the endoscope is a medical endoscope.
This is particularly advantageous because in the field of medical endoscopy, the requirement for high resolution images with a long depth of field is high.
13. An endoscopic system comprising an optical scope system according to any one of clauses 6 to 10 and an optical instrument configured to capture light from the scene for use by the lens arrangement of the optical device in forming an initial image of the scene on each image sensor of the optical device.
14. A method of capturing an image of a scene in an optical scope system, the method comprising:
15. A recording medium storing a computer program for controlling a computer to perform a method according to clause 14.
16. An optical scope system (800) for capturing an image of a scene, the optical device comprising:
means operable to generate the captured image of the scene on the basis of image data from one or more of the captured initial images.
Obviously, numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the disclosure may be practiced otherwise than as specifically described herein.
In so far as embodiments of the disclosure have been described as being implemented, at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine-readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like, is also considered to represent an embodiment of the present disclosure.
It will be appreciated that the above description for clarity has described embodiments with reference to different functional units, circuitry and/or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and/or processors may be used without detracting from the embodiments.
Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and/or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and/or processors.
Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in any manner suitable to implement the technique.
| Number | Date | Country | Kind |
|---|---|---|---|
| 16204872 | Dec 2016 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2017/082891 | 12/14/2017 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2018/109109 | 6/21/2018 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5908391 | Muzilla et al. | Jun 1999 | A |
| 6322496 | Iida et al. | Nov 2001 | B1 |
| 10324300 | Duckett | Jun 2019 | B2 |
| 10571679 | Duckett, III | Feb 2020 | B2 |
| 20040125228 | Dougherty | Jul 2004 | A1 |
| 20050143624 | Iddan | Jun 2005 | A1 |
| 20130038689 | McDowall | Feb 2013 | A1 |
| 20130041221 | McDowall et al. | Feb 2013 | A1 |
| 20130286176 | Westwick et al. | Oct 2013 | A1 |
| 20140031802 | Melsky et al. | Jan 2014 | A1 |
| 20160286187 | Takenaga | Sep 2016 | A1 |
| 20160309141 | Pulli | Oct 2016 | A1 |
| 20160360091 | Lindskog et al. | Dec 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 3072437 | Sep 2016 | EP |
| Entry |
|---|
| International Search Report dated Feb. 22, 2018 for PCT/EP2017/082891 filed on Dec. 14, 2017, 11 pages. |
| Guichard et al., “Extended Depth-of-Field using sharpness transport across Color Channels” vol. 7250, XP002676338, Jan. 1, 2009, 12 pages. |
| Hariharan, “Extending Depth of Field via Multifocus Fusion”, University of Tennessee, Dec. 2011, pp. 1-157. |
| Number | Date | Country | |
|---|---|---|---|
| 20190327421 A1 | Oct 2019 | US |
