Patent Application
20150045619
1. Field of the Invention
The present invention relates to a system and a method for mosaicing endoscope images, and more particularly to a system and a method for a real-time mosaicing of endoscope images using a wide angle view endoscope.
2. Description of the Prior Art
Minimally invasive surgery (MIS) is an indispensable component of the current surgical procedures. The small size of incision and the shorter duration of hospital stay make MIS very attractive to the patients. MIS has gone through various advancements and is widely applied to different areas of surgery such as abdomen, pelvis and paranasal sinuses. One of the limitations of the MIS technique is the narrow field of view through the endoscope which impedes the visual-spatial orientation of the surgeon. A broader field of view is needed in getting the correct spatial orientation of the visual display space.
Creating a panoramic image based on the captured endoscopic images is one of the solutions for increasing the field of view. Mosaicing is a method of creating a panoramic image by stitching many small images together. Several studies have reported the application of the mosaicing techniques to create a panoramic view of organs such as a bladder, retina, placenta and sinus in an invasive surgery. All of these systems use feature extraction and tracking of those features in the video frames to create a panoramic view.
However, currently no endoscope mosaicing system uses wide angle view endoscope device for dynamically extending the field of the endoscope view, i.e. near real-time mosaicing of the endoscope images.
To overcome the shortcomings of the endoscopes, the present invention provides a system and a method for a near real-time mosaicing of endoscope images using a wide angle view endoscope to mitigate or obviate the aforementioned problems.
The main objective of the present invention is to develop a system which would provide an extended dynamic field view during endoscopic procedures. The system uses an endoscope with a wide viewing angle to capture video images. The wide angle endoscope is used for the extended view of the scene and creating a mosaic view with such captured images.
The system for mosaicing endoscope images comprises the following components:
an endoscope having a body, an image channel extending from the body, a wide angle lens mounted to the tip of the image channel, and a camera in the body for capturing consecutive video images of a target area via the wide angle lens;
a computer connecting to and communicating with the endoscope for processing the captured consecutive video images received to correct the distortion due to the wide angle lens and to generate a mosaiced image; and
a monitor connecting to the computer for displaying the mosaiced image received from the computer.
In accordance with the present invention, the system uses the wide angle view endoscope to capture images. The wide angle view endoscope is able to capture a relative broader view of the scene compared to the conventional endoscopy. Therefore, the present system can create a larger view mosaic of the scene in near real-time.
Other objectives, advantages and novel features of the invention will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.
With reference to
The endoscope 10 has a body 11, an imaging channel 12 extending from the body 11, a wide angle lens 13 is mounted to a tip of the imaging channel 12. The endoscope 10 electronically connects to and communicates with the computer 20 via a wired communication such as a cable 21 or a wireless communication. In this embodiment, the viewing angle of the wide angle lens 13 is 30 degrees.
The body 11 contains a camera 14 or a connection to a camera and a light source or a connection to a light source. The camera 14 may comprise, but is not limited to monochrome imagers, single CCD cameras, multi-CCD cameras. The endoscope camera transmits captured images to the computer 20 via the cable 21.
The computer 20 has the hardware and software to correct the wide angle lens distortion and then create a mosaiced image of the captured video images from the camera. The mosaiced image is displayed on the monitor 30 connected to the computer via a signal cable 31.
With reference to
In the step of capturing video (S201), the Karl Storz 26003 BA product may be used in the invention as the endoscope 10 to capture consecutive video images with the resolution of 720×480 pixels and frame rate (frame frequency) of upto 30 per second. The captured images are acquired at a proper frame rate chosen so that the motion occurring between the images is sufficiently small for the large number of feature points between overlapped images. The captured consecutive video images are transmitted to and processed by the computer 20 as described in the steps S202-S206. Preferably, the captured consecutive video images will be converted to a digital format for processing. If the captured video images are color images, they will be converted to grayscale images in advance for reducing processing time.
In the step of the wide angle distortion correction (S202), a fully automatic Hough-based calibration method and calibration results are applied to achieve real-time distortion correction, and a graphics processing unit (GPU) to interpolate image in parallel is used. This method of the correction of the distortion due to wide angle camera was described in detail in the article “Wide-angle distortion correction by Hough transform and gradient estimation” published in the proceedings of Visual Communications and Image Processing (VCIP), 2011 IEEE.
For the method, a formula of wide-angle lens distortion and focal length was derived, and a relationship between image boundary and focal length was established. It can estimate focal length on-line and achieve real-time distortion correction.
In the step of detecting feature points on the captured video images (S203), a speeded-up robust feature (SURF) detection technique is applied for detecting feature points a1-a4, b1-b4, c1-c4 on the captured video images 110A, 110B as shown in
In the step of matching feature points between consecutive video images (S204), each detected feature point a1-a4, b1-b4, c1-c4 is assigned a feature descriptor, using binary string feature point descriptor called as BRIEF described in research paper by M. Calonder etal. published in “Brief: Binary robust independent elementary features” in the journal of Computer Vision ECCV 2010. In the BRIEF method, a patch (P) of size 48 is created and smoothed around each of the detected feature points a1-a4, b1-b4, c1-c4, and multiple pairs of pixels (x, y) inside each patch are selected randomly. A binary vector based upon comparison of the intensities (I) of the pixels “x” and “y” is created such that the BRIEF descriptor is 1 if I(x)>I(y) otherwise the BRIEF descriptor is 0. For matching the descriptor, a nearest neighbor search is done in the descriptor space using Hamming distance measurements. A threshold of 5 distances is used for matching.
In the step of tracking the video images (S205), using the matched feature points a global transformation, called homography, for images is estimated. The homography is described by an affine transformation with six degrees of freedom and is estimated using Random Sample Consensus (RANSAC) fitting model described by Li et al. in “Computing homography with RANSAC algorithm: a novel method of registration”, in the proceedings of SPIE. This is an iterative process in which three sets of points among the matched points are selected randomly and a homography is estimated using only those points which satisfied the threshold of displacement. A reference coordinate system is made by the first video frame F0, and the subsequent images Fi are transformed by the global homography matrix. The local transformation between consecutive images is used to derive the global transformation.
In the step of aligning images and generating mosaiced image (S206), all images are brought to a single reference frame and stitched together based upon the tracked feature points to form a mosaiced image, i.e. a panoramic view as shown in
Even though numerous characteristics and advantages of the present invention have been set forth in the foregoing description, together with details of the structure and features of the invention, the disclosure is illustrative only. Changes may be made in the details, especially in matters of shape, size, and arrangement of parts within the principles of the invention to the full extent indicated by the broad general meaning of the terms in which the appended claims are expressed.
