The present invention relates generally to an endoscopic imaging system and, in particular, to image exposure adjustment of in vivo images.
Several in vivo measurement systems are known in the art. They include swallowed electronic capsules which collect data and which transmit the data to an external receiver system. These capsules, which are moved through the digestive system by the action of peristalsis, are used to measure pH (“Heidelberg” capsules), temperature (“CoreTemp” capsules) and pressure throughout the gastro-intestinal (GI) tract. They have also been used to measure gastric residence time, which is the time it takes for food to pass through the stomach and intestines. These capsules typically include a measuring system and a transmission system, wherein the measured data is transmitted at radio frequencies to a receiver system.
U.S. Pat. No. 5,604,531, assigned to the State of Israel, Ministry of Defense, Armament Development Authority, and incorporated herein by reference, teaches an in vivo measurement system, in particular an in vivo camera system, which is carried by a swallowed capsule. In addition to the camera system there is an optical system for imaging an area of the GI tract onto the imager and a transmitter for transmitting the video output of the camera system. The capsule is equipped with a number of LEDs (light emitting diodes) as the lighting source for the imaging system. The overall system, including a capsule that can pass through the entire digestive tract, operates as an autonomous video endoscope. It images even the difficult to reach areas of the small intestine.
U.S. patent application No. 2003/0023150 A1, assigned to Olympus Optical Co., LTD., and incorporated herein by reference, teaches a design of a swallowed capsule-type medical device which is advanced through the inside of the somatic cavities and lumens of human beings or animals for conducting examination, therapy, or treatment. Signals including images captured by the capsule-type medical device are transmitted to an external receiver and recorded on a recording unit. The images recorded are retrieved in a retrieving unit, displayed on the liquid crystal monitor and to be compared by an endoscopic examination crew with past endoscopic disease images that are stored in a disease image database.
One problem associated with the capsule imaging system is a non-uniform lighting over the imaging area due to the nature of this miniature device. Especially, when the capsule travels along a tube-like anatomical structure, the field of view of the camera system covers a section of the anatomical structure inner wall which is nearly parallel with the camera optical axis. Obviously, in this field of view, part of the anatomical structure inner wall away from the capsule receives less photon flux than that of the anatomical structure inner wall close to the capsule. The resultant is a non-uniform photon flux field. In return, part of the image produced by the camera image sensor is either under exposure or over exposure depends on how the camera is calibrated. Therefore, details of texture and color will be lost, which not only affects physicians' ability of abnormality diagnosis using these in vivo images, but also reduces the effectiveness of neighboring in vivo image stitching in applications such image mosiacing.
In general, in order to maximize the use of photon flux, the in vivo camera is calibrated such that there will be no over exposure in the captured images. Thus the non-uniform photon flux distribution results in under exposure in various areas of certain in vivo images. This under exposure of in vivo image is similar to the light falloff in regular photographic images.
U.S. patent application No. 2003/0007707 A1, assigned to Eastman Kodak Company, and incorporated herein by reference, teaches a method for compensating for light falloff caused by the non-uniform exposure which is produced by lenses at their focal plane when imaging a uniformly lit surface. For instance, the light from a uniformly gray wall perpendicular to the camera optical axis will pass through a lens and form an image that is brightest at the center and dims radially. When the lens is an ideal thin lens, the intensity of light in the image will form an intensity pattern described by cos4 of the angle between the optical axis of the lens and the point in the image plane. The visible effect of this phenomenon is referred to as falloff. The light compensating method taught in 0007707 describes a compensation function that relies on the value of the distance from a pixel location to the center of the image. Such a method is particularly useful for falloffs caused by lenses distortions. Invention 0007707 teaches a compensation equation:
Where dd is the distance in pixels from the (x,y) position to the center of the digital image and cvs is the number of code value per stop of exposure (cvs indicates scaling of the log exposure metric). The parameter f represents the focal length of a lens (in pixels) for which the falloff compensator will correct the falloff. This method is however less desirable for problems caused by non-uniform photon flux field when the endoscopic capsule traveling alone the GI tract, because regions with inadequate exposure do not have the geometric properties stated in the aforementioned equation.
Also the principal advantage of the invention described in 0007707 is that a falloff compensation may be applied to a digital image in such a manner that the balance of the compensated digital image is similar to that of the original digital image, which results in a much more pleasing effect that sometimes may causing problems such as blurring boundaries.
There is a need therefore for an improved endoscopic imaging system that overcomes the problems set forth above.
These and other aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from a review of the following detailed description of the preferred embodiments and appended claims, and by reference to the accompanying drawings.
The need is met according to the present invention by providing a digital image processing method for exposure adjustment of in vivo images that includes the steps of acquiring in vivo images; detecting any crease feature found in the in vivo images; preserving the detected crease feature; and adjusting exposure of the in vivo images with the detected crease feature preserved.
In the following description, various aspects of the present invention will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention. However, it will also be apparent to one skilled in the art that the present invention may be practiced without the specific details presented herein. Furthermore, well-known features may be omitted or simplified in order not to obscure the present invention.
During a typical examination of a body lumen, the in vivo camera system captures a large number of images. The images can be analyzed individually, or sequentially, as frames of a video sequence. An individual image or frame without context has limited value. Some contextual information is frequently available prior to or during the image collection process; other contextual information can be gathered or generated as the images are processed after data collection. Any contextual information will be referred to as metadata. Metadata is analogous to the image header data that accompanies many digital image files.
Referring to
An image packet 206 comprises two sections: the pixel data 208 of an image that has been captured by the in vivo camera system, and image specific metadata 210. The image specific metadata 210 can be further refined into image specific collection data 212, image specific physical data 214 and inferred image specific data 216. Image specific collection data 212 contains information such as the frame index number, frame capture rate, frame capture time, and frame exposure level. Image specific physical data 214 contains information such as the relative position of the capsule when the image was captured, the distance traveled from the position of initial image capture, the instantaneous velocity of the capsule, capsule orientation, and non-image sensed characteristics such as pH, pressure, temperature, and impedance. Inferred image specific data 216 includes location and description of detected abnormalities within the image, and any pathologies that have been identified. This data can be obtained either from a physician or by automated methods.
The general metadata 204 contains such information as the date of the examination, the patient identification, the name or identification of the referring physician, the purpose of the examination, suspected abnormalities and/or detection, and any information pertinent to the examination bundle 200. It can also include general image information such as image storage format (e.g., TIFF or JPEG), number of lines, and number of pixels per line.
Referring to
It will be understood and appreciated that the order and specific contents of the general metadata or image specific metadata may vary without changing the functionality of the examination bundle.
Referring now to
Data received in the in vitro computing device is examined for any sign of disease in a step of Abnormality detection 310. Details of the step of abnormality detection can be found in commonly assigned, co-pending U.S. patent application Ser. No. 10/679,711, entitled “Method And System For Real-Time Automatic Abnormality Detection For In Vivo Images” and filed on 6 Oct. 2003 in the names of Shoupu Chen, Lawrence A. Ray, Nathan D. Cahill and Marvin M. Goodgame, and which is incorporated herein by reference.
Note that unlike taking photographic images in natural scenes (indoor or outdoor), in vivo imaging takes place inside the GI tract which is a controlled environment and in general is an open space within the field of the view of the camera. A controlled environment means that there are no sources of lighting other than that from the LEDs of the capsule. An open space implies that there should be no occlusions that cause shadows (under exposure). Also, the reflectance should be the same locally along the GI tract inner wall in general, at least with the same order of magnitude. (This is not the case in real world where the reflectance of photographic objects could vary dramatically causing darker or brighter areas in the resultant images.) Thus, in an ideal case, an in vivo image should not present significant brightness differences in different areas. In reality, because of the uneven photon flux field generated by the limited lighting source, under exposure areas (low brightness areas) exist. Those low brightness areas need to be corrected to become brighter. While in photographic images of natural scenes (indoor or outdoor), low brightness areas could be a result of low reflection of a dark object surface which should not be corrected in an image.
The step of Image adjusting 309 is detailed in
There are variety methods could be used to lift the brightness of an under exposure area in image I (501). A preferred algorithm is described below.
Referring back to
statsA=F(I∩{overscore (I)}MA) (1)
where I∩{overscore (I)}MA is a logical AND operation, {overscore (I)}MA is the logical inverse of IMA, F(●) is a statistical analysis operation, and statsA (503) is a structure containing mean, median and other statistical quantities of the operand which is the result of the logical AND operation, I∩{overscore (I)}MA. The structure is a C language like data type and statsA (503) is defined as
where stats is the structure name and statsA.mean is the mean intensity of I∩{overscore (I)}MA, statsA.median is the median intensity of I∩{overscore (I)}MA, statsA.minimum is the minimal intensity of I∩{overscore (I)}MA and statsA.maximum is the maximal intensity of I∩{overscore (I)}MA.
Note that the logical AND operation, I∩{overscore (I)}MA, excludes under exposure pixels in the original image I (501) from the statistical analysis operation F(●). The purpose of this exclusion is to learn the statistics only in the normal exposure regions and the learned statistics will be used in a later procedure to lift the brightness level of under exposure regions so that the final image appears coherent.
Since the image adjustment operation is only applied to regions of under exposure (such as 704) caused by the non-uniform photon flux field, a second mask needs to be formed to exclude low brightness regions (such as 706) that belong to crease features. The second mask, mask B, is formed in a step of Forming mask B (504). The step of Forming mask B (504) is further detailed next.
A first operation of forming mask B (504) is a medial axis transformation that is applied to the threshold image IB (702) (see “Algorithm for image processing and computer vision”, by J. R. Parker, Wiley Computer Publishing, John Wiley & Sons, Inc., 1997). A medial axis transformation defines a unique compressed geometrical representation of an object. The medial axis transformation is also referred to as morphological skeletonization. The morphological skeletonization uses erosion and opening as basic operations. The result of the morphological skeletonization is a skeleton image. Denote the skeleton image by IS and its pixel by IS (m, n). Then IS (m, n)=S(IB (m, n)), where S is the medial axis transformation function. IS (m, n) (722), an exemplary result of applying the medial axis transformation to image IB (702), is shown in
Denote the second mask, mask B, by IMB and its pixel by IMB (m, n). First, initialize IMB by letting IMB (m, n)=IB (m, n)|∀m,∀n, where ∀m,∀n means all m and all n. Denote an eraser window 732 by W. Exemplary width and height of the eraser window W(732) are 3w, where w is the average width of lines 706. To determine if a one-valued pixel at location (m, n) of the image IMB belongs to crease features such as lines 706, center the eraser window W 732 at the location (m, n) 728 of IS (in operation, the window W is also centered at the location (m, n) 728 of IMB).
In general, there are various types of patterns of the geometry relationship between the window W(732) and the one-valued pixels that belong to crease features such as lines 706. Four exemplary representations of patterns are shown in
When pattern 802 occurs, pixel IMB (m, n) and its associated east-west neighboring one-valued pixels are erased. When pattern 804 occurs, pixel IMB (m, n) and its associated north-south neighboring one-valued pixels are erased. When pattern 806 occurs, pixel IMB (m, n) and its associated north west-south east neighboring one-valued pixels are erased. When pattern 808 occurs, pixel IMB (m, n) and its associated north east-south west neighboring one-valued pixels are erased.
The operation of erosion can be described by the following code:
Note that the above erosion operation produces an intermediate mask B image, IMB, 902 shown in
Those skilled in the art should understand that alternative erasing methods exist. For example, erasing operation can be implemented without performing medial axis transformation by checking more pixels.
Now referring to
A cluster may be eliminated if it contains too few one-valued pixels no matter it is a cluster of pixels of crease features or a cluster of pixels of an under exposure region. A cluster contains too few one-valued pixels suggests that the cluster does not have much influence on diagnosis. For example, if the number of pixels in a cluster is less than V, then this cluster is erased from IMB. Example V value could be 10. The above operations are done in a step of Mask property check 602. A query step 604 branches the process to stop 606 if there are no qualified clusters in mask B IMB, or to step 610 if there is at least on qualified cluster. An exemplary qualified mask B IMB 912 is shown in
Mask B IMB 912 is now ready to assist applying image adjustment to image I (501) in step 510. Image adjustment is further detailed by steps 610 and 612.
The exposure correction is accomplished in step 610. First, denote an image adjustment process by Φ(●). Denote an adjusted image by Iadj. The adjusted image by Iadj can be obtained by the following equation:
Iadj=(I∩{overscore (I)}MB)∪Φ(I∩IMB) (2)
where {overscore (I)}MB is the logical inverse of IMB, symbol ∪ is a logic OR operator, and symbol ∩ is a logic AND operator. The operation (I∩IMB) signifies that the adjustment process Φ(●) applies to pixels within region 704 in image I(501). On the other hand, the operation (I∩{overscore (I)}MB) signifies that the pixels outside the region 704 in image I(501) keep their original value in this stage.
An exemplary of a preferred algorithm of the present invention for the adjustment process Φ(●) is described below:
Note that in the above implementation, the adjustment coefficient cf is guaranteed to be greater than or equal to one since statsA=F(I∩{overscore (I)}MA) and (I∩{overscore (I)}MA) contains pixels having intensity greater than or equal to T (505), where T=mean(I)−K*std(I). On the other hand, statsB=F(I∩IMB) and (I∩IMB) contains pixels having intensity less than (505).
Notice also that statistics other than median could be used to compute the adjustment coefficient cf. and the adjustment could be applied to individual color channels, (R, G and B), independently. The adjustment operation, Ĩadj(m, n)=cfI(m, n), in this embodiment is a linear function. But other types of nonlinear functions such as log adjustment or LUT (look up table) also can be used.
Since the exposure correction is conducted only in areas such as 504 in image I (501), intensity discontinuity between the exposure corrected (adjustment) and uncorrected (non-adjustment) areas may exist along the boundary line such 1004 in
In
The cross boundary smoothing operation can be realized in one-dimensional space or two-dimensional space.
D is less than or equal to d. Exemplary value for D is 1, and 10 for d.
From the above code, it can be seen that the new x(0) is the moving average of pixels from both sides of the boundary line 1014. The influence of pixels from one side to the other side is propagated through newly updated x(i). Starting the process from x(0) helps the propagation of information across the boundary.
The operation described by the above discussion is assumed to be operated in an sRGB space (see Stokes, Anderson, Chandrasekar and Motta, “A Standard Default Color Space for the Internet—sRGB”, http://www.color.org/sRGB.html).
Images in sRGB have already been optimally rendered for video display, typically by applying a 3×3 color transformation matrix and then a gamma compensation lookup table. Any adjustment to the brightness, contrast, and gamma characteristics of an sRGB image will degrade the optimal rendering. If a digital image contained pixel values representative of a linear or logarithmic space with respect to the original scene exposures, the pixel values could be adjusted without degrading any subsequent rendering steps. For those skilled in the art, the ideas and algorithms of the present invention can be applied to spaces such as de-rendered logarithmic space.
It is well understood that the transmission of data over wireless links is more prone to requiring the retransmission of data packets than wired links. There is a myriad of reasons for this, a primary one in this situation is that the patient moves to a point in the environment where electromagnetic interference occurs. Consequently, it is preferable that all data from the Examination Bundle be transmitted to a local computer with a wired connection. This has additional benefits, such as the processing requirements for image analysis are easily met, and the primary role of the data collection device on the patient's belt is not burdened with image analysis. It is reasonable to consider the system to operate as a standard local area network (LAN). The device on the patient's belt 100 is one node on the LAN. The transmission from the device on the patient's belt 100 is initially transmitted to a local node on the LAN enabled to communicate with the portable patient device 100 and a wired communication network. The wireless communication protocol IEEE-802.11, or one of its successors, is implemented for this application. This is the standard wireless communications protocol and is the preferred one here. It is clear that the Examination Bundle is stored locally within the data collection device on the patient's belt, as well at a device in wireless contact with the device on the patient's belt. However, while this is preferred, it will be appreciated that this is not a requirement for the present invention, only a preferred operating situation. The second node on the LAN has fewer limitations than the first node, as it has a virtually unlimited source of power, and weight and physical dimensions are not as restrictive as on the first node. Consequently, it is preferable for the image analysis to be conducted on the second node of the LAN. Another advantage of the second node is that it provides a “back-up” of the image data in case some malfunction occurs during the examination. When this node detects a condition that requires the attention of trained personnel, then this node system transmits to a remote site where trained personnel are present, a description of the condition identified, the patient identification, identifiers for images in the Examination Bundle, and a sequence of pertinent Examination Bundlettes. The trained personnel can request additional images to be transmitted, or for the image stream to be aborted if the alarm is declared a false alarm. Details of requesting and obtaining additional images for further diagnosis can be found in commonly assigned, co-pending U.S. patent application Ser. No. (our docket 86570SHS), entitled “Method And System For Real-Time Remote Diagnosis Of In Vivo Images” and filed on 1 Mar. 2004 in the names of Shoupu Chen, Lawrence A. Ray, Nathan D. Cahill, and Marvin M. Goodgame, and which is incorporated herein by reference. To ensure diagnosis accuracy, images to be transmitted are those exposure adjusted in step 309.
The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.