Method and Apparatus for Identifying Capsule Camera Location inside Gastrointestinal Tract

Abstract

A method and system for identifying a GI (gastrointestinal) part associated with a location of a capsule device within a human GI tract are disclosed. According to this method, one or more structured-light images captured by the capsule device when the capsule device is inside the human GI tract are received, where the structured-light images are captured by projecting a plurality of structured light beams from the capsule device onto a surrounding lumen wall. Distance information is derived based on the structured-light images for a target set of measuring rays from the capsule device to the surrounding lumen wall. The GI part associated with the location of the capsule device associated with a distance-measuring frame related to said one or more structured-light images is determined based on the distance information, where the GI part belongs to a group comprising a transverse colon.

Description

FIELD OF THE INVENTION

The present invention relates to imaging the gastrointestinal (GI) tract and processing the images captured using a capsule camera. In particular, the present invention is focused on determining, based on structured-light images captured by the capsule camera, the GI anatomical part associated with the images captured by the capsule camera.

BACKGROUND AND RELATED ART

Devices for imaging body cavities or passages in vivo are known in the art and include endoscopes and autonomous encapsulated cameras. Endoscopes are flexible or rigid tubes that pass into the body through an orifice or surgical opening, typically into the esophagus via the mouth or into the colon via the rectum. An image is formed at the distal end using a lens and transmitted to the proximal end, outside the body, either by a lens-relay system or by a coherent fiber-optic bundle. A conceptually similar instrument might record an image electronically at the distal end, for example using a CCD or CMOS array, and transfer the image data as an electrical signal to the proximal end through a cable. Endoscopes allow a physician control over the field of view and are well-accepted diagnostic tools. However, they do have a number of limitations, present risks to the patient, are invasive and uncomfortable for the patient, and their cost restricts their application as routine health-screening tools.

Because of the difficulty traversing a convoluted passage, endoscopes cannot easily reach the majority of the small intestine and special techniques and precautions, that add cost, are required to reach the entirety of the colon. Endoscopic risks include the possible perforation of the bodily organs traversed and complications arising from anesthesia. Moreover, a trade-off must be made between patient pain during the procedure and the health risks and post-procedural down time associated with anesthesia.

An alternative in vivo image sensor that addresses many of these problems is the capsule endoscope. A camera is housed in a swallowable capsule, along with a radio transmitter for transmitting data, primarily comprising images recorded by the digital camera, to a base-station receiver or transceiver and data recorder outside the body. The capsule may also include a radio receiver for receiving instructions or other data from a base-station transmitter. Instead of radio-frequency transmission, lower-frequency electromagnetic signals may be used. Power may be supplied inductively from an external inductor to an internal inductor within the capsule or from a battery within the capsule.

An autonomous capsule camera system with on-board data storage was disclosed in the U.S. Pat. No. 7,983,458, entitled “In Vivo Autonomous Camera with On-Board Data Storage or Digital Wireless Transmission in Regulatory Approved Band,” granted on Jul. 19, 2011. This patent describes a capsule system using on-board storage such as semiconductor nonvolatile archival memory to store captured images. After the capsule passes from the body, it is retrieved. Capsule housing is opened and the images stored are transferred to a computer workstation for storage and analysis. For capsule images either received through wireless transmission or retrieved from on-board storage, the images will have to be displayed and examined by diagnostician to identify potential anomalies.

FIG. 1 illustrates an exemplary capsule system with on-board storage. The capsule system 110 includes illuminating system 12 and a camera that includes optical system 14 and image sensor 16. A semiconductor nonvolatile archival memory 20 may be provided to allow the images to be stored and later retrieved at a docking station outside the body, after the capsule is recovered. System 110 includes battery power supply 24 and an output port 26. Capsule system 110 may be propelled through the GI tract by peristalsis.

Illuminating system 12 may be implemented by LEDs. In FIG. 1, the LEDs are located adjacent to the camera's aperture, although other configurations are possible. The light source may also be provided, for example, behind the aperture. Other light sources, such as laser diodes, may also be used. Alternatively, white light sources or a combination of two or more narrow-wavelength-band sources may also be used. White LEDs are available that may include a blue LED or a violet LED, along with phosphorescent materials that are excited by the LED light to emit light at longer wavelengths. The portion of capsule housing 10 that allows light to pass through may be made from bio-compatible glass or polymer.

Optical system 14, which may include multiple refractive, diffractive, or reflective lens elements, provides an image of the lumen walls on image sensor 16. Image sensor 16 may be provided by charged-coupled devices (CCD) or complementary metal-oxide-semiconductor (CMOS) type devices that convert the received light intensities into corresponding electrical signals. Image sensor 16 may have a monochromatic response or include a color filter array such that a color image may be captured (e.g. using the RGB or CYM representations). The analog signals from image sensor 16 are preferably converted into digital form to allow processing in digital form. Such conversion may be accomplished using an analog-to-digital (A/D) converter, which may be provided inside the sensor (as in the current case), or in another portion inside capsule housing 10. The A/D unit may be provided between image sensor 16 and the rest of the system. LEDs in illuminating system 12 are synchronized with the operations of image sensor 16. Processing module 22 may be used to provide processing required for the system such as image processing and video compression. The processing module may also provide needed system control such as to control the LEDs during image capture operation. The processing module may also be responsible for other functions such as managing image capture and coordinating image retrieval.

The capsule camera in FIG. 1 corresponds to an end-facing capsule, where the camera is located at the proximity of one capsule end and has a field of view along the longitudinal direction of the capsule. A capsule camera with a panoramic view has been disclosed in U.S. Pat. No. 7,817,354 issued on Oct. 19, 2010, where the capsule has multiple cameras facing in directions perpendicular to the longitudinal direction of the capsule. Panoramic imaging as described in U.S. Pat. No. 7,817,354 has the benefit of viewing mucosa squarely rather than mostly looking in lumen tunnel view as in end-facing cameras.

In a colonoscopy procedure, the lumen is insufflated and the view are most through air as the media. The appearance of the same pathology (e.g., a polyp) may be different from the one captured by a capsule camera since the colon is not insufflated by air. Instead, the camera may look at the polyp through liquid. Often there is a need to match the polyp from the capsule endoscopy with the polyp seen in the colonoscopy. The capsule endoscopy is a convenient alternative to the colonoscopy. However, if any anomaly (e.g., a polyp) is found during the imaging procedure by the capsule endoscopy, the anomaly has to be treated by a subsequent colonoscopy procedure. Accordingly, the location information of the polyp becomes helpful to match the polyp from the capsule endoscopy with the polyp seen in the colonoscopy. For example, if a polyp seen in the capsule endoscopy and a polyp seen in the colonoscopy are found to be from different anatomical parts of the colon (e.g., one from transverse colon and the other from the descending colon), these two polyps are apparently not the same one. Accordingly, a method to determine the anatomical parts of the colon associated with captured images are disclosed in the present invention.

BRIEF SUMMARY OF THE INVENTION

A method and system for identifying a GI (gastrointestinal) anatomical part associated with a location of a capsule device within a human GI tract are disclosed. According to this method, one or more structured-light images captured by the capsule device when the capsule device is inside the human GI tract are received, where the structured-light images are captured by projecting a plurality of structured light beams from the capsule device onto a surrounding lumen wall. Distance information is derived based on the structured-light images for a target set of measuring rays from the capsule device to the surrounding lumen wall. Information regarding the GI anatomical part associated with the location of the capsule device associated with a distance-measuring frame related to said one or more structured-light images is provided based on the distance information, where the GI anatomical part belongs to a group comprising a transverse colon.

In one embodiment, the target set of measuring rays are on a plane perpendicular to a longitudinal axis of the capsule device and each of the measuring rays ends at an intersection of the plane and an interior point of the surrounding lumen wall. The present invention may further comprise a step of displaying visual representation of ending points of the measuring rays in a Cartesian coordinate on a display device.

In one embodiment, the present invention may further comprise a step of providing information indicative of a shape of the surrounding lumen wall based on the distance information, where the location of the capsule device is indicated according to whether the shape of the surrounding lumen wall is closer to a circle or a triangle. In one embodiment, curve fitting is used to determine whether the shape of the surrounding lumen wall resembles the triangle.

In one embodiment, the method further comprises displaying a plot on a display device for a user to visualize, where the plot is representative of the shape of the surrounding lumen wall in one axis and a frame number or a frame time in another axis. The GI anatomical part associated with the location of the capsule device is indicated according to the shape of the surrounding lumen wall, where if the shape resembles the triangle, the GI anatomical part associated with the location of the capsule device is the transverse colon.

In one embodiment, the method further comprises displaying a plot on a display device for a user to visualize, where the plot is representative of a shape index of the surrounding lumen wall in one axis and a frame number or a frame time in another axis, and wherein the shape index is derived based on frequency-domain data of the distance information represented in a polar coordinate. The GI anatomical part associated with the location of the capsule device is indicated according to the shape index and if the shape index indicates the triangle, the GI anatomical part associated with the location of the capsule device is the transverse colon.

In one embodiment, the present invention may further comprise a step of determining a size of the surrounding lumen wall based on the distance information. A plot can be displayed on a display device, where the plot is representative of the size of the surrounding lumen wall in one axis and the frame number or the frame time in another axis, where the location of the capsule device is indicated according to the size of the surrounding lumen wall. A section of the human tract corresponds to an ascending colon if the section corresponds to a first part of the human colon and the size is reduced or the size is gradually reducing. In another embodiment, the size of the surrounding lumen wall is indicated based on the distance information associated with multiple distance-measuring frames close to each other in a temporal order. The GI anatomical part associated with the location of the capsule device is determined based on the distance information associated with the multiple distance-measuring frames, and a decision of the GI anatomical part associated with the location of the capsule device is confirmed when a same decision out of multiple consecutive decisions is reached. In yet another embodiment, the plot representative of the size of the surrounding lumen wall corresponds an average or filtered size of the surrounding lumen wall associated with multiple measuring frames

In one embodiment, the present invention may further comprise a step of transforming the distance information represented in a polar coordinate into discrete frequency-domain information, where the GI anatomical part associated with the location of the capsule device is indicated by the discrete frequency-domain information. For example, Discrete Fourier Transform (DFT) or Fast Fourier Transform (FFT) can be used to transform the distance information into the discrete frequency-domain information. For example, if all frequency terms are insignificant except for zero-th frequency term, the GI anatomical part associated with the location of the capsule device corresponds to ascending or descending colon. In another embodiment, if the discrete frequency-domain information has a maximum at zero-th frequency term of the discrete frequency-domain information and a second largest term at third frequency term of the discrete frequency-domain information, the GI anatomical part associated with the location of the capsule device corresponds to the transverse colon. In one embodiment, the present invention may further comprise a step of determining a magnitude ratio of third frequency term and zero-th frequency term of the discrete frequency-domain information, where if the magnitude ratio is larger than a threshold, the GI anatomical part associated with the location of the capsule device corresponds to the transverse colon and, otherwise the GI anatomical part associated with the location of the capsule device corresponds to the ascending/descending colon. In one example, the threshold includes a range between 0.13 and 0.27.

In one embodiment, the present invention may further comprise applying correction to the distance information prior to transforming the distance information into the discrete frequency-domain information. For example, coordinate translation can be applied to the distance information to correct eccentricity. In another example, coordinate rotation can be applied to the distance information to correct tilting. In yet another example, coordinate rotation followed by coordinate translation are applied to the distance information to correct tilting and eccentricity respectively. In yet another example, coordinate translation followed by coordinate rotation are applied to the distance information to correct eccentricity and tilting respectively.

In one embodiment, the present invention may further comprise a step of receiving a regular image capsuled by a camera in the capsule device for a scene in a field view of the camera and determining an anomaly in the regular image, where the regular image is temporally close to said one or more structured-light images and the GI anatomical part associated with the location of the anomaly within the human GI tract is determined based on the distance information.

In one embodiment, the present invention may further comprise a step of determining fold depth information of the surrounding lumen wall based on the distance information. Furthermore, a plot can be displayed on a display device for a user to visualize, where the plot is representative of the fold depth information of the surrounding lumen wall in one axis and a frame number or a frame time in another axis. the GI anatomical part associated with the location of the anomaly, within the human GI tract is indicated based on the fold depth information.

In another embodiment, the method further comprises displaying a plot on a display device for a user to visualize, wherein the plot is representative of the discrete frequency-domain information in one axis and a frame number or a frame time in another axis. For example, the discrete frequency-domain information corresponds to a ratio of a magnitude of third frequency term to a magnitude of zero-th frequency term.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary capsule system with on-board storage, where the capsule system includes illuminating system and a camera that includes optical system and image sensor.

FIG. 2 illustrates an example of structured lights for measuring distances from a capsule camera to surrounding lumen walls.

FIG. 3 illustrates an example of structured lights for measuring distances, where the flat measuring cone plane passing through the capsule.

FIG. 4 illustrates another example of structured lights for measuring distances, where the measuring cone is at a forward location of the capsule.

FIG. 5A illustrates an example of the distances of the target set of rays for the case of no eccentricity and no tilting.

FIG. 5B represents a plot of the measured distances versus the polar angle, i.e., distances in the polar coordinate.

FIG. 5C illustrates an example of the distances of the target set of rays for the case of eccentricity and no tilting.

FIG. 5D represents a plot of the measured distances versus the polar angle, i.e., distances in the polar coordinate.

FIG. 6A illustrates an example of distances from the capsule to the lumen wall for the transverse colon with ideal triangular wrinkles or folds the capsule centered and without tilting.

FIG. 6B illustrates the distances in the polar coordinate for the example of FIG. 6A.

FIG. 6C illustrates another example of distances from the capsule to the lumen wall for the transverse colon with ideal triangular wrinkles or folds and with the capsule centered and without tilting.

FIG. 6D illustrates the distances in the polar coordinate for the example of FIG. 6B.

FIG. 6E illustrates an example of distances from the capsule to the lumen wall for the transverse colon similar to FIG. 6A, but with eccentricity.

FIG. 6F illustrates the distances in the polar coordinate for the example of FIG. 6E.

FIGS. 7A-B illustrate examples of measured lumen size profile for the ascending colon during a course of imaging the colon by a capsule device.

FIGS. 8A-E illustrate the amplitudes of the lower frequency terms of the frequency response for the centered circle case of FIG. 5A, the circle with eccentricity case of FIG. 5C, the centered triangle case of FIG. 6A, the centered triangle with sharper corners case of FIG. 6C, and the triangle with eccentricity case of FIG. 6E respectively.

FIG. 9A illustrates an example of distances from the capsule to the circular lumen wall with tilting, where the capsule is tilted by about 60° from the longitudinal direction of the GI tract.

FIG. 9B shows an example of tilting with eccentricity, where the capsule is tilted by about 60° from the longitudinal direction of the GI tract and 23 mm off center.

FIG. 9C illustrates the distances in the polar coordinate for FIG. 9A.

FIG. 9D illustrates the distances in the polar coordinate for FIG. 9B.

FIG. 9E illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 9A.

FIG. 9F illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 9B.

FIG. 10A illustrates an example of distances from the capsule to the triangular lumen wall with tilting, where the capsule is tilted by about 60° from the longitudinal direction of the GI tract.

FIG. 10B shows an example of tilting with eccentricity, where the capsule is tilted by about 60° from the longitudinal direction of the GI tract and 19 mm off center.

FIG. 10C illustrates the distances in the polar coordinate for FIG. 10A.

FIG. 10D illustrates the distances in the polar coordinate for FIG. 10B.

FIG. 10E illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 10A.

FIG. 10F illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 10B.

FIG. 11A illustrates an example of distances from the capsule to the lumen wall of the transverse colon in an in-vitro environment, where the capsule is off center without tilting.

FIG. 11B illustrates the corresponding distances in the polar coordinate for FIG. 11A.

FIG. 11C illustrates the distances for FIG. 11A after eccentricity correction according to an embodiment of the present invention.

FIG. 11D illustrates the distances in the polar coordinate for FIG. 11C.

FIG. 11E illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 11A.

FIG. 11F illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 11C.

FIGS. 12A-B illustrate examples of shape index of measured lumen shape profile for the colon during a course of imaging the colon by a capsule device.

FIG. 13 illustrates an exemplary flowchart for identifying a GI (gastrointestinal) anatomical part associated with a location of a capsule device within a human GI tract according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

It will be readily understood that the components of the present invention, as generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the systems and methods of the present invention, as represented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. References throughout this specification to “one embodiment,” “an embodiment,” or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, etc. In other instances, well-known structures, or operations are not shown or described in detail to avoid obscuring aspects of the invention. The illustrated embodiments of the invention will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of apparatus and methods that are consistent with the invention as claimed herein.

Endoscopes are normally inserted into the human body through a natural opening such as the mouth or anus. Therefore, endoscopes are preferred to be small sizes so as to be minimally invasive. As mentioned before, endoscopes can be used for diagnosis of human gastrointestinal (GI) tract. The captured image sequence can be viewed to identify any possible anomaly. If any anomaly is found, it is of interest to identify the location of the polyp, which is useful for subsequent treatment such as removal of the polyp. Accordingly, the present invention discloses an endoscope incorporating a means for determining the location of the capsule when a target image is captured, in particular, when the target image is determined to contain a polyp.

There are various known ways to determine the location of a capsule. For example, in U.S. Pat. No. 10,506,921 issued on Dec. 17, 2019, a method is disclosed to estimate the travelled distance measuring by a capsule camera in the gastrointestinal tract. Accordingly, the location of the capsule can be determined in terms of travelled distance inside the GI tract. In U.S. Pat. No. 10,354,382 issued on Jul. 16, 2019, a method is disclosed to use features, such as the geometric shape of the colon in the captured images to recognize the segment of the colon. For example, the lumen of the transverse colon has a particularly triangular shape. An image of the colon's geometric shape can be compared with a database of images the colon's geometric shape to determine which segment of the colon the imaging device is in. In U.S. Pat. No. 10,102,334 issued on Jul. 16, 2019, a method is disclosed to use pattern recognition based on the wrinkle patterns or wall structure of the colon to automatically detect whether the wrinkles of the tissue walls correspond to the transverse colon or ascending/descending colon.

Also, certain kinds of localization devices, such as an accelerator, gyrator, etc., have been used to trace the ingestible device in the GI tract when the ingestible capsule containing these localization devices passes through the GI tract. However, the subject usually does not remain stationary after swallowing the ingestible capsule. When the subject moves, these localization devices cannot reliably differentiate the movement of torso or the capsule in the GI tract.

In the present invention, a method to differentiate the anatomical parts of the GI tract based on structured-light images is disclosed. The capsule camera in the present invention incorporates light sources and micro-lens array to project structured light onto the surrounding lumen walls. The structured-light images are captured and used to derived depth or shape information of the corresponding regular images captured by the capsule camera. Methods and apparatus for capturing regular images and structured light images using a single image sensor are disclosed in U.S. Pat. No. 9,936,151, issued on Apr. 3, 2018. In U.S. Pat. No. 11,019,327, issued on May 25, 2021 discloses details of the structured light source and the micro-lens array. According to U.S. Pat. No. 9,936,151, the structured-light images can be used to derive depth or shape information for the corresponding regular images.

In one embodiment, structured light beams are projected onto the lumen walls and distances for the measuring rays from the capsule to the lumen wall are derived from the structured light, where a set of measuring rays from the capsule to the lumen wall are used to determine the distances from the capsule to a perimeter of the GI tract. The ending points of the measuring rays are located on the perimeter. The perimeter of the GI tract is preferred to be aligned with the field of view for regular images since the distance information will be used by the corresponding regular images. When a pathology feature, such as a polyp, is found in a regular picture, the corresponding distance may be used to determine where the polyp is located in the GI tract. The set of measuring rays may have a cone shape with the tip of cone at the capsule. In an extreme case, the cone may be fully open to become flat. In this case, the perimeter corresponds to the intersection of the GI tract and a plane perpendicular to the longitudinal axis of the capsule. The cone for the set of measuring rays may be in a forward or backward direction of the capsule.

For the purpose of illustration, we now assume the structure of the lumen wall can be modelled by a target geometry shape such as a cylinder or a round triangular prism. If a section of the GI tract is model as a cylinder, then a cross section of the lumen wall perpendicular to the longitudinal direction becomes a circle. FIG. 2 illustrates an example, where a capsule 220 projects structured light beams (e.g., 240, 242, 244 and 246) onto the lumen wall of a section of GI tract 210. The capsule is assumed to be located at the center of the lumen wall with the longitudinal axis of the capsule (shown as a dashed line 230) aligned with the longitudinal direction of the GI tract 230′ (shown as a dot-dash line 230′). Various patterns of the structured light beams can be used. For imaging the GI tract, a pattern comprising layers of circular dots is disclosed in U.S. Pat. No. 11,019,327. Nevertheless, the present invention is not limited by any specific pattern of structured light beams. Any beam pattern that can be used to derive the distances for a target set of rays from the capsule to the intersection of the lumen wall and a measuring cone, where the measuring cone can be perpendicular to the longitudinal direction of the GI tract. When the GI tract is modelled as a cylinder, the intersection of the lumen wall and the measuring cone becomes a circle.

FIG. 3 illustrates an exemplary target set of measuring rays (341, 343, 345, etc.) associated with a flat measuring cone passing through the capsule 320. The intersection between the flat measuring cone and the GI wall 310 is a circle 340. It is understood that the flat measuring cone is perpendicular to the longitudinal axis 330 of the capsule (also the GI tract) and contains the intersection circle 340. The target set of rays (341, 343, 345, etc.) and the intersection points (342, 344, 346, etc.) of the rays and the GI wall are shown in FIG. 3. The distances of the rays in this case have a constant value (i.e., radius R1). FIG. 4 illustrates another example of the measuring cone, where the measuring cone is located in a forward position to the capsule. The intersection between the measuring cone and the GI wall 310 is still a circle 350. The target set of rays (441, 443, 445, etc.) and the intersection points (442, 444, 446, etc.) of the rays and the GI wall are shown in FIG. 4. The distances of the rays in this case have a constant value (i.e., radius R2), which is larger than the radius (i.e., R1) for the example in FIG. 3. For a panoramic capsule camera, the images captured by the camera will be centered around a plane perpendicular to the longitudinal axis of the camera when the longitudinal direction of the capsule is aligned with the longitudinal direction of the GI tract. Therefore, the flat measuring cone through the capsule is preferred for the panoramic camera. For an end-facing capsule, the measuring cone toward the imaging end of the capsule is preferred.

As shown in FIG. 2, the structured-light beams often cover similar field of view as the regular images. Therefore, the structured-light image may cover a large region of lumen wall around the capsule device. On the other hand, the distance information for measuring the size/shape of the lumen wall may be confined to the perimeter as shown in FIG. 3 and FIG. 4. The target set of rays for measuring the size/shape of the lumen wall may not be at the exact same locations of the structured light beams. However, as is known in the art, the distances of the target set of rays can be derived from the structured light beams.

FIG. 5A illustrates an example of the distances of the target set of rays (530, 532, 534, etc.) for the case of no eccentricity (i.e., capsule 520 at the center of the GI tract 510) and no tilting (i.e., the longitudinal axis of the capsule aligned with the longitudinal direction of the GI track). The “eccentricity” is defined as the eccentricity of capsule center relative to the GI tract center. In FIG. 5A, the distances information as represented by the measuring rays from the capsule to the lumen wall are displayed in an x-y coordinate (i.e., in a Cartesian coordinate) with the center of the shape as the origin, which renders a useful visual presentation on a display device for a user (e.g., a medical professional) to visualize the size/shape of the lumen wall. In particular, the ending points of the measuring rays provide a good visual representation of the contour of the lumen wall for the size and/or shape of the lumen wall. FIG. 5B represents a plot of the measured distances versus the polar angle. In this case, all distances are the same and the plot becomes a flat line. FIG. 5C illustrates an example of the distances of the target set of rays (540, 542, 544, etc.) in the x-y coordinate with the center of the shape as the origin for the case of eccentricity (i.e., capsule 520 off the center of the GI tract 510 ) and no tilting (i.e., the longitudinal axis of the capsule aligned (i.e., parallel) with the longitudinal direction of the GI track). The “tilting” is defined as the tilting of capsule longitudinal axis relative to GI tract centerline. FIG. 5D represents a plot of the measured distances versus the polar angle. In this case, the distance varies along the polar angle with a maximum at 0° (or 360°) and a minimum at 180°.

In U.S. Pat. No. 9,936,151, an endoscope device (tethered or capsule) is disclosed, which is capable of capturing regular images as well as structured-light images. The structured-light images are captured along with the regular images in an interleaved fashion so that each regular image is always accompanied by one or more close-by (in the temporal order) structured-light images. Therefore, the depth data extracted from the close-by structured-light images can be used for the corresponding regular image. For example, if a polyp is found in a regular image, the size information of the polyp can be estimated from the corresponding close-by structured-light images. For a capsule device, when polyp is found, the polyp, has to be removed in a subsequent colonoscopy procedure. Therefore, the location information, in terms of which part of the colon, such as ascending colon, transverse or descending colon, regarding the polyp becomes useful.

As mentioned earlier, the cylinder model fits the ascending and descending colons well. However, for the transverse colon, a triangular model fits better. As disclosed in various literatures related to the colon, where the wrinkles of the tissue walls in the transverse colon show a triangular structure. Therefore, the transverse colon can be modelled as triangular wrinkles or folds inside the GI tract. An intersection of the interior of the lumen wall and a flat measuring cone perpendicular to the longitudinal direction should have a triangular shape with round corners. FIG. 6A illustrates an example of distances from the capsule to the lumen wall for the transverse colon with ideal triangular wrinkles or folds, the capsule centered and without tilting, where triangle 610 corresponds to the intersection of the interior of the lumen wall and a flat measuring cone perpendicular to the longitudinal direction and capsule 620 is located at the center of the GI tract. The distance versus polar angle for the example of FIG. 6A is shown in FIG. 6B. As shown in FIG. 6B, the distance oscillates up and down for three cycles from 0° to 360° . FIG. 6C illustrates another example of distances from the capsule to the lumen wall for the transverse colon with ideal triangular wrinkles or folds and with the capsule centered and without tilting, where triangle 630 corresponds to the intersection of the interior of the lumen wall and a flat measuring cone perpendicular to the longitudinal direction and capsule 640 is located at the center of the GI tract. In this example, the model for the transverse colon has sharper corners. The distance versus polar angle for the example of FIG. 6C is shown in FIG. 6D. As shown in FIG. 6D, the distance oscillates up and down for three cycles from 0° to 360°. FIG. 6E illustrates an example of distances from the capsule to the lumen wall for the transverse colon similar to FIG. 6A. However, in this case, the capsule is off the center of the GI tract and however, the longitudinal axis of the capsule is still aligned (i.e., parallel) with the longitudinal direction of the GI tract. The distance versus polar angle for the example of FIG. 6E is shown in FIG. 6F.

From the planar (i.e., two-dimensional) contour plots (e.g. FIGS. 5A, 5C, 6A, 6C and 6E) of the lumen wall based on the derived distances from the capsule to the lumen wall, the shape of the lumen wall (in particular, the interior wall) can be identified visually. Therefore, depending on whether the shape looks triangular or round, we can differentiate whether the capsule is in the transverse colon or the ascending/descending colon. The decision for the capsule location within the GI tract is made based on the distance information, which is derived from the structured-light images. Therefore, the capsule location determined is temporally associated with the structured-light image (referred as a frame). The distance information may also be derived from multiple structured-light images. In this case, the time associated with the capsule location can be determined based on the frame times. For example, if the distance information is derived from two frames of the structured-light images, the time associated with the capsule location can be the average of the two frame times. Alternatively, it may also designate any of the two frame times as the time associated with the capsule location. If the distance information is derived from three structured-light image frames, the time associated with the capsule location can be the center of the three frame time. In general, the center of the multiple frame times can be designated as the time associated with the capsule location. Furthermore, we may track the shape/size transition to determine when the capsule enters the transverse colon from the ascending colon or when the capsule enters the descending colon from the transverse colon.

Since there are variations among different subjects and the colon is not a uniform tract, this dependence on the size could be based on the frame number or the time along one axis and the size in another axis. Consequently, according to another embodiment of the present invention, the indication of anatomical parts of the colon associated with the measuring frames is based on the size or the trend of the size along the frame number axis or the time axis. Alternatively, it can also be based on the shape or the trend of shape or based on the distance information. For example, the shape resembles a triangular type is associated with the transverse colon and round; or otherwise with the ascending, descending, sigmoid colon or rectum. The size or the trend of the size along the frame number axis or the time axis can be displayed on a display device for a user (e.g., a medical professional) to visualize. The user can determine the anatomical parts of the colon associated with the measuring frames based on the displayed information.

We have observed that the beginning section (closer to the cecum) of the ascending colon is usually larger than the distal section (i.e., closer to transverse colon) in size. Therefore, the distance between the capsule and the mucosal surface (i.e., the lumen wall) tends to be larger for the beginning section of the ascending colon. As mentioned earlier, the size of the lumen can be determined from the distance between the capsule and the lumen wall. Therefore, we can generate a drawing corresponding to the distance information or lumen size information versus elapse time or travel time, where the elapse time or travel time is associated with the time when the capsule travelling through the colon. The elapse time or capsule travel time can also be replaced by frame number since the frames of structured-light images are captured sequentially when the capsule travels through the GI tract. FIGS. 7A-B illustrate two examples of the lumen size information versus the capsule travel time in the GI tract. As shown in FIGS. 7A-B, the size profiles for two different subjects appear to be quite different. However, the lumen size profile shows a trend of reducing size along time, which corresponds to the capsule device moves from ascending colon to distal part of colon. The trend of the lumen size along the frame number axis or the time axis can be displayed on a display device for a user (e.g., a medical professional) to visualize. The user can determine the anatomical parts of the colon associated with the measuring frames based on the displayed information.

The colon fold in the ascending colon is also larger and deeper than the distal colon. By using 3D information, we can also discern the fold depth and accordingly discern whether it is in the ascending colon or not based on the fold depth or its trend along the time or frame axis. The depth of a particular fold can be estimated by the difference of the distance to the edge of the fold and the distance to the base, i.e. the surrounding mucosa next to the fold. The fold depth or the trend of the fold depth along the frame number axis or the time axis can be displayed on a display device for a user (e.g., a medical professional) to visualize. The user can determine the anatomical parts of the colon associated with the measuring frames based on the displayed information.

While the task of identifying which part of the colon that the capsule is located can be performed by a human subject, the task can also be automated, or at least partially automated, using a computing device such as a personal computer, a notebook, a workstation, a mobile device such as smart phone, or an embedded processor. As mentioned earlier, the shape of the colon fits a circular model or a triangular model depending on whether the capsule camera is located in the ascending/descending colon or the transverse colon. Therefore, we can fit the distance ending dots corresponding to the intersection of the lumen wall and a measuring cone with the two candidate shapes, i.e., a circle and a triangle. If it fits the triangle model better, the corresponding capsule location is determined to be the transverse colon. Otherwise, the corresponding capsule location is determined to be in the ascending/descending colon. Furthermore, if the observed shape changes from a circle to a triangle, this implies that the capsule is entering the transverse colon from the ascending colon. If the observed shape changes from a triangle to a circle, this implies that the capsule is entering the descending colon from the transverse colon.

In a real-world situation, the lumen wall is neither a perfect triangular shape nor a perfect circular shape. In order to develop a reliable method to differentiate the lumen wall shape, a frequency-domain method is disclosed. According to one embodiment of the present invention, the data series corresponding to the distances at various polar angles (as shown in FIGS. 5B, 5D, 6B, 6D and 6F) are determined. For example, if the data series contains N data and the N data are taken uniformly over 360°, two neighboring data are 360°/N apart. For example, if N is equal to 180, the distance data is determined every 2°. In this case, a₀ is the distance at 0 ° , a₁ is the distance at 2°, a₂ is the distance at 4°, . . . , and a_(180-1) is the distance at 358°. According to one embodiment of the present invention, the data series a₀, a₁, a₂, . . . , a_(N-1) are converted to transform domain data. Any discrete transform, such as Discrete Fourier Transform (DFT) can be used to obtain the transform domain data. The DFT, A₀, A₁, A₂, . . . , A_(N-1) for the input data, i.e., the data series a₀, a₁, a₂, . . . , a_(N-1) can be derived from the input data, i.e., the data series a₀, a₁, a₂, . . . , a_(N-1), according to:

$\begin{array}{cc}{A}_k={\sum }_{n=0}^{\left(N-1\right)}{a}_n{e}^{-\frac{2\pi i}{N}\mathrm{kn}},k=0,\dots ,\left(N-1\right)& \left(1\right)\end{array}$

The input data can be recovered by applying inverse DFT (IDFT) to the frequency-domain data, A₀, A₁, A₂, . . . , A_(N-1):

$\begin{array}{cc}{a}_n=\frac{1}{N}{\sum }_{k=0}^{\left(N-1\right)}{A}_n{e}^{\frac{2\pi i}{N}\mathrm{kn}},n=0,\dots ,\left(N-1\right)& \left(2\right)\end{array}$

The term A₀ is also referred as the DC term since A₀=(a₀+a₁+a₂+. . . +a_(N-1)). In other words, A₀ corresponds to the sum of input data. In this disclosure, A_i is referred as the i-th frequency term, where i=1, . . . , (N-1). It is well known in the field of digital signal processing that Fast Fourier Transform (FFT) is often applied when N is equal to a power of 2 number such as 128, 256, 512, etc. In this case, computationally efficient algorithms can be used to substantially reduce the required computations for forward and inverse DFT. The present invention can also be benefitted from the FFT.

The amplitudes of the lower frequency terms of the frequency-domain data are shown in FIGS. 8A-E for the centered circle case of FIG. 5A, the circle with eccentricity case of FIG. 5C, the centered triangle case of FIG. 6A, the centered triangle with sharper corners case of FIG. 6C, and the triangle with eccentricity case of FIG. 6E respectively. For the case of a centered circle of FIG. 5A, only the DC term, i.e., A₀, is non-zero and all the other frequency terms are zero as shown in FIG. 8A. When there is eccentricity in the case of a circle, the frequency domain still shows a dominant frequency term at A₀ as shown in FIG. 8B. However, there is also a notable frequency term at A₁ and a small frequency term at A₂ as shown in FIG. 8B. All other terms are negligible since their magnitudes are extremely small. Based on FIG. 8A and FIG. 8B, the effect of eccentricity is to cause some additional frequency terms, in particular A₁ and A₂. When the eccentricity is small (denoted as E), the A₀ in FIG. 8B can be derived as A₀=R−E²/4R, where R is the radius of the circle. When E is equal to 0 (i.e., no eccentricity), A₀ becomes R as expected. Furthermore, A₁ can be derived as A₁=E and A₂ can be derived as A₂=E²/4R. Therefore, larger eccentricity will cause larger A₁ and A₂.

In FIG. 8C, the amplitudes of the lower frequency terms are shown for a centered triangle, where a peak value exists at A₀. However, there is also a notable term at A₃ and a small term at A₆. In FIG. 8D, the amplitudes of the lower frequency terms are shown for a centered triangle with sharper corners, where a peak value exists at A₀. However, there is also a more notable term at A₃ and a small term at A₆. Compared with FIG. 8C, the frequency terms A₃ and A₆ are higher for FIG. 8D. In other words, sharper corners cause larger amplitudes in A₃ and A₆. In an extreme case where three corners are fully rounded as sections of a circle (i.e., the triangle becomes a circle), A₃ and A₆ become zero as expected. In the case of a triangle with eccentricity (as shown in FIG. 6E), the frequency terms are shown in FIG. 8E, where in addition to A₀, A₁ also has significant amplitude. As shown in FIG. 8E, there are also a few noticeable frequency terms in the low frequency region (e.g., A₂ to A₇).

As evidenced in FIGS. 8C and 8D, the centered triangular shape always has a notable frequency term at A₃ in additional to A₀. For a centered circle case, only A₀ term exists. Therefore, for the cases of centered circle and centered triangle, the frequency domain signal can provide a clean indication as being a circle or a triangle. In particular, the A₃ term can be a signature for the centered triangle shape. Therefore, checking the amplitude of A₃ will provide a good indication regarding whether an underlying shape is triangular or not. In one embodiment, the magnitude ratio of A₃ and A₀ is compared with a threshold. For example, the threshold includes a range from 0.13 to 0.27. If the magnitude ratio of A₃ and A₀ is greater than the threshold, the shape is determined to be a triangle. Otherwise, the shape is determined to be a circle.

Furthermore, when the capsule travels in the GI tract, it is almost impossible to keep the longitudinal axis of the capsule always aligned with the longitudinal direction of the GI tract. Therefore, tilting often exists when the capsule travels through the GI tract. When a circular model is used for the GI tract, the intersection of a measuring cone and the lumen wall has an elliptical shape instead of the circular shape when tilting exists. FIG. 9A illustrates an example of distances from the capsule to the circular lumen wall with tilting, where the capsule is tilted by about 60° from the longitudinal direction of the GI tract. As shown in FIG. 9A, the shape of the intersection of the lumen wall and a measuring cone in the case of tilting becomes an ellipse. FIG. 9B shows an example of tilting with eccentricity. The distance plots in the polar coordinate are shown in FIG. 9C and FIG. 9D for FIG. 9A and FIG. 9B respectively. The frequency response corresponding to FIG. 9A is shown in FIG. 9E, where the frequency response shows substantial frequency terms at A₂ (roughly over 30% of A₀) and A₄ (roughly 10% of A₀). Also, there is a small response at A₆ (roughly less than 5% of A₀ ). The frequency terms at A₁, A₃, A₅, etc. are negligible. Compared to FIG. 8A for the case without tilting, the effect of tilting causes some frequency response at frequency terms other than A₀ (i.e., A₂, A₄, A₆, etc.). While the tilting results in some noticeable frequency terms in the low frequency region, the signal at A₃ or its harmonics is very minimal. Therefore, tilting alone in the case of circular colon model doesn't seem to cause misclassification as a triangular colon. The frequency response corresponding to FIG. 9B is shown in FIG. 9F, where the amplitude of A₁ is almost as high as A₀. A₃ is also very sizeable (roughly 30% of A₀'s amplitude). Furthermore, there are also noticeable amplitudes for other lower frequency terms as shown in FIG. 9F. Since there is a sizeable signal strength at A₃, it may be hard to distinguish whether it is a circle or a triangle.

When a triangular model is used for the GI tract, the intersection of a measuring cone and the lumen wall will not have an equilateral shape anymore when tilting exists. FIG. 10A illustrates an example of distances from the capsule to the triangular lumen wall with tilting, where the capsule is tilted by about 60° from the longitudinal direction of the GI tract. As shown in FIG. 10A, the shape of the intersection of the lumen wall and a measuring cone in the case of tilting becomes an isosceles triangle. While the example in FIG. 10A shows an isosceles triangle due to tilting, the tilting may result in an arbitrary triangle. FIG. 10B shows an example of tilting with eccentricity. The distance plots in the polar coordinate are shown in FIG. 10C and FIG. 10D for FIG. 10A and FIG. 10B respectively. The frequency response corresponding to FIG. 10A is shown in FIG. 10E, where the frequency response shows noticeable frequency terms at A₁, A₂, A₄, A₅ and A₇ in additional to A₃ and A₆. Compared to FIG. 8C for the case without tilting, the effect of tilting causes some frequency response at frequency terms (i.e., A₁, A₂, A₄, A₅ and A₇, etc.). In this case, the tilting results in some noticeable frequency terms in the low frequency region, which may affect correct segmentation. The frequency response corresponding to FIG. 10B is shown in FIG. 10F, where the low frequency terms other than A₃ and A₆ become more prominent, which may further affect correct segmentation.

As illustrated above, the frequency domain characteristics between a centered circle and a centered triangle without tilting are very distinct. When tilting and/or eccentricity occurs, the distinction in the frequency domain characteristics becomes less clear, which may affect the correctness of frequency-domain based segmentation. In order to overcome this issue, an embodiment of the present invention corrects the eccentricity and/or tilting before applying the DFT. In order to offset the eccentricity, a method of coordinate translation is disclosed. In the coordinate translation process, we need to find the centroid of the 2D contour (e.g., the contours in FIG. 5C and FIG. 6E). In the (x,y) coordinate, the locations ((x_i,y_i), i=0, . . . , (N-1)) of the dots corresponding to the ending point of the measuring rays can be determined. The image moments M_pg for the image associated with the dots ((x_i,y_i) are defined as:

M_pq=Σ_i=0^N-1Σ_j=0^N-1x_i^py_j^q.   (3)

The centroid {x, y} can then be derived as:

$\begin{array}{cc}\left\{\stackrel{\_}{x},\stackrel{\_}{y}\right\}=\left\{\frac{M_{10}}{M_{00}}\frac{M_{01}}{M_{00}}\right\}.& \left(4\right)\end{array}$

Upon the determination of the centroid {x, y}, the contour for the lumen wall can be shifted so that the capsule is equivalently located in the virtual center of the lumen wall.

In yet another embodiment of the present invention, the tilting can be corrected by coordinate rotation. For the coordinate rotation process, the central moments μ_pq of the image is first derived according to:

μ_pq=Σ_i=0^N-1Σ_j=0^N-1(x_i−x)^p(y_j−y)^q.   (5)

Furthermore, a set of central moments are determined as follows:

$\begin{array}{cc}{\mu }_{20}^{\prime }=\frac{{\mu }_{20}}{{\mu }_{00}}=\frac{M_{20}}{M_{00}}-{\stackrel{\_}{x}}^2& \left(6\right)\end{array}$ $\begin{array}{cc}{\mu }_{02}^{\prime }=\frac{{\mu }_{02}}{{\mu }_{00}}=\frac{M_{02}}{M_{00}}-{\stackrel{\_}{y}}^2& \left(7\right)\end{array}$ $\begin{array}{cc}{\mu }_{11}^{\prime }=\frac{{\mu }_{11}}{{\mu }_{00}}=\frac{M_{11}}{M_{00}}-\stackrel{\_}{x}\stackrel{\_}{y}& \left(8\right)\end{array}$

A covariance matrix

$\left[\begin{array}{cc}{\mu }_{20}^{\prime }& {\mu }_{11}^{\prime }\\ {\mu }_{11}^{\prime }& {\mu }_{02}^{\prime }\end{array}\right]$

is then defined accordingly. The eigenvectors of this matrix correspond to the major and minor axes of the image intensity. The tilting angle can be determined according to:

$\begin{array}{cc}\cos \phi =\sqrt{\frac{I_b}{I_a}}.& \left(9\right)\end{array}$

In the above equation, I_a and I_b correspond to the eigenvalues of the major axis and the minor axis respectively.

The coordinate translation and coordinate rotation processes mentioned above can be applied to the distance information to offset the eccentricity and correct the tilting. After the coordinate translation and coordinate rotation processes, the eccentricity and tilting should be removed or substantially reduced. Therefore, the eccentricity/tilting corrected distance information should provide more reliable segmentation. The coordinate translation and coordinate rotation processes can be performed by applying the coordinate translation first followed by the coordinate rotation. Alternatively, the coordinate rotation can be applied first and followed by the coordinate translation. Furthermore, a system may choose to apply only one of these processes. For example, a system may apply coordinate translation without coordinate rotation. In another example, the system may choose to use coordinate rotation without coordinate translation.

FIG. 5C illustrates an example of eccentricity without tilting for the circular GI tract. The coordinate translation process mentioned above can be applied to the distance information to correct the eccentricity. In other words, the process as described in equations (3) and (4) can be applied to the distance information to determine the centroid. After the centroid is obtained, the distance information as represented in the (x,y)-coordinate can be offset by the derived centroid. After the coordinate translation process, the eccentricity-offset distance information becomes centered as shown in FIG. 5A.

FIG. 6E illustrates an example of eccentricity without tilting for the triangular GI tract. The coordinate translation process mentioned above can be applied to the distance information. After the coordinate translation process, the eccentricity-offset distance information becomes centered as shown in FIG. 6A.

As shown in FIGS. 8A and 8C, the characteristics of frequency response between a centered circle and a centered triangle without tilting are very distinct. Therefore, the distance information of FIG. 5C becomes a centered circle as shown in FIG. 5A and the distance information of FIG. 6E becomes a centered triangle as shown in FIG. 6A after the coordinate translation process. As shown in FIGS. 8A and 8C, the frequency-domain characteristics of the centered circle and the centered triangle are very distinct. Thus, the segmentation can be performed reliably.

FIG. 9A illustrates an example of a centered capsule with tilting for the circular GI tract. The coordinate rotation process mentioned above can be applied to the distance information (i.e., an ellipse) to correct the tilting. In other words, the process as described in equations (3)-(9) can be applied to the distance information to determine the tilting angle. After the tilting angle is obtained, the distance information as represented in the (x,y)-coordinate can be rotated by the derived tilting angle. After the coordinate rotation process, the rotation-corrected distance information becomes a circle as shown in FIG. 5A.

FIG. 10A illustrates an example of a centered capsule with tilting for the triangular GI tract. The coordinate rotation process mentioned above can be applied to the distance information. After the coordinate rotation process, the rotation-corrected distance information becomes a circle as shown in FIG. 6A.

As shown in FIGS. 8A and 8C, the characteristics of frequency response between a centered circle and a centered triangle without tilting are very distinct. Therefore, the distance information of FIG. 9A is processed by coordinate rotation to result in a centered circle without tilting as shown in FIG. 5A and the distance information of FIG. 10A is processed by coordinate rotation to result in a centered triangle without tilting as shown in FIG. 6A. the frequency-domain characteristics of the centered circle and the centered triangle are very distinct. Thus, the segmentation can be performed reliably.

FIG. 9B illustrates an example of eccentricity with tilting for the circular GI tract. The coordinate rotation process mentioned above can be applied to the distance information. After the coordinate rotation process, the rotation-corrected distance information is only affected by eccentricity as shown in FIG. 5C. The coordinate translation process mentioned above can be further applied to the processed distance information. After the coordinate translation process, the further processed distance information becomes centered as shown in FIG. 5A. While the coordinate rotation is applied and followed by the coordinate translation in the above example, the order of the two processes can also be swapped, i.e., the coordinate translation first followed by the coordinate rotation. In this case, the processed distance information after the first step is shown in FIG. 9B. After the coordinate rotation is applied, the processed distance information after the second step is shown in FIG. 5A.

FIG. 10B illustrates an example of eccentricity with tilting for the triangular GI tract. The coordinate rotation process mentioned above can be applied to the distance information. After the coordinate rotation process, the rotation-corrected distance information is only affected by eccentricity as shown in FIG. 6E. The coordinate translation process mentioned above can be further applied to the processed distance information. After the coordinate translation process, the further processed distance information becomes centered as shown in FIG. 6A. While the coordinate rotation is applied and followed by the coordinate translation in the above example, the order of the two processes can also be swapped, i.e., coordinate translation first followed by coordinate rotation. In this case, the processed distance information after the first step is shown in FIG. 10A. After the coordinate rotation is applied, the processed distance information after the second step is shown in FIG. 6A.

As shown in FIGS. 8A and 8C, the characteristics of frequency response between a centered circle and a centered triangle without tilting are very distinct. Therefore, the distance information of FIG. 9B is processed by coordinate translation and coordinate rotation to result in a centered circle without tilting as shown in FIG. 5A and the distance information of FIG. 10B is processed by coordinate translation and coordinate rotation to result in a centered triangle without tilting as shown in FIG. 6A. The frequency-domain characteristics of the centered circle and the centered triangle are very distinct. Thus, the segmentation can be performed reliably.

In reality, the shape of the lumen wall of the colon is never a perfect circle nor a perfect triangle. It is expected that there will be some variations between an actual lumen wall and the ideal model. The method of eccentricity correction has been applied to a set of measured distance data in-vitro. FIG. 11A illustrates an example of distances from the capsule to the lumen wall of the transverse colon in an in-vitro environment, where the capsule is off center without tilting. As shown in FIG. 11A, the transverse colon wall has a triangular shape. FIG. 11B illustrates the corresponding distances in the polar coordinate for FIG. 11A. As mentioned earlier, the centroid can be determined according to equations (3) and (4). The eccentricity can be corrected accordingly. FIG. 11C illustrates the distances for FIG. 11A after eccentricity correction according to an embodiment of the present invention. FIG. 11D illustrates the distances in the polar coordinate for FIG. 11C. FIG. 11E illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 11A. As shown in FIG. 11E, the eccentricity causes significant frequency terms at A₁, A₂ and A₄. FIG. 11F illustrates the amplitude of the lower frequency terms of the frequency response for FIG. 11C. After eccentricity correction, frequency terms at A₁, A₂ and A₄ are substantially suppressed while frequency term at A₃ becomes larger. Based on the frequency response in FIG. 11F, the characteristics of a triangular shape becomes very obvious.

In another embodiment, we may utilize a derivative of the distance information for determining the segmentation of the colon associated with the capsule location. For example, we may use the lumen size (i.e., the y axis) as a function of time (i.e., the x axis) and determine the segmentation based on the derivative of the lumen size function. In another embodiment, the frame number could be used for the x axis. In another embodiment, an index of the shape is used to indicate that the capsule is in ascending colon, transverse colon, descending colon or more distant colon based on the index. For example, the index of the shape may be based on the frequency transformation of the geometry derived from distance information. For example, the frequency transformation corresponds to a ratio of the third frequency-term magnitude and the zero-th frequency-term magnitude. Two examples of the shape index based segmentation are shown in FIGS. 12A-B, where the ascending colon, transverse colon and descending colon can be discerned from the shape index. Since the distance information or its derivative varies from frame to frame, a smoothed distance information may be more reliable. For example, the smoothed distance information may be derived as a moving average of every N frames or a filtered information can be used to draw the plot.

In another embodiment, the travelled distance of the capsule device can be used as the x-axis for the plots (e.g., distance, shape, shape index or frequency information) instead of the frame number or the frame time. A method of measuring the travelled distance of the capsule device has been disclosed in U.S. Pat. No. 10,506,921, issued on Dec. 17, 2019. According to U.S. Pat. No. 10,506,921, the motion vectors can be used to derive the travelled distance of the capsule device in the longitudinal direction of the GI tract. Furthermore, U.S. Pat. No. 10,506,921 also discloses a method to refine or adjust the travelled distance based on the distance information between the capsule device and the surrounding lumen wall. Therefore, the anatomical geometry information along the colon longitudinal direction can be assessed by a GI doctor before the colonoscopy procedure is performed to remove the polyp to make the procedure with less efforts while enhancing safety.

FIG. 13 illustrates an exemplary flowchart for identifying a GI (gastrointestinal) part associated with a location of a capsule device within a human GI tract according to an embodiment of the present invention. According to this method, one or more structured-light images captured by the capsule device when the capsule device is inside the human GI tract are received in step 1310, where said one or more structured-light images are captured by projecting a plurality of structured light beams from the capsule device onto a surrounding lumen wall. The structured-light images may be captured along with regular images so that the shape/size information derived from the structured-light image may be used by the regular images. Distance information is derived based on said one or more structured-light images for a set of measuring rays from the capsule device to the surround lumen wall in step 1320. Information regarding the GI part associated with the location of the capsule device is provided based on the distance information in step 1330, wherein the GI part belongs to a group comprising a transverse colon. The GI part associated with the location of the capsule device may provide useful information for the corresponding regular images. For example, if a polyp is found in a regular image, the location of the capsule device (presumably the regular image with the polyp is captured at this location) can be used to identify the location of the polyp.

The method for identifying a GI (gastrointestinal) part associated with the location of the capsule device within the human GI tract can be implemented using one or more electronic circuits or processors. Furthermore, said one or more electronic circuits or processors can be programmable according to software or firmware. Said one or more electronic circuits or processors can be embedded within the capsule device and perform all the processing steps mentioned above. In this case, the location of the capsule device within the human GI tract can be determined in real-time or with a small processing delay after one or more structured-light images are received. The fully embedded approach will consume more precious battery power of the capsule device. However, the capsule device may take advantage of the real-time or near real-time location information and perform certain task adaptively. For example, the capsule device may incorporate a camera to capture regular images of the lumen wall. The capture rate may be different for the transverse colon and ascending/descending colon.

In another embodiment, the processing steps mentioned above may be performed by one or more electronic circuits or processors external to the capsule device. For example, the structured-light images may be transmitted by a radio-frequency transmitter inside the capsule device to an external radio-frequency receiver. One or more electronic circuits or processors, a laptop, a computer or a workstation may be used to derive the location of the capsule device within the human GI tract. The location of the capsule device within the human GI tract can be correlated with regular images captured by the capsule device to determine the corresponding GI part associated with the regular images. For example, if a regular image contains abnormality (e.g., a polyp), a medical professional will be able to determine which part of the GI tract (e.g., descending color) that the abnormality is located. In another example, the capsule device may have on-board storage to store the captured regular images and structured images. Upon excretion from the human body, the capsule device can be retrieved and the stored images can be down loaded. The processing steps mentioned above can be performed by one or more electronic circuits or processors, a laptop, a computer or a workstation according to one embodiment of the present invention.

The above description is presented to enable a person of ordinary skill in the art to practice the present invention as provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those with skill in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features herein disclosed. In the above detailed description, various specific details are illustrated in order to provide a thorough understanding of the present invention. Nevertheless, it will be understood by those skilled in the art that the present invention may be practiced.

The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

1. A method of identifying a colon anatomical part associated with a location of a capsule device within a human colon, the method comprising: receiving one or more structured-light images captured by the capsule device when the capsule device is inside the human colon, wherein said one or more structured-light images are captured by projecting a plurality of structured light beams from the capsule device onto a surrounding lumen wall;deriving distance information based on said one or more structured-light images for a target set of measuring rays from the capsule device to the surrounding lumen wall; andproviding information regarding the colon anatomical part associated with the location of the capsule device associated with a distance-measuring frame related to said one or more structured-light images based on the distance information, wherein the colon anatomical part belongs to a group comprising a transverse colon.

2. The method claim 1, wherein the target set of measuring rays are on a plane perpendicular to a longitudinal axis of the capsule device and each of the measuring rays ends at an intersection of the plane and an interior point of the surrounding lumen wall.

3. The method claim 2, further comprising displaying visual representation of ending points of the measuring rays in a Cartesian coordinate on a display device.

4. The method claim 1, further comprising providing information indicative of a shape of the surrounding lumen wall based on the distance information, wherein the location of the capsule device is indicated according to whether the shape of the surrounding lumen wall resembles a circle or a triangle.

5. The method claim 4, wherein curve fitting is used to indicate whether the shape of the surrounding lumen wall resembles the triangle.

6. The method claim 4, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of the shape of the surrounding lumen wall in one axis and a frame number or a frame time in another axis.

7. The method claim 6, wherein the colon anatomical part associated with the location of the capsule device is indicated according to the shape of the surrounding lumen wall, and wherein if the shape resembles the triangle, the colon anatomical part associated with the location of the capsule device is the transverse colon.

8. The method claim 4, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of a shape index of the surrounding lumen wall in one axis and a frame number or a frame time in another axis, and wherein the shape index is derived based on frequency-domain data of the distance information represented in a polar coordinate.

9. The method claim 8, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of the shape index of the surrounding lumen wall in one axis and a travelled distance of the capsule device in another axis.

10. The method claim 8, wherein the colon anatomical part associated with the location of the capsule device is indicated according to the shape index and if the shape index indicates the triangle, the colon anatomical part associated with the location of the capsule device is the transverse colon.

11. The method claim 4, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of the shape of the surrounding lumen wall in one axis and a travelled distance of the capsule device in another axis.

12. The method claim 1, further comprising determining a size of the surrounding lumen wall based on the distance information between the capsule device and the surrounding lumen wall.

13. The method claim 12, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of the size of the surrounding lumen wall in one axis and a frame number or a frame time in another axis.

14. The method claim 13, wherein the colon anatomical part associated with the location of the capsule device is indicated according to the size or a trend of the size of the surrounding lumen wall.

14. od claim 14, wherein a section of the human colon corresponds to an ascending colon if the section corresponds to a first part of the human colon and the size is reduced or the size is gradually reducing.

16. The method claim 13, wherein the size of the surrounding lumen wall is determined based on the distance information associated with multiple distance-measuring frames close to each other in a temporal order.

17. The method claim 16, wherein the colon anatomical part associated with the location of the capsule device is indicated based on the distance information associated with the multiple distance-measuring frames, and a decision of the colon anatomical part associated with the location of the capsule device is confirmed when a same decision out of multiple consecutive decisions is reached.

18. The method claim 13, wherein the plot representative of the size of the surrounding lumen wall corresponds an average or filtered size of the surrounding lumen wall associated with multiple measuring frames.

19. The method claim 12, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of the size of the surrounding lumen wall in one axis and a travelled distance of the capsule device in another axis.

20. The method claim 1, further comprising transforming the distance information represented in a polar coordinate into discrete frequency-domain information, wherein the colon anatomical part associated with the location of the capsule device is indicated by the discrete frequency-domain information.

21. The method claim 20, wherein Discrete Fourier Transform (DFT) or Fast Fourier Transform (FFT) is used to transform the distance information into the discrete frequency-domain information.

22. The method claim 20, wherein if all frequency terms are insignificant except for zero-th frequency term, the colon anatomical part associated with the location of the capsule device corresponds to ascending or descending colon.

23. The method claim 20, wherein if the discrete frequency-domain information has a maximum at zero-th frequency term of the discrete frequency-domain information and a second largest term at third frequency term of the discrete frequency-domain information, the colon anatomical part associated with the location of the capsule device corresponds to the transverse colon.

24. The method claim 20, further comprising determining a magnitude ratio of zero-th frequency term and third frequency term of the discrete frequency-domain information, wherein if ratio is larger than a threshold, the colon anatomical part associated with the location of the capsule device corresponds to the transverse colon and, otherwise the colon anatomical part associated with the location of the capsule device corresponds to ascending/descending colon, and wherein the threshold includes a range between 0.13 and 0.27.

25. The method claim 20, further comprising applying coordinate translation to the distance information represented in the polar coordinate to correct eccentricity prior to said transforming the distance information into the discrete frequency-domain information.

26. The method claim 20, further comprising applying coordinate rotation to the distance information represented in the polar coordinate to correct tilting prior to said transforming the distance information into the discrete frequency-domain information.

27. The method claim 20, further comprising applying coordinate translation to the distance information represented in the polar coordinate to correct eccentricity following by applying coordinate rotation to the distance information to correct tilting prior to said transforming the distance information into the discrete frequency-domain information.

28. The method claim 20, further comprising applying coordinate rotation to the distance information represented in the polar coordinate to correct tilting following by applying coordinate translation to the distance information to correct eccentricity prior to said transforming the distance information into the discrete frequency-domain information.

29. The method claim 1, further comprising receiving a regular image capsuled by a camera in the capsule device for a scene in a field view of the camera and determining an anomaly in the regular image, wherein the regular image is temporally close to said one or more structured-light images and the colon anatomical part associated with the location of the anomaly within the human colon is indicated based on the distance information.

30. The method of claim 1, further comprising determining fold depth information of the surrounding lumen wall based on the distance information.

31. The method of claim 30, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of the fold depth information of the surrounding lumen wall in one axis and a frame number or a frame time in another axis.

32. The method of claim 31, wherein the colon anatomical part associated with the location of an anomaly within the human colon is indicated based on the fold depth information.

33. The method of claim 30, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of the fold depth information of the surrounding lumen wall in one axis and a travelled distance of the capsule device in another axis.

34. The method of claim 1, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of discrete frequency-domain data in one axis and a frame number or a frame time in another axis, wherein the discrete frequency-domain data are derived by applying frequency transformation to the distance information represented in a polar coordinate.

35. The method of claim 34, wherein the discrete frequency-domain data corresponds to a ratio of a magnitude of third frequency term to a magnitude of zero-th frequency term.

36. The method of claim 1, further comprising displaying a plot on a display device for a user to visualize, wherein the plot is representative of discrete frequency-domain data in one axis and a travelled distance of the capsule device in another axis, wherein the discrete frequency-domain data are derived by applying frequency transformation to the distance information represented in a polar coordinate.

37. A system for identifying a colon anatomical part associated with a location of a capsule device within a human colon, the system comprising one or more electronic circuits or processors configured to: receive one or more structured-light images captured by the capsule device when the capsule device is inside the human colon, wherein said one or more structured-light images are captured by projecting a plurality of structured light beams from the capsule device onto a surrounding lumen wall;derive distance information based on said one or more structured-light images for a target set of measuring rays from the capsule device to the surrounding lumen wall; andprovide information regarding the colon anatomical part associated with the location of the capsule device based on the distance information, wherein the colon anatomical part belongs to a group comprising a transverse colon.