Patent Grant
11219424
This application claims priority to U.S. application Ser. No. 16/155,943 filed Oct. 10, 2018 which claims priority of U.S. application Ser. No. 15/081,788 (U.S. Pat. No. 10,136,869 issued Nov. 27, 2018) filed Mar. 25, 2016, the entire disclosure of which is hereby incorporated herein by reference.
This invention relates generally to methods and systems of image analysis. More particularly, in certain embodiments, the invention relates to detection and localization of a bone central axis from an image of a subject (e.g., mammal), e.g., captured with a computed tomography (CT) scanner.
There is a wide array of technologies directed to in vivo and ex vivo imaging of mammals—for example, bioluminescence, fluorescence, X-ray computed tomography, and multimodal imaging technologies. In vivo imaging of small mammals and ex vivo imaging of samples from small mammals is performed by a large community of investigators in various fields, e.g., oncology, infectious disease, and drug discovery.
Micro computed tomography (hereafter, “microCT”) imaging, is an x-ray-based technology that can image tissues, organs, and non-organic structures with an extremely high resolution. MicroCT has evolved quickly, requiring low dose scanning and fast imaging protocols to facilitate multi-modal applications and enable longitudinal experimental models. Similarly, nano-computed tomography (nanoCT) systems designed for high-resolution imaging of ex vivo samples are also now used. Multi-modal imaging involves the fusion of images obtained in different ways, for example, by combining fluorescence molecular tomography (FMT), PET, MRI, CT, and/or SPECT imaging data.
Conventional image analysis applications and/or imaging systems typically allow for visualization, analysis, processing, segmentation, registration and measurement of biomedical images. These applications and systems also provide volume rendering tools (e.g., volumetric compositing, depth shading, gradient shading, maximum intensity projection, summed voxel projection, signal projection); manipulation functions (e.g., to define areas of structures of interest, delete unwanted objects, edit images and object maps); and measurement functions (e.g., calculation of number of surface voxels, number of exposed faces, planar area of a region, estimated surface area of a region).
Acquisition of animal images can be time consuming, and rapid analysis of the acquired images is key to the efficiency of the process. Three dimensional (3D) imaging software, including microCT image analysis, enables extraction of structural, biological, and anatomical attributes from images, such as thickness, porosity, anisotropy, and other measures, of organs of interest, such as bones. Due to the anatomical contrast and high spatial resolution provided by microCT systems, they are widely used for studying skeletal bone formation, structure, and diseases. Automation of such analyses improves throughput, accuracy, and efficacy. In classical bone analysis approaches, researchers were required to visually and manually quantify the structural attributes of bones using printed images produced by the microCT platform. While some image analysis systems have been developed for computer-aided bone analysis, the digital workflows offered by bone analysis software still require considerable manual input and interaction from users and researchers. For example, such manual feedback is currently required to obtain stereological measures of cortical and trabecular bone compartments, e.g., manual selection of discrete 2-D slices of a 3-D bone image from which averaged thicknesses or other properties are determined.
Some conventional image analysis systems focus on locating the principal axes of the bones to extract the direction of 2-D slices of the bone. But principal axes do not carry detailed shape and directional information. Principal axes represent the major and minor directional axes of a bone, as shown in
There is a need for automated, precise, and improved methods for stereological analysis and slice-by-slice characterization of bones in images, such as microCT images.
Automated detection of central axes of skeletal bones significantly improves the speed, efficiency, and automation of slice-by-slice measurements and analyses of bones. Central axes of long bones (e.g., bones of the extremities that have a length greater than the width, e.g., femur) can effectively encapsulate the spatial features, direction, orientation, and shape of long bones. Calculation of central axes is essential for performing automated and accurate 2-D slice-by-slice planar studies such as stereological studies on bones, for example, the femur and the tibia. The 2-D planes perpendicular to the central axes constitute the slices that are used in 2-D bone analysis or stereology measurements. Automated detection of bone central axis and the 2-D stereology slices allows for fully automated computer-based stereological measurements of bones.
Presented herein are efficient and reliable systems and methods for calculating and extracting three-dimensional central axes of skeletal bones of animal subjects—for example, animal subjects scanned by in vivo microCT platforms and ex vivo samples of animal subjects scanned by microCT or nanoCT platforms—to capture both the general and localized tangential directions of the bone, along with its shape, form, curvature, and orientation. With bone detection and segmentation algorithms, the bones of animal subjects scanned by CT, nanoCT, or microCT scanners can be detected, segmented, and visualized automatically. Three dimensional central axes determined using these methods provide important information about the bones.
The detection and localization of central axes improves the speed and accuracy of stereology studies performed visually and manually and circumvents the limitations of principal axes and provides, inter alia, directional, shape, and curvature information regarding the bone. The detection and localization of central axes reveals a variety of features relating to the shape, direction, and curvature of the bone that are not available through the existing method of using the principal axes. This is particularly useful for analysis of curved or non-straight long bones, and for 2-D slice-by-slice analysis, e.g., 2-D planar slice-by-slice stereology studies of the tibia or pelvic girdle. The central axis of a bone represents the medial path that describes the main center-line, shape, and orientation of a bone.
An automated procedure for identifying a central bone axis is not a simple problem, since the procedure would need to accurately identify the central bone axis for a wide range of sizes and shapes of the bones being imaged (e.g., it would need to account for different kinds of bones as well as variability between the same bone across multiple subjects) without user interaction or training, and it would need to be a computationally efficient procedure.
Presented herein, in certain embodiments, are systems and methods for automated computation of a bone central axis from a 3D anatomical image. An area of the subject (e.g., mammal) including a bone of interest is scanned (e.g., with a microCT system), and a 3D anatomical image of the area of the subject is obtained. In some embodiments, in a first step, a binary mask of the bone of interest is filled using morphological processing. In some embodiments, the binary mask of the bone is filled by morphological processing to more accurately reflect the internal composition of the bone (e.g., to accurately model the distinct trabecular and cortical components of the bone). In some embodiments, in a second step, skeletonization (e.g., morphological skeletonization) is performed on the filled bone by iterative 3-D thinning. In some embodiments, in a third step, the skeleton is pruned (e.g., thinned) and reduced to a single (e.g., a main and/or central) branch. In some embodiments, the skeleton is pruned down to the single branch and smoothed, yielding a single-branched curve that follows the medial path of the bone, effectively identifying and isolating the central axis of the bone.
Also described herein are systems and methods to efficiently fill the morphological holes on the six exterior faces of a three-dimensional (hereafter, “3-D”) binary image. Some embodiments described herein relate to systems and methods for filling 2-D morphological holes which extend across three faces. In some embodiments, these morphological holes on the boundary are due to hollow internal compartments of partially out-of-view bones.
Some example embodiments described herein relate to calculating and extracting central axes of skeletal bones (e.g., long bones) to capture both the general and localized tangential directions of the bones. The calculation of central axes also identifies, among other things, the shape, form, and curvature of the bones. That is, the central axis represents a medial path that describes, among other things, the main center-line, shape, and orientation of a long bone.
In one aspect, the invention is directed to a method for automatically identifying a three-dimensional (3-D) central axis of a bone of interest in a 3-D image, the method comprising: receiving, by a processor of a computing device, the 3-D image of one or more bones, comprising the bone of interest (i.e., at least a portion of the bone of interest), of a mammal; isolating, by the processor, the bone of interest from the one or more bones in the 3-D image (e.g., yielding an isolated image of an exterior surface of a cortical tissue of the bone of interest); generating, by the processor (e.g., after the isolating of the bone of interest), a binary bone mask of the bone of interest; generating, by the processor, a filled bone mask for the bone of interest using the binary bone mask; generating, by the processor, a skeleton of the bone of interest (e.g., by performing iterative 3-D thinning of the filled bone mask); and generating, by the processor, a pruned skeleton to reduce the skeleton to a branch (e.g., a single branch, central branch, and/or main branch) corresponding to the 3-D central axis of the bone of interest.
In certain embodiments, the bone of interest is a long bone of the mammal (e.g., a femur, tibia, fibula, humerus, radius, ulna, metacarpal, metatarsal, phalange, and clavicle). In certain embodiments, the bone of interest is a non-long bone (e.g., a short bone, flat bone, sesamoid bone, or irregular bone) of the mammal (e.g., pelvic girdle).
In certain embodiments, the 3-D image is obtained by a computed tomography scanner (e.g., a micro computed or nano computed tomography scanner). In certain embodiments, the 3-D image is captured in vivo. In certain embodiments, the 3-D image is captured ex vivo. In certain embodiments, the 3-D image is a computed tomography image of an exterior surface of cortical tissue of the one or more bones.
In certain embodiments, generating the filled bone mask for the bone of interest comprises performing, by the processor, morphological processing of the portion of the 3-D image corresponding to the bone of interest, said processing comprising: performing 3-D binary dilation of the binary bone mask of the bone of interest (e.g., with a spherical structuring element) to form a dilated bone mask; and identifying and filling borders and/or morphological holes (e.g., gaps and/or discontinuities) of the dilated bone mask, then processing the result (e.g., performing 3-D binary erosion on the result of the border and hole filling operations) to generate the filled bone mask for the bone of interest.
In certain embodiments, the method comprises filling borders of the bone of interest by: representing image data from the binary bone mask of the bone of interest digitally as one or more data-cubes; identifying a vertex of a data-cube, the vertex having all edges connected to the vertex associated with true (e.g., binary true) voxels; forming a 2-D image from the three faces connected to the identified vertex (e.g., by adding an all-zero face as one of the quadrants and diagonally connecting binary true voxels on the boundaries of the all-zero face quadrant); filling morphological holes in the thusly formed 2-D image to produce a filled surface; and mapping the filled surface back to the three corresponding faces of the data-cube.
In certain embodiments, generating the 3-D skeleton of the bone of interest comprises performing, by the processor, morphological processing of the filled bone mask, the processing comprising performing iterative 3-D thinning of the filled bone mask.
In certain embodiments, generating the pruned 3-D skeleton comprises performing, by the processor, morphological processing of the skeleton for the bone of interest, said processing comprising: identifying a single-branched centerline tree or a single-cycle main loop of the skeleton as a main path; pruning the skeleton by removing minor branches not included in the main path; and smoothing the pruned skeleton (e.g., by point averaging), thereby generating the pruned 3-D skeleton.
In certain embodiments, the method comprises characterizing the bone of interest according to the 3-D central axis corresponding to the bone of interest (e.g., identifying an abnormality of the bone and/or identifying the bone as a specific bone of the mammal).
In certain embodiments, the method comprises rendering an image using at least the 3-D central axis of the bone of interest.
In certain embodiments, the method comprises performing, by the processor, a stereological measurement of the bone of interest using the identified 3-D central axis of the bone of interest, said performing of the stereological measurement comprising: producing a plurality of graphical 2-D cross-sections (e.g., 2-D image slices) of the bone of interest in planes perpendicular to the identified 3-D central axis at various locations along a length of the bone of interest; for each of the graphical 2-D cross-sections, determine a measurement of the bone as depicted in the graphical 2-D cross section (e.g., identifying a cortical thickness for each of the 2-D image slices); and obtaining the stereological measurement of the bone of interest using the measurements determined from the plurality of graphical 2-D cross-sections (e.g., obtaining an average cortical thickness for the bone of interest as an average of the measurements determined from the 2-D image slices).
In certain embodiments, the method comprises determining, by the processor, one or more of (i) to (iii)—(i) the presence of a disease state, (ii) a disease state risk, and/or (iii) an extent of disease progression (e.g., staging of a disease)—using the identified 3-D central axis of the bone of interest (e.g., based on one or more stereological measurements of the bone of interest determined using the identified 3-D central axis of the bone of interest).
In another aspect, the invention is directed to a method of automatically filling borders in an image of an object (e.g., a bone) of interest, the method comprising: digitally representing image data from a binary mask of an object (e.g., a bone of interest) as one or more data-cubes; identifying, by a processor of a computing device, a vertex of a data-cube of the one or more data-cubes, the vertex having all edges connected to the vertex associated with true (e.g., binary true) voxels; forming, by the processor, a 2-D image from the three faces connected to the identified vertex (e.g., by adding an all-zero face as one of the quadrants and diagonally connecting binary true voxels on the boundaries of the all-zero face quadrant); filling, by the processor, morphological holes in the thusly formed 2-D image to produce a filled surface; and mapping, by the processor, the filled surface back to the three corresponding faces of the data-cube.
In another aspect, the invention is directed to a system for automatically identifying a three-dimensional (3-D) central axis of a bone of interest in a 3-D image, the system comprising: a processor; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to: receive the 3-D image of one or more bones, comprising the bone of interest (i.e., at least a portion of the bone of interest), of a mammal; isolate the bone of interest from the one or more bones in the 3-D image (e.g., yielding an isolated image of an exterior surface of a cortical tissue of the bone of interest); generate (e.g., after the isolating of the bone of interest), a binary bone mask of the bone of interest; generate a filled bone mask for the bone of interest using the binary bone mask; generate a skeleton of the bone of interest (e.g., by performing iterative 3-D thinning of the filled bone mask); and generate a pruned skeleton to reduce the skeleton to a branch (e.g., a single branch, central branch, and/or main branch) corresponding to the 3-D central axis of the bone of interest.
In certain embodiments, the bone of interest is a long bone of the mammal (e.g., a femur, tibia, fibula, humerus, radius, ulna, metacarpal, metatarsal, phalange, and clavicle). In certain embodiments, the bone of interest is a non-long bone (e.g., a short bone, flat bone, sesamoid bone, or irregular bone) of the mammal (e.g., pelvic girdle). In certain embodiments, the 3-D image is obtained by a computed tomography scanner (e.g., a micro computed or nano computed tomography scanner).
In certain embodiments, the 3-D image is captured in vivo. In certain embodiments, the 3-D image is captured ex vivo. In certain embodiments, the 3-D image is a computed tomography image of an exterior surface of cortical tissue of the one or more bones.
In certain embodiments, the instructions cause the processor to generate the filled bone mask for the bone of interest by performing morphological processing of the portion of the 3-D image corresponding to the bone of interest, said processing comprising: performing 3-D binary dilation of the binary bone mask of the bone of interest (e.g., with a spherical structuring element) to form a dilated bone mask; and identifying and filling borders and/or morphological holes (e.g., gaps and/or discontinuities) of the dilated bone mask, then processing the result (e.g., performing 3-D binary erosion on the result of the border and hole filling operations) to generate the filled bone mask for the bone of interest.
In certain embodiments, the instructions cause the processor to fill borders of the bone of interest by: representing image data from the binary bone mask of the bone of interest digitally as one or more data-cubes; identifying a vertex of a data-cube, the vertex having all edges connected to the vertex associated with true (e.g., binary true) voxels; forming a 2-D image from the three faces connected to the identified vertex (e.g., by adding an all-zero face as one of the quadrants and diagonally connecting binary true voxels on the boundaries of the all-zero face quadrant); filling morphological holes in the thusly formed 2-D image to produce a filled surface; and mapping the filled surface back to the three corresponding faces of the data-cube.
In certain embodiments, the instructions cause the processor to generate the 3-D skeleton of the bone of interest by performing morphological processing of the filled bone mask, the processing comprising performing iterative 3-D thinning of the filled bone mask.
In certain embodiments, the instructions cause the processor to generate the pruned 3-D skeleton by performing morphological processing of the skeleton for the bone of interest, said processing comprising: identifying a single-branched centerline tree or a single-cycle main loop of the skeleton as a main path; pruning the skeleton by removing minor branches not included in the main path; and smoothing the pruned skeleton (e.g., by point averaging), thereby generating the pruned 3-D skeleton.
In certain embodiments, the instructions cause the processor to characterize the bone of interest according to a central axis corresponding to the bone of interest (e.g., identifying an abnormality of the bone and/or identifying the bone as a specific bone of the mammal). In certain embodiments, the instructions cause the processor to render an image using at least the 3-D central axis of the bone of interest. In certain embodiments, the instructions cause the processor to perform a stereological measurement of the bone of interest using the identified 3-D central axis of the bone of interest, said performing of the stereological measurement comprising: producing a plurality of graphical 2-D cross-sections (e.g., 2-D image slices) of the bone of interest in planes perpendicular to the identified 3-D central axis at various locations along a length of the bone of interest; for each of the graphical 2-D cross-sections, determining a measurement of the bone as depicted in the graphical 2-D cross section (e.g., identifying a cortical thickness for each of the 2-D image slices); and obtaining the stereological measurement of the bone of interest using the measurements determined from the plurality of graphical 2-D cross-sections (e.g., obtaining an average cortical thickness for the bone of interest as an average of the measurements determined from the 2-D image slices). In certain embodiments, the instructions cause the processor to determine one or more of (i) to (iii)-(i) the presence of a disease state, (ii) a disease state risk, and/or (iii) an extent of disease progression (e.g., staging of a disease)—using the identified 3-D central axis of the bone of interest (e.g., based on one or more stereological measurements of the bone of interest determined using the identified 3-D central axis of the bone of interest).
In another aspect, the invention is directed to a system for automatically filling borders in an image of an object (e.g., a bone) of interest, the system comprising: a processor; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to: digitally represent image data from a binary mask of an object (e.g., a bone of interest) as one or more data-cubes; identify a vertex of a data-cube of the one or more data-cubes, the vertex having all edges connected to the vertex associated with true (e.g., binary true) voxels; form a 2-D image from the three faces connected to the identified vertex (e.g., by adding an all-zero face as one of the quadrants and diagonally connecting binary true voxels on the boundaries of the all-zero face quadrant); fill morphological holes in the thusly formed 2-D image to produce a filled surface; and map the filled surface back to the three corresponding faces of the data-cube.
Embodiments described with respect to one aspect of the invention may be, applied to another aspect of the invention (e.g., features of embodiments described with respect to one independent claim are contemplated to be applicable to other embodiments of other independent claims).
The foregoing and other objects, aspects, features, and advantages of the invention will become more apparent and may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:
It is contemplated that systems, devices, methods, and processes of the claimed invention encompass variations and adaptations developed using information from the embodiments described herein. Adaptation and/or modification of the systems, devices, methods, and processes described herein may be performed by those of ordinary skill in the relevant art.
Throughout the description, where articles, devices, and systems are described as having, including, or comprising specific components, or where processes and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are articles, devices, and systems of the present invention that consist essentially of, or consist of, the recited components, and that there are processes and methods according to the present invention that consist essentially of, or consist of, the recited processing steps.
It should be understood that the order of steps or order for performing certain action is immaterial so long as the invention remains operable. Moreover, two or more steps or actions may be conducted simultaneously.
The mention herein of any publication, for example, in the Background section, is not an admission that the publication serves as prior art with respect to any of the claims presented herein. The Background section is presented for purposes of clarity and is not meant as a description of prior art with respect to any claim.
As used herein, an “image”—for example, a 3-D image of mammal—includes any visual representation, such as a photo, a video frame, streaming video, as well as any electronic, digital or mathematical analogue of a photo, video frame, or streaming video. Any apparatus described herein, in certain embodiments, includes a display for displaying an image or any other result produced by the processor. Any method described herein, in certain embodiments, includes a step of displaying an image or any other result produced via the method.
As used herein, “extracting” or “extraction” of bone axes refers to the detection, segmentation, calculation, visualization, and the like, of axes (e.g., central axes) of bones.
As used herein, “3-D” or “three-dimensional” with reference to an “image” means conveying information about three dimensions. A 3-D image may be rendered as a dataset in three dimensions and/or may be displayed as a set of two-dimensional representations, or as a three-dimensional representation.
As used herein, “long bone” means a bone of an extremity (e.g., of a mammal, e.g., mouse, rat, etc.) that has a length greater than the width (e.g., femur bone). In some embodiments, a long bone is a bone of the legs, the arms, the hands, the feet, the fingers, the toes, or collar bones. In some embodiments, a long bone is selected from the following: femora, tibiae, fibulae, humeri, radii, ulnae, metacarpals, metatarsals, phalanges, and clavicles (e.g., of collar bones). Certain embodiments described herein apply to either long bones or non-long bones, including, for example, short bones, flat bones, sesamoid bones, and irregular bones. In certain embodiments, non-long bones include bones with partially circular shapes, e.g., the pelvic girdle.
As used herein, a “mask” is a graphical pattern that identifies a 2-D or 3-D region and is used to control the elimination or retention of portions of an image or other graphical pattern.
Described herein are systems and methods for automated detection of bone central axes from in vivo or ex vivo images (e.g., 3-D images). In some example embodiments, the 3-D image is an in vivo image of an animal subject (e.g., a mammal such as a mouse). In some embodiments, the 3-D image is an ex vivo image of a sample (e.g., bone sample) from an animal subject (e.g., a mammal such as a mouse). In some embodiments, images can be acquired and/or processed by medical imaging devices such as CT scanners, microCT scanners, and the like. It should be understood that an image, such as a 3-D image, may be a single image or a set or series of multiple images.
Central axes of bones can effectively encapsulate characteristics and data of bones, including, for example, spatial features, direction, orientation, and shape of a bone. Using bone detection and segmentation algorithms, the skeletal bones of animal subjects (e.g., scanned by CT or microCT scanners) are detected, segmented, and visualized, as shown in
Some example embodiments described herein relate to calculating and extracting central axes of skeletal bones to capture both the general and localized tangential directions of the bones. The calculation of central axes also identifies, among other things, the shape, form, and curvature of the bones. That is, the central axis represents a medial path that describes, among other things, the main center-line, shape, and orientation of a bone.
Description and calculation of bone central axes is challenging because bones are not homogeneous solid objects with simple regular shapes; they can take arbitrary shapes and include hollow regions and holes with various densities and porosities. In addition, morphological skeletons of binary bone masks, which reduce 3D regions into line sets, may not be represented by a single-branched axis, but rather, they may include multiple branches, especially at the distal ends, confounding traditional analysis. Thus, in certain embodiments, mere 3D skeletonization of a binary bone mask, or even a filled bone mask, is found to be insufficient for extracting a central axis. As shown in
In certain embodiments, bone central axes are extracted from morphological skeletons. Skeletons are generally defined for filled 2-D or 3-D binary objects. In some embodiments, a skeleton may be represented and/or illustrated as a skeletal tree. In some embodiments, in filled 3-D objects, the morphological skeleton is extracted by performing iterative thinning on the 3-D object binary mask. The process of extracting the morphological skeleton is referred to as skeletonization. In some embodiments, skeletonization involves extracting the locus of the center of all maximally inscribed spheres in a 3-D object. Referring to
Results of direct 3-D skeletonization of bones are often not useful candidates for the central axis calculation and extraction due to two primary reasons. First, a skeletal bone is almost never a homogenously filled 3-D object, and its inner compartment (e.g., the trabeculae), is a porous structure distributed over the marrow. A direct 3-D skeletonization of the bone mask (without undergoing filling operations) would represent the medial tree spanning the cortical shell and trabecular network of the bone, rather than the morphological skeleton of the bone in its entirety including the cortical, trabecular, and marrow compartments. Second, because of peripheral branches (e.g., multi-branched segments extending into the conical sections 210, 212), the skeleton, in its raw form, is not a single-branched central axis that represents the orientation and form of a 3-D object. The peripheral branches of the 3-D skeleton only carry localized structural information especially at the distal ends and are not useful in guiding and automating slice-by-slice measurements.
In
After morphological isolation, the 3-D image of the bone(s) is converted to a binary bone mask [406]. In several embodiments, a 3-D binary bone mask is a three-dimensional array comprising voxels in an included (e.g., binary true) or excluded (e.g., binary false) state. A voxel in the binary true state in the mask corresponds to a region containing bone tissue in the 3-D image of the bone(s). Conversely, a voxel in a binary false state in the mask corresponds to an empty or non-bone tissue in the 3-D image. As such, in certain embodiments, the binary bone mask represents at least the cortical and trabecular compartments of the bone(s). In further embodiments, the binary bone mask is initially filled (e.g., the interior portion contents such as marrow) by binary true voxels (e.g., the binary bone mask represents a solid 3D bone volume composed of cortical, trabecular, and marrow compartments). An example technique for generating a binary bone mask is further described in detail in U.S. patent application Ser. No. 14/162,693 filed Jan. 23, 2014, which is incorporated herein by reference in its entirety.
In certain embodiments, the binary mask of the bone is filled by morphological processing [408], which is described in further detail in the flowchart depicted in
Referring to
Still with reference to
The borders of Image1 are then identified and filled [606]. Morphological holes (e.g., gaps and/or discontinuities) are also filled [608]. Border filling is described in more detail below with reference to
In certain embodiments, the result of the 3D hole filling and border filling operations is stored as Image2. Image1 is then subtracted from Image2 [610], resulting in a mask effectively representing the location of filled holes and cracks. In certain embodiments, small spots, defined as connected components with volumes smaller than empirically determined bounds, are removed from the resulting mask and the image is again dilated, and stored as Image3. In turn, a new image is generated by combining Image0 with Image3, resulting in the binary bone mask of Image0 superimposed with the filled holes represented by Image3. The borders of the resulting image (Image0+Image3) are filled and subsequently the holes of the 3D image are filled. The holes are identified as the empty or binary false voxels located in the internal compartment of the dilated bone mask (morphologically disconnected from the background image by the dilated bone mask). The holes are filled by being added to the bone mask (or their voxel values being updated to 1 or binary true).
More specifically, to initiate border filling, the binary bone mask used in the exemplary embodiment of
If all of the vertices have been checked and processed (e.g., completed) [816], the method proceeds to step 818. Otherwise, the method returns to select the next vertex in the data-cube, in step 804. With all vertices completed, a first data-cube edge is selected [818]. The edge is checked to ensure both of the faces connected to the edge and the edge itself contain binary true voxels [820]. If the edge check is passed (e.g., edge is associated with true voxels), a 2-D image is formed from concatenating the two faces connected to edge [822]. Otherwise the next data-cube edge is selected as in step 818. Morphological holes in the 2-D image are filled and the filled surfaces are mapped back to the two corresponding faces of the data cube [824]. If all edges have been completed [826], the method continues to step 828, otherwise the next edge is selected. After all edges are completed, the holes on each individual face of the data-cube are filled [830], thereby generating a border-filled bone mask. The border-filled bone mask is stored in memory [828].
As mentioned above, identifying a central bone axis also includes a step of 3-D thinning. The concept of 3-D thinning is described in more detail in Building Skeleton Models via 3-D medial Surface/Axis Thinning Algorithms (Lee, T. C., Graphical Models and Image Processing, Vol. 56, No. 6, November, pp. 462-478, 1994), which is incorporated herein by reference in its entirety.
Generally, 3-D thinning shrinks or reduces solid objects, areas, or volumes, such as a filled 3-D object (e.g., bone) to a morphological skeleton (see
More specifically, the iterative deletion of border points provides 3-D thinning while maintaining topological properties of the image being thinned. In some embodiments, each thinning iteration is divided into subcycles according to the type of border point (e.g., north, south, west, east, up, bottom). The border points that are deleted are restricted based on topological and geometric constraints, for example, to avoid undesired object separation or elimination in the image. In some implementations, medial surface thinning (MST) and/or medial axis thinning (MAT) may be used as geometric constraints of the border deletion process of a 3-D thinning operation. That is, MST is used to identify surface points that are not deleted during thinning. That is, medial surface thinning identifies surfaces that are approximately located to a center line. MAT differs from MST in that the extracted skeleton consists of arcs and/or curves instead of surfaces that approximate the center line. MST and MAT are both described in further detail in Building Skeleton Models via 3-D medial Surface/Axis Thinning Algorithms (Lee, T. C., Graphical Models and Image Processing, Vol. 56, No. 6, November, pp. 462-478, 1994), which is incorporated herein by reference in its entirety.
In the pruning step [1106], the minor branches of the skeleton are removed from the identified main path, thus reducing the skeleton to a single branch. To prevent the irregular shapes of the distal ends of bones from affecting the central axis determination, the pruned branch is further smoothed by point averaging [1108]. The end result of this step is the bone central axis [1110], an example of which is illustrated in
In some implementations, the identified central axis of the bone is used to quantify structural characteristics (e.g., features) of the bone, such as the bone's shape, form, localized tangential directions, curvature, and the like. These characteristics may be used, in turn, to characterize the bone, for example, by identifying abnormalities, identifying the bone as a specific bone, and the like. In some implementations, the identified central axis is used to render images of the bone or the set of bones with which the bone is associated. It should be understood that the characteristics of a bone, identified using the bone's central axis, can be used, for example, for other imaging (e.g., rendering), diagnostics, and therapeutic purposes, as well as for other applications.
For example, in certain embodiments, the identified central axis of the bone is used for stereological measurements and slice-by-slice studies of the bone.
As discussed above, certain embodiments of the procedures described herein relate to extracting the central axis of a bone to guide stereological measurements or capture, for example, of the direction, overall shape, and spatial characteristics of the bone. Various other embodiments utilize the image processing methods described herein, including procedures such as border filling, bone filling, and pruning/smoothing, for other applications. For example, the image processing methods described herein may be used in bone segmentation/separation using morphological separation approaches such as watershed. In some embodiments, the morphological separation is performed on filled bones rather than the original bone masks. Moreover, border and bone filling are also useful in segmenting the cortical and trabecular compartment of a bone, for example.
The cloud computing environment 1400 may include a resource manager 1406. The resource manager 1406 may be connected to the resource providers 1402 and the computing devices 1404 over the computer network 1408. In some implementations, the resource manager 1406 may facilitate the provision of computing resources by one or more resource providers 1402 to one or more computing devices 1404. The resource manager 1406 may receive a request for a computing resource from a particular computing device 1404. The resource manager 1406 may identify one or more resource providers 1402 capable of providing the computing resource requested by the computing device 1404. The resource manager 1406 may select a resource provider 1402 to provide the computing resource. The resource manager 1406 may facilitate a connection between the resource provider 1402 and a particular computing device 1404. In some implementations, the resource manager 1406 may establish a connection between a particular resource provider 1402 and a particular computing device 1404. In some implementations, the resource manager 1406 may redirect a particular computing device 1404 to a particular resource provider 1402 with the requested computing resource.
The computing device 1500 includes a processor 1502, a memory 1504, a storage device 1506, a high-speed interface 1508 connecting to the memory 1504 and multiple high-speed expansion ports 1510, and a low-speed interface 1512 connecting to a low-speed expansion port 1514 and the storage device 1506. Each of the processor 1502, the memory 1504, the storage device 1506, the high-speed interface 1508, the high-speed expansion ports 1510, and the low-speed interface 1512, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 1502 can process instructions for execution within the computing device 1500, including instructions stored in the memory 1504 or on the storage device 1506 to display graphical information for a GUI on an external input/output device, such as a display 1516 coupled to the high-speed interface 1508. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).
The memory 1504 stores information within the computing device 1500. In some implementations, the memory 1504 is a volatile memory unit or units. In some implementations, the memory 1504 is a non-volatile memory unit or units. The memory 1504 may also be another form of computer-readable medium, such as a magnetic or optical disk.
The storage device 1506 is capable of providing mass storage for the computing device 1500. In some implementations, the storage device 1506 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 1502), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine-readable mediums (for example, the memory 1504, the storage device 1506, or memory on the processor 1502).
The high-speed interface 1508 manages bandwidth-intensive operations for the computing device 1500, while the low-speed interface 1512 manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface 1508 is coupled to the memory 1504, the display 1516 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 1510, which may accept various expansion cards (not shown). In the implementation, the low-speed interface 1512 is coupled to the storage device 1506 and the low-speed expansion port 1514. The low-speed expansion port 1514, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.
The computing device 1500 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 1520, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer 1522. It may also be implemented as part of a rack server system 1524. Alternatively, components from the computing device 1500 may be combined with other components in a mobile device (not shown), such as a mobile computing device 1550. Each of such devices may contain one or more of the computing device 1500 and the mobile computing device 1550, and an entire system may be made up of multiple computing devices communicating with each other.
The mobile computing device 1550 includes a processor 1552, a memory 1564, an input/output device such as a display 1554, a communication interface 1566, and a transceiver 1568, among other components. The mobile computing device 1550 may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 1552, the memory 1564, the display 1554, the communication interface 1566, and the transceiver 1568, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.
The processor 1552 can execute instructions within the mobile computing device 1550, including instructions stored in the memory 1564. The processor 1552 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 1552 may provide, for example, for coordination of the other components of the mobile computing device 1550, such as control of user interfaces, applications run by the mobile computing device 1550, and wireless communication by the mobile computing device 1550.
The processor 1552 may communicate with a user through a control interface 1558 and a display interface 1556 coupled to the display 1554. The display 1554 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 1556 may comprise appropriate circuitry for driving the display 1554 to present graphical and other information to a user. The control interface 1558 may receive commands from a user and convert them for submission to the processor 1552. In addition, an external interface 1562 may provide communication with the processor 1552, so as to enable near area communication of the mobile computing device 1550 with other devices. The external interface 1562 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.
The memory 1564 stores information within the mobile computing device 1550. The memory 1564 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 1574 may also be provided and connected to the mobile computing device 1550 through an expansion interface 1572, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 1574 may provide extra storage space for the mobile computing device 1550, or may also store applications or other information for the mobile computing device 1550. Specifically, the expansion memory 1574 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory 1574 may be provided as a security module for the mobile computing device 1550, and may be programmed with instructions that permit secure use of the mobile computing device 1550. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.
The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier and, when executed by one or more processing devices (for example, processor 1552), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory 1564, the expansion memory 1574, or memory on the processor 1552). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver 1568 or the external interface 1562.
The mobile computing device 1550 may communicate wirelessly through the communication interface 1566, which may include digital signal processing circuitry where necessary. The communication interface 1566 may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver 1568 using a radio-frequency. In addition, short-range communication may occur, such as using a Bluetooth®, Wi-Fi™, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 1570 may provide additional navigation- and location-related wireless data to the mobile computing device 1550, which may be used as appropriate by applications running on the mobile computing device 1550.
The mobile computing device 1550 may also communicate audibly using an audio codec 1560, which may receive spoken information from a user and convert it to usable digital information. The audio codec 1560 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 1550. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on the mobile computing device 1550.
The mobile computing device 1550 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 1580. It may also be implemented as part of a smart-phone 1582, personal digital assistant, or other similar mobile device.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
While the invention has been particularly shown and described with reference to specific preferred embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7539332 | Al-Dayeh et al. | May 2009 | B1 |
| 8306305 | Porat | Nov 2012 | B2 |
| 9192348 | Ollilainen et al. | Nov 2015 | B2 |
| 20050163358 | Moeller | Jul 2005 | A1 |
| 20070031019 | Lesage et al. | Feb 2007 | A1 |
| 20080107318 | Kiraly | May 2008 | A1 |
| 20080273777 | Luboz | Nov 2008 | A1 |
| 20100128954 | Ostrovsky-Berman et al. | May 2010 | A1 |
| 20120143037 | Najarian et al. | Jun 2012 | A1 |
| 20120224758 | Treece et al. | Sep 2012 | A1 |
| 20130039592 | Lang et al. | Feb 2013 | A1 |
| 20130163836 | Pau et al. | Jun 2013 | A1 |
| 20130272594 | Zelzer et al. | Oct 2013 | A1 |
| 20160038124 | Tsujita | Feb 2016 | A1 |
| 20160242852 | Yosibash | Aug 2016 | A1 |
| 20170032518 | Behrooz et al. | Feb 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 2194505 | Mar 2015 | EP |
| 2006-102353 | Apr 2006 | JP |
| 2008-093229 | Apr 2008 | JP |
| 2013-172792 | Sep 2013 | JP |
| 2015-066311 | Apr 2015 | JP |
| 2009101560 | Aug 2009 | WO |
| Entry |
|---|
| Aug. 31, 2020—(CA) Notice of Allowance—App No. 3,014,608. |
| Mar. 3, 2020—(JP) Office Action—App 2018-549919. |
| Nov. 10, 2020—(IN) First Examination Report—Application No. 201847035537. |
| Jan. 23, 2020—(EP) Office Action—App No. 16715704.9. |
| Jun. 22, 2020—(JP) Notice of Allowance—App 2018-549919. |
| Jan. 11, 2017—(WO) International Search Report—App PCT/US2016/024372. |
| Staal, J. et al., “Automatic Rib Segmentation and Labeling in Computer Tomography Scans Using a General Framework for Detection, Recognition and Segmentation of Objects in Volumetric Data” Medical Image Analysis 11:35-46 (2007). |
| Staal, J. et al. “Automatice Rib Segmentation in CT Data” CVAMIA-MMBIA, LNCS 3117:193-204 (2004). |
| Sun, S. et al. “Automated 3-D Segmentation of Lungs With Lung Cancer in CT Data Using a Novel Robust Active Shape Model Approach” IEEE Transactions on Medial Imaging, 31(2):449-460 (2012). |
| Wang, H. et al. “Estimation of Mouse Organ Locations Through Registration of a Statistical Mouse Atlas With Micro-CT Images” 31(1):88-102 (2012). |
| Wildeman, M. H. et al., “2D/3D Registration of Micro-CT Data to Multi-View Photographs Based on a 3D Distance Map” Biomedical Imaging, IEEE International Sympodium ON, 987-990 (2009). |
| Wlodarczyk, J. et al., “Segmentation of bones in magnetic resonance images of the wrist” International Journal of Computer Assisted Radiology and Surgery, 10(4):419-431 (2014). |
| Wu, D. et al., “A Learning Based Deformable Template Matching Method for Automatic Rib Centerline Extraction and Labeling in CT Images” IEEE, 980-987 (2012). |
| Yin, Y. et al. “Hierarchical Decision Framework with Priori Shape Models for Knee Joint Cartilage Segmentation” MICCAI Grand Challenge, Depts. of Electrical & Computer Engineering and Orthopaedics & Rehabilitation, University of Iowa, pp. 241-250 (2010). |
| AnalyzeDirect, “Analyze 12.0 Bone Microarchitecture Analysis Manual” AnalyzeDirect, Inc. and BIR, Mayo Clinic, 56 Pages, 1999-2014. |
| Computer Vision Demonstration Website, Electronics and Computer Science University of Southampton, Standard and Hysteresis Thresholding, 2 pages (2005) [retrieved May 3, 2017 <http://users.ecs.saotan.ac.uk/msn/book/new_demo/thresholding/>. |
| Eddins, Steve “The Watershed Transform: Strategies for Image Segmentation” MathWorks, 8 pages (2002) [retrieved May 3, 2017 <https:www.mathworks.com/company/newsletters/articles/the-watershed-transform-strategies-for-image-segmentation.html>. |
| May. 15, 2017—(WO) International Search Report and Written Opinion—PCT/US2016/024372. |
| Mavrogenis, A.F. et al. “Heterotopic Ossification Revisited, Orthoedics” 34:(3)177- (2011). |
| Otsu, Nobuyuki “A Threshold Selection Method from Gay-Level Histograms” IEEE Transactions on Systems, Man and Cybernetics, SMC-9(1):62-66 (1979). |
| Smolka, Jakub “Watershed based region growing algorithm” Annales UMCS Informatica AI, 3:169-178 (2005). |
| Waarsing, J. H. et al. “An Improved Segmentation Method for in Vivo CT Imaging” Journal of Bone and Mineral Research, 19:1640-1650 (2004). |
| Wikipedia “Canny Edge Detector” 9 pages (2017) [retrieved May 3, 2017 <https://en.wikipedia.org/wiki/Canny_edge_detector>]. |
| Li, Q et al., “Selective enhancement filters for nodules, vessels, and airway walls in two and three-dimensional CT scans” Med. Phys. 30(8):2040-2051 (2003). |
| Sato, Y. et al. “3D Multi-Scale Line Filter for Segmentation and Visualization of Curvilinear Sructures in Medical Images” Medical Image Analysis 2(2)L143-168 (1998). |
| Apr. 2, 2018—U.S. Office Action—U.S. Appl. No. 16/155,943. |
| May 16, 2019—(RU) Office Action—App 2018136627. |
| May 31, 2019—(CA) Office Action—App 3,014,608. |
| Aug. 21, 2019—(JP) Office Action—App 2018-549919. |
| Aug. 15, 2019—(RU) Office Action—App 2018136627. |
| Tao, Li, “Three-Dimensional Reconstruction of Medical CT Image,” China Outstanding Master's Thesis Database Information Technology, No. 12, Dec. 15, 2006. |
| Number | Date | Country | |
|---|---|---|---|
| 20200178919 A1 | Jun 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15081788 | Mar 2016 | US |
| Child | 16155943 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16155943 | Oct 2018 | US |
| Child | 16724722 | US |
