Patent Grant
11200443
The present invention relates to an image processing apparatus, an image processing method, and an image processing system.
Recently, there has been growing interest in technology for computer aided diagnosis (CAD) using information obtained by analyzing medical images using computers to assist diagnosis. In such technology, to analyze features of lesions or judge the benignity or malignancy of the lesions, accuracy for identifying and extracting regions to be analyzed is important. Some regions to be analyzed are located near other structures such as organs, blood vessels, or lesions. PTL 1 discloses that when a region to be analyzed and any other region, which are joined to each other, are extracted as a single structure, the regions of the structure are reduced to separate the regions from each other.
PTL 1 Japanese Patent Laid-Open No. 2007-312892
In some cases, a region to be analyzed may deform in shape when located near any other structure. Doctors may make diagnoses by observing regions less affected by the deformation. In CAD, regions less affected by the deformation may also be analyzed. Merely separating regions extracted as a single structure into a plurality of structures in accordance with a process for reducing the regions may be insufficient to make analysis taking into account that the structures are located near each other.
An image processing apparatus according to an embodiment of the present invention includes extraction means for extracting a first region and a second region different from the first region from a medical image, identification means for identifying a third region, the third region being included in the second region and being at a distance greater than or equal to a threshold from the first region, and acquisition means for acquiring a feature value on the basis of the third region, the feature value being a value indicating a feature of the second region.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
The following describes embodiments of the present invention with reference to the drawings.
In recent years, an increasing number of medical images have been utilized for diagnosis and applications of the medical images to CAD have been expected to be available. For example, to analyze a suspected lesion region presented on a medical image for CAD, information on the shape or signal value (pixel value) of the suspected lesion region is used. For example, in the judgment of a pulmonary nodule that can be lung cancer, information on the shape of the pulmonary nodule is used as an index. A malignant pulmonary nodule is known to be likely to take a lobulated shape, and a benign pulmonary nodule is known to be likely to take a round shape.
However, lesions may occur near peripheral structures such as peripheral organs or other lesions. In this case, a lesion may greatly differ in shape depending on whether the lesion is attached to a peripheral structure (attached region) or not (unattached region). That is, a portion of a lesion, which is located near any other structure, may deform in shape. For example, the pleura may be indented towards a nearby pulmonary nodule, which causes a deformation such as a pleural invagination. In terms of features such as density in pixel value, the boundary between a lesion region and a nearby structure may be unclear.
When a doctor makes diagnosis while observing a medical image, the observation may be mainly focused on an unattached region, which is considered to be less affected by the positioning of a lesion near any other structure. Also in CAD, therefore, when the lesion to be analyzed is located near any other structure, it may be preferable to perform analysis taking the influence of the positioning of the lesion near any other structure into account. An image processing apparatus 100 according to a first embodiment is intended to identify an attached region where the region to be analyzed is likely to be attached to any other region and an unattached region where the region to be analyzed is considered not to be attached to any other region in CAD, and to acquire a feature value while taking into account that the region to be analyzed is located near any other region. Even when a pulmonary nodule is not attached to the pleura, the pleura may be indented towards the pulmonary nodule if the pulmonary nodule is located near the pleura. In the following, a region affected by a lesion located near a peripheral structure even though not necessarily in contact with the peripheral structure, such as a pleural indentation region, is referred to as an attached region. The image processing apparatus 100 appropriately acquires a feature value such as the shape of the region to be analyzed. Accordingly, more accurate CAD can be performed.
In the following, a computed tomography (CT) image of the chest of a subject is used as an example of a medical image to be used for CAD, and an example will be described in which a pulmonary nodule near the pleura is to be analyzed. The image processing apparatus 100 identifies a boundary region (attached region) where a pulmonary nodule (lesion region) near the pleura is in contact with the pleura from the CT image to be used for CAD on the basis of the distance between the pulmonary nodule and the pleura. Then, the image processing apparatus 100 acquires the feature value of the lesion region on the basis of an unattached region, which is different from the attached region.
Configuration of Image Processing Apparatus 100
The image processing apparatus 100 is, for example, a computer and performs processing according to an embodiment of the present invention. The image processing apparatus 100 includes a central processing unit (CPU) 11, a main memory 12, a magnetic disk 13, and a display memory 14. The internal components of the image processing apparatus 100 are connected to one another via a common bus 18.
The CPU 11 mainly controls the operation of each component. The main memory 12 stores a control program to be executed by the CPU 11 or provides a work area when the CPU 11 executes a program. Examples of the main memory 12 include a random access memory (RAM) and a read only memory (ROM). The magnetic disk 13 stores an operating system (OS), device drivers for peripherals, and programs for implementing various software applications including a program for performing a process and the like described below. Examples of the magnetic disk 13 include a hard disk drive (HDD) and a solid state drive (SSD). The CPU 11 executes a program stored in the main memory 12 or the magnetic disk 13 to implement functions (software) of the image processing apparatus 100 illustrated in
Examples of the display unit 160 include a cathode-ray tube (CRT) monitor and a liquid crystal monitor. The display unit 160 displays an image, text, and so on based on data from the display memory 14. A mouse 170 and a keyboard 180 are operated by a user for pointing input and for character input or the like, respectively. The display unit 160 may be a touch panel monitor that accepts operation input. In this case, the mouse 170 and the keyboard 180 may be integrated into a function of the touch panel monitor. Alternatively, the image processing apparatus 100 may be connected to any other operation unit, such as a stylus pen, together with the touch panel monitor. In the following, a component configured to input information to the image processing apparatus 100, such as the mouse 170 or the keyboard 180, is referred to sometimes as an operation unit.
The CPU 11 is an example of a processor. The image processing apparatus 100 may include a plurality of processors. The image processing apparatus 100 may include a graphics processing unit (GPU) dedicated to image processing. The image processing apparatus 100 may include a field-programmable gate array (FPGA) for programming some or all of the processes described below, or include an application specific integrated circuit (ASIC) for a specific process. The main memory 12, the magnetic disk 13, and the display memory 14 are examples of a memory, and the image processing apparatus 100 may include a plurality of memories.
The system 190 includes the image processing apparatus 100 and a data server 120. In the first embodiment, the data server 120 is connected to a modality 110.
The modality 110 is a device for acquiring a medical image. Examples of the modality 110 include a CT device, a nuclear magnetic resonance imaging (MRI) device, and a digital radiography (DR) device for capturing a two-dimensional radiographic image. The modality 110 transmits an acquired medical image to the data server 120. A control unit (not illustrated) that controls the modality 110 may be included in the image processing apparatus 100.
The data server 120 is a device that stores medical images. Examples of the data server 120 include a picture archiving and communication system (PACS). The image processing apparatus 100 acquires a medical image from the data server 120 via a network such as a local area network (LAN).
The image acquisition unit 1000 acquires a medical image from the data server 120. The image acquisition unit 1000 may acquire a medical image designated by a user or acquire a medical image under a predetermined condition. The image acquisition unit 1000 outputs an input image to the extraction unit 1010 and the display control unit 1040.
The extraction unit 1010 extracts a specific region from the medical image (hereinafter referred to as the input image) acquired by the image acquisition unit 1000 for analysis in CAD. In the first embodiment, the extraction unit 1010 includes a first extraction unit 1011 and a second extraction unit 1012.
The first extraction unit 1011 extracts a region of the anatomical structure of the subject from the input image. In the example of the chest CT image, the first extraction unit 1011 extracts a region of structures such as lung fields, the pleura, bronchi, and blood vessels. The region extracted by the first extraction unit 1011 is hereinafter referred to sometimes as a first region. A region of the anatomical structure is an example of the first region. The first extraction unit 1011 outputs information on the extracted region to the second extraction unit 1012, the identification unit 1020, and the display control unit 1040.
The second extraction unit 1012 extracts a lesion region to be analyzed from the input image. In the example of the chest CT image, the second extraction unit 1012 extracts a pulmonary nodule region. When a plurality of lesion regions are extracted, the second extraction unit 1012 may select one lesion region of interest. The following describes an example in which a single lesion region of interest is extracted by the second extraction unit 1012. The region extracted by the second extraction unit 1012 is hereinafter referred sometimes as a second region. A lesion region is an example of the second region. The second extraction unit 1012 outputs information on the extracted region to the identification unit 1020 and the display control unit 1040.
The identification unit 1020 identifies a region where the lesion region is located near any other structure, that is, an attached region, on the basis of the processing results obtained by the first extraction unit 1011 and the second extraction unit 1012. In the first embodiment, the identification unit 1020 identifies an attached region within the pulmonary nodule region extracted by the second extraction unit 1012 on the basis of the distance from a pleural region extracted by the first extraction unit 1011. That is, the identification unit 1020 identifies a sub-region that is included in the lesion region as an attached region. In the following, the attached region identified by the identification unit 1020 is referred to sometimes as a fourth region. Further, the identification unit 1020 identifies a region different from the attached region as an unattached region. In the first embodiment, the identification unit 1020 identifies a region included in the lesion region and not including the attached region as an unattached region. The unattached region is not limited to that in the example described above and may be any region that is different from the attached region and that is less affected by the positioning of the lesion region near any other region during analysis. In the following, the unattached region identified by the identification unit 1020 is referred to sometimes as a third region. The identification unit 1020 outputs information on the identified regions to the feature value acquisition unit 1030 and the display control unit 1040.
The feature value acquisition unit 1030 acquires a feature value, which is a value indicating a feature of the lesion region extracted by the second extraction unit 1012. The acquired feature value may be represented as a value such as the mean or variance of density (luminance) of the region to be analyzed in the input image or a value based on filter output. The feature value may be represented as words representing a feature of the lesion region, or so-called findings, or may be represented as words converted based on the value described above. The feature value acquisition unit 1030 designates the lesion region extracted by the second extraction unit 1012 and the attached region and unattached region identified by the identification unit 1020 as analytical targets, and acquires a feature value. To acquire a feature value for the shape of the lesion region, such as circularity, among feature values, it is desirable to designate the unattached region identified by the identification unit 1020 as the analytical target. The feature value acquisition unit 1030 outputs information on the acquired feature value to the display control unit 1040. The following describes an example of acquiring a feature value for the shape of the lesion region by using the unattached region as the analytical target.
The display control unit 1040 controls the output of display data to be displayed on the display unit 160. In the first embodiment, the display control unit 1040 outputs display data for displaying on the display unit 160 pieces of information, namely, the input image acquired by the image acquisition unit 1000, information indicating the region extracted by the extraction unit 1010, the information identified by the identification unit 1020, and the information acquired by the feature value acquisition unit 1030, as appropriate, to the display unit 160.
The functional components of the image processing apparatus 100, described with reference to
Regarding Series of Processes
In step S1100, the image acquisition unit 1000 acquires an input image from the data server 120. In another example, the image acquisition unit 1000 directly acquires image data captured by the modality 110 via a communication means and acquires a medical image subjected to image processing or the like to obtain an image suitable for diagnosis as an input image according to this embodiment. For example, when the modality 110 is a CT device, a CT image having a pixel value based on a signal value, called a Hounsfield unit (HU) value, is acquired from the modality 110 as an input image.
The input image is constituted by a plurality of pixels whose positions can be each identified using three-dimensional Cartesian coordinates (x, y, z). The pixel size, which is one of the image attributes, is defined for each of three axis coordinate directions. The first embodiment describes an example in which the pixel sizes in the x, y, z directions are represented as r_size_x, r_size_y, and r_size_z, respectively, and all the values are 1.0 mm. The pixel values of pixels constituting the input image are defined for each value of three-dimensional coordinates (x, y, z). Accordingly, the input image can be regarded as data defined by a function I(x, y, z) that uses three-dimensional coordinate values as an argument.
In step S1110, the first extraction unit 1011 extracts a first region from an input image I(x, y, z). The first region is, for example, a region of the anatomical structure, such as lung fields on a chest CT image and the pleura adjacent to the lung fields. The first extraction unit 1011 extracts a lung field region, which is a region of the anatomical structure of the subject, from the input image I(x, y, z). The lung field region includes regions such as an air (alveolar) region, some of the bronchi, some pulmonary vessels, and pulmonary nodules.
The first extraction unit 1011 performs a process, such as a threshold process, a process using a filter that enhances a specific shape, a process using the graph-cut method, a process using the level-set method, or a process using the organ atlas (model) based on anatomy, on the input image I(x, y, z) to extract a region of the anatomical structure. Here, a technique using the graph-cut method will be described briefly, by way of example.
In the graph-cut method, first, a graph for the input image I(x, y, z) is generated. The graph is constituted by nodes (image nodes), each of which corresponds to one of the pixels of the input image, a node representing a lung field region (terminal node F), a node representing a non-lung-field region (terminal node B), edges (n-link) each connecting adjacent image nodes, and edges (t-link) each connecting one of the image nodes and the two terminal nodes. Then, values each representing similarity of image density values (e.g., the reciprocal of the difference between the image density values) are calculated for all combinations of adjacent pixels. Then, the calculated values are each assigned to the edge (n-link) connecting the image nodes corresponding to the associated adjacent pixels as a weight. Further, for each pixel, a value representing a similarity between the pixel value of the pixel and the pixel value of a typical pixel in the lung field region (e.g., the reciprocal of the difference between image density values) is calculated. Then, the calculated value is assigned to the edge (t-link) connecting the image node corresponding to the pixel and the terminal node F as a weight. Likewise, for each pixel, a value representing a similarity between the pixel value of the pixel and the pixel value of a typical pixel in the non-lung-field region (e.g., the difference between image density values) is calculated and assigned to the edge (t-link) connecting the image node corresponding to the pixel and the terminal node B as a weight.
After the generation of a graph, the graph is divided into two subgraphs by using a graph segmentation algorithm (e.g., the Ford-Fulkerson algorithm). The graph is divided such that one of the subgraphs includes at least the terminal node F, and the other subgraph includes the terminal node B. As a result of the segmentation of the graph under the constraints described above, the graph is divided into two subgraphs by an edge (n-link) created between an image node corresponding to a “pixel likely to belong to the lung field region” and an image node corresponding to a “pixel likely to belong to the non-lung-field region”. In this case, the subgraph (subgraph GF) including the terminal node F includes the image node corresponding to the “pixel likely to belong to the lung field region”. The subgraph (subgraph GB) including the terminal node B includes the image node corresponding to the “pixel likely to belong to the non-lung-field region”. The pixels corresponding to the pixel nodes included in the subgraph GF are referred to as pixels belonging to the lung field region. The pixels corresponding to the pixel nodes included in the subgraph GB are referred to as pixels belonging to the non-lung-field region. In this way, an anatomical structure region (a lung field region) Vlung is acquired.
In the graph creation process described above, if a pixel that certainly belongs to the lung field region is known in advance, the weight of the edge (t-link) created between the image node corresponding to the pixel and the terminal node F may be set infinite. Likewise, if a pixel that certainly belongs to the non-lung-field region is known in advance, the weight of the edge (t-link) created between the image node corresponding to the pixel and the terminal node B may be set infinite.
When an anatomical structure region (a lung field region) is extracted by using the techniques described above, the extraction of some regions, such as some of the bronchi, some pulmonary vessels, or pulmonary nodules, may fail. To address this, after the extraction of an anatomical structure region (a lung field region), a process for integrating the missing element regions into the anatomical structure region is performed. For example, the anatomical structure region (the lung field region) may be subjected to closing, which is one of the morphological operations. Through this process, regions such as some of the bronchi, some pulmonary vessels, and pulmonary nodules, which are present in the anatomical structure region (the lung field region), are included in the anatomical structure region (the lung field region).
The first extraction unit 1011 may extract, as a first region, an area designated by the user on an input image displayed on the display unit 160 by using the mouse 170 or the like. In the following, an extracted lung field region is represented as a lung field region Vlung.
The first extraction unit 1011 further extracts a pleural region. In the first embodiment, the first extraction unit 1011 extracts a pleural region after the acquisition of the lung field region Vlung. In the following, the extracted pleural region is represented as a pleural region Vpleural.
The first extraction unit 1011 acquires, as the pleural region Vpleural, for example, a region that is located outside the lung field region Vlung and that is at a distance less than or equal to a predetermined distance from the edge of the lung field region Vlung. In another example, the first extraction unit 1011 may enlarge the lung field region Vlung with a predetermined magnification and acquire, as the pleural region Vpleural, a region corresponding to the difference between the enlarged lung field region and the original lung field region before enlargement.
The first extraction unit 1011 may further extract regions of other anatomical structures presented on the chest CT image, such as a pulmonary vessel region and a bronchus region, in addition to the lung field region Vlung and the pleural region Vpleural. The first extraction unit 1011 may extract a region of an anatomical structure in accordance with CAD analysis or extract regions of all extractable structures among the anatomical structures presented on the input image I(x, y, z).
In step S1120, the second extraction unit 1012 extracts a second region from the input image I(x, y, z). The second region is a lesion region, for example, a pulmonary nodule.
The second extraction unit 1012 extracts a region of a lesion on the basis of information such as a characteristic signal value (pixel value) or shape of the target lesion. Specifically, the second extraction unit 1012 performs a process, such as a threshold process, a process using a filter that enhances a specific shape, a process using the graph-cut method, a process using the level-set method, or a process using the organ atlas (model) based on anatomy, on the input image I(x, y, z) to extract a region of the anatomical structure. The following describes the extraction of a pulmonary nodule region, by way of example.
The second extraction unit 1012 extracts a pulmonary nodule region from the lung field region Vlung extracted in step S1110. First, the second extraction unit 1012 performs a threshold process or a process using a filter, such as a Hessian matrix, on the input image I(x, y, z) to detect a pulmonary nodule region from the lung field region Vlung. Then, the second extraction unit 1012 performs a process using the graph-cut method or a process using the level-set method on the detected pulmonary nodule region to extract the pulmonary nodule region.
Further, the second extraction unit 1012 may prompt the user to designate a target pulmonary nodule. In this case, the user designates coordinates belonging to the target pulmonary nodule by using the mouse 170 while referring to the input image displayed on the display unit 160. The second extraction unit 1012 acquires the designated coordinates as the coordinates of a seed point of the pulmonary nodule. The second extraction unit 1012 extracts the pulmonary nodule region by using information on the seed point in addition to the technique described above. Accordingly, the second extraction unit 1012 extracts a pulmonary nodule region including the coordinates designated by the user. As another example, the second extraction unit 1012 may perform a process on a volume of interest (VOI) image including a target pulmonary nodule region to extract the pulmonary nodule region.
In the following, a pulmonary nodule of interest to be analyzed is referred to as Cnodule. In the following, furthermore, the pulmonary nodule region extracted by the second extraction unit 1012 in step S1120 is referred to as a pulmonary nodule region Vnodule.
Pulmonary nodules may occur at various sites in the lungs. In the first embodiment, an example process will be described for acquiring a feature value, taking into account the influence of the attachment of the pulmonary nodule extracted in step S1120 to other structures.
In step S1130, the identification unit 1020 identifies a fourth region. The fourth region is an attached region that is included in the second region acquired in step S1120 and that is likely to be attached to other regions.
Referring to
As a result of the process for extracting the pulmonary nodule 705 illustrated in
A doctor who observes the VOI image 600, the VOI image 700, and medical images including the VOI image 600 and the VOI image 700 may consider that an attached region where a pulmonary nodule, when observed, is attached to any other structure, or the fourth region, is not the true pulmonary nodule region, and thus may not take the attached region into account during observation. The following describes an example of identifying the fourth region to take into account during analysis a region that is likely to deform due to being located near any other structure.
In step S1130, the identification unit 1020 identifies a fourth region. Specifically, the identification unit 1020 identifies an attached region, which is a region where the pulmonary nodule region Vnodule is located near the pleural region Vpleural, on the basis of the lung field region Vlung and the pleural region Vpleural extracted in step S1110 and the pulmonary nodule region Vnodule extracted in step S1120.
First, the identification unit 1020 extracts a boundary region Snodule constituted by boundary pixels in the pulmonary nodule region Vnodule. Specifically, the identification unit 1020 determines, for each of the pixels belonging to the pulmonary nodule region Vnodule, whether any of the pixels (adjacent pixels) adjacent to the pixel “does not belong to the pulmonary nodule region Vnodule, but belongs to the lung field region Vlung”. Then, if at least one adjacent pixel does not belong to the pulmonary nodule region Vnodule, but belongs to the lung field region Vlung, the at least one adjacent pixel is determined to be a pixel belonging to the boundary region Snodule. In another example, the boundary region Snodule is extracted by, for example, performing a process using a differential filter or the like on the pulmonary nodule region Vnodule.
The identification unit 1020 repeatedly performs a region growing process on each of the pixels belonging to the boundary region Snodule. The identification unit 1020 repeatedly performs the region growing process until each pixel reaches the pleural region Vpleural, and acquires the number of repetitions of the region growing process as the shortest path length from the pixel to the pleural region Vpleural. That is, the identification unit 1020 acquires information concerning the distance between the second region and the first region on the basis of a process for searching for a pixel that is similar to a pixel in the boundary of the second region and a non-second region and that is not included in the second region or the first region. The region growing process is an example of the process described above, and the shortest path length is an example of the information concerning the distance. The following describes a process for acquiring a shortest path length in detail.
The identification unit 1020 sets the number of repetitions N of region growing of a specific pixel (hereinafter referred to as a pixel of interest) Pnodule-i belonging to the boundary region Snodule to 0. The identification unit 1020 searches pixels within a neighboring region of the pixel of interest Pnodule-i for a pixel satisfying region growing conditions as a growing pixel. That is, the pixel of interest Pnodule-i is an example of a seed point of the region growing process. The number of repetitions N increases by 1 each time a search is performed. Then, the identification unit 1020 repeatedly performs the process described above on the found growing pixel until a termination condition is satisfied. The neighboring region described above refers to a region including pixels adjacent to the pixel of interest. In the region growing process, a neighboring region typically includes 4 or 8 pixels adjacent to the pixel of interest for a two-dimensional image. For a three-dimensional image, a neighboring region includes 6 or 26 pixels adjacent to the pixel of interest. The configuration of the neighboring region may be determined in accordance with the result of the region growing process.
The region growing conditions are (1) the growing pixel does not belong to the extracted pulmonary nodule 715, and (2) the density value falls within a predetermined range [T1, T2]. The values T1 and T2 are constants indicating the upper and lower limits of the density value that a pixel belonging to the non-extracted region may take, respectively. These values are determined from a medical image (learning image) similar to the input image. Specifically, first, the processing of steps S1110 and S1120 is actually applied to a plurality of learning images to acquire a non-extracted region that occurs between the first region and the second region. Then, the region growing process in step S1130 is performed. At this time, the region growing process is performed using a plurality of combinations of T1 and T2, and a combination of T1 and T2 is selected with which the most exact number of pixels belonging to the non-extracted region can be extracted by using the region growing process. The combination of T1 and T2 obtained in the way described above is used as the values T1 and T2 for the image processing apparatus 100. The values T1 and T2 may be set by the user.
The termination condition of the region growing process is that (1) a pixel belonging to the pleural region Vpleural is present in the neighboring region of the pixel of interest Pnodule-i or that (2) a pixel belonging to the pleural region Vpleural is not present in the neighboring region of the pixel of interest Pnodule-i nor is present a pixel satisfying the region growing conditions. When the termination condition (1) is satisfied, the identification unit 1020 finishes the region growing process and uses the number of repetitions N as the shortest path length of the pixel of interest. When the termination condition (2) is satisfied, the shortest path length is set to −1. Additionally, a termination condition that (3) the number of repetitions of region growing exceeds a preset value Titeration may be added as a parallel condition. When the termination condition (3) is satisfied, the value Titeration may be used as the shortest path length.
The identification unit 1020 performs the region growing process described above on each of the pixels belonging to the boundary region Snodule, and acquires the shortest path length from the pixel to the pleural region Vpleural. In the example illustrated in
The identification unit 1020 acquires an attached region Sattached on the basis of the shortest path lengths for the respective pixels within the boundary region Snodule. That is, the identification unit 1020 identifies a fourth region, which is included in the second region and is located near the first region, on the basis of the distance from the first region. The identification unit 1020 further acquires an unattached region on the basis of the shortest path lengths for the respective pixels within the boundary pixel Snodule. That is, the identification unit 1020 identifies a third region different from the fourth region. In the first embodiment, the identification unit 1020 identifies a pixel having a shortest path length of −1 as an unattached region, that is, as a third region. The identification unit 1020 further identifies an attached region, that is, a fourth region, from among pixels each having a shortest path length of 0 or more.
The boundary region Snodule of the pulmonary nodule is sometimes in contact with, in addition to a pleural indentation region, a region having similar density values, such as a noise or blood vessel region. In this case, in the region growing process described above, a pixel within the boundary region Snodule is likely to reach the pleural region Vpleural via a noise or blood vessel region. In general, the distance to the pleural region Vpleural via a region such as blood vessels is longer than the distance to the pleural region Vpleural via a pleural indentation region. Accordingly, for example, setting a threshold [0, Tattached] for the shortest path length of each of the pixels within the boundary region Snodule makes a pixel reaching the pleural region Vpleural via a pleural indentation region and a pixel reaching the pleural region Vpleural via any other region, such as blood vessels, distinguishable from each other. The value Tattached may be set by the user or may be determined based on the learning image.
Depending on the accuracy of extraction of a pulmonary nodule region or the value of the threshold Tattached, the attached region Sattached acquired using the threshold process described above may include a missing element region (a so-called hole). Further, the acquired attached region Sattached may not be one concatenated region (single concatenated region), but may be divided into a plurality of regions. Accordingly, the identification unit 1020 may further perform a process for filling the hole generated in the attached region Sattached or a process for integrating the plurality of separate attached regions Sattached into a single concatenated region (hereinafter referred to as a connection process).
The connection process of the attached regions Sattached is performed using, for example, a morphological operation. The identification unit 1020 dilates the attached regions Sattached by using the morphological operation. Then, the identification unit 1020 acquires a region, which is obtained as a result of logical ANDs of the dilated attached regions obtained by morphological operation and the boundary region Snodule, as a corrected attached region. The logical ANDs of the dilated attached regions and the boundary region Snodule are determined in order to limit an attached region obtained in the connection process within the boundary region Snodule.
A hole generated in the attached region Sattached can be filled by dilating the attached region Sattached by morphological operation. A plurality of separate regions are dilated by morphological operation to bring each of the regions into contact with a neighboring region. Consequently, the plurality of regions can be integrated into a single concatenated region.
In step S1140, the feature value acquisition unit 1030 acquires a feature value, which is a value indicating a feature of the second region to be analyzed. Here, the acquisition of a feature value for the shape of a lesion (hereinafter referred to as a shape feature value), which is an example feature value, will be described, by way of example. Any type of feature value indicating a feature, other than a shape feature value, examples of which include a density value and the presence of a pleural indentation, may be acquired by the feature value acquisition unit 1030 in accordance with a known technique.
The feature value acquisition unit 1030 acquires a shape feature value of the pulmonary nodule region Vnodule on the basis of the lung field region Vlung and the pleural region Vpleural extracted in step S1110, the pulmonary nodule region Vnodule extracted in step 1120, and the attached region Sattached acquired in step S1130.
The shape feature value may be a value indicating a feature, such as mean curvature of a boundary region of a pulmonary nodule, the proportion of a linear area, or the proportion of a convex area. For the purpose of using a feature value regarding a shape for CAD, it is desirable to accurately capture the shape of a lesion. However, the attached region Sattached acquired in step S1130 may not correspond to the true boundary of the lesion region, and it is preferable to acquire the shape feature value taking the attached region Sattached into account. In the first embodiment, the feature value acquisition unit 1030 acquires a shape feature value on the basis of the unattached region.
While an example has been described in which a morphological operation is performed prior to step S1140 in order to address the possible occurrence of a missing element region, this process is optional. For example, the identification unit 1020 may not perform a morphological operation, and the feature value acquisition unit 1030 may acquire feature values only for consecutive regions.
If the identification of an attached region fails in step S1130, the feature value acquisition unit 1030 acquires a shape feature value on the basis of the boundary region Snodule. Alternatively, the feature value acquisition unit 1030 may also acquire a shape feature value on the basis of, for example, the boundary region Snodule, without taking the attached region Sattached into account.
In step S1150, the display control unit 1040 causes the display unit 160 to display a result of the processing up to step S1140. The display control unit 1040 causes the display unit 160 to display, as the result, a result of at least one process among the extraction of a pulmonary nodule region, the extraction of a pleural region, the identification of an attached region, and the acquisition of a feature value. The display control unit 1040 may display a feature value acquired without taking the attached region Sattached into account and a feature value acquired on the basis of the unattached region in such a manner that these feature values can be compared with each other.
The display control unit 1040 may cause the display unit 160 to display information on the result described above and the input image in such a manner that the information is superimposed on the input image. The display control unit 1040 may generate information on a three-dimensional image on which the pieces of information described above are superimposed by using volume rendering, and cause the display unit 160 to display the three-dimensional image. Alternatively, the display control unit 1040 may generate a predetermined image of a cross section of the three-dimensional image with the superimposed information, and may cause the display unit 160 to display the image.
In the first embodiment, accordingly, even for a lesion region whose shape is not accurately extractable, such as an attached pulmonary nodule, a feature value for the shape can be acquired more accurately by taking the influence of the attachment of the lesion region to any other structure into account. The acquisition of an accurate feature value results in an improvement in the performance of CAD technology using the feature value, such as abnormal opacity region detection or assistance for benignity/malignancy judgment.
A modification of a process for identifying the attached region Sattached by using the identification unit 1020 in step S1130 will be described. The identification unit 1020 extracts at least a portion of the non-extracted region by using a threshold process, and combines the extracted region, the boundary region Snodule, and the pleural region Vpleural to generate a connected region. The identification unit 1020 identifies the attached region Sattached on the basis of the distances between pixels within the connected region and the pleural region Vpleural.
As described above, lesions may occur near peripheral structures such as peripheral organs or other lesions. However, when a process of extracting a lesion region is performed, it is not necessarily determined whether the extracted lesion region is located near a peripheral structure from the extraction result. In a second embodiment, an example will be described of determining whether the first region and the second region are likely to be located near each other on the basis of the distance between the first region and the second region. Configurations, functions, or processes similar to those of the first embodiment are as described above and will not be described in detail.
Configuration of Image Processing Apparatus 200
An image processing apparatus 200 is, for example, a computer and has a hardware configuration similar to that in the example illustrated in
The determination unit 1021 determines whether, on the basis of information on an attached region identified by the identification unit 1020, a region of a structure to be analyzed, that is, the second region, is likely to be attached to any other structure. The determination unit 1021 further determines whether to correct the second region extracted by the second extraction unit 1012, on the basis of the information on the attached region. The determination unit 1021 is an example of determination means.
The correction unit 1022 corrects the second region extracted by the second extraction unit 1012. The correction unit 1022 is an example of correction means.
Regarding Series of Processes
The processing of steps S2100 to S2130 is similar to the processing of steps S1100 to S1130 according to the first embodiment described with reference to
In step S2132, the determination unit 1021 determines whether the second region is likely to be located near the first region on the basis of the identification of the fourth region in step S2130. Accordingly, it is determined that the extracted lesion region is likely to be located near a region of any other structure. In the second embodiment, the determination unit 1021 further determines whether to correct the extracted pulmonary nodule region in accordance with the identification of the fourth region. In the following, a process of correcting the extracted pulmonary nodule region is referred to as a correction process. The correction process is a process for, when the extracted pulmonary nodule region is smaller than the original pulmonary nodule region, making the size of the extracted pulmonary nodule region close to the original size.
In the example illustrated in
The determination unit 1021 acquires the number of pixels identified as being included in the attached region Sattached among the pixels within the boundary region Snodule acquired in step S2120. A proportion Psurface, which is the ratio of the pixels identified as being included in the attached region Sattached to the pixels within the boundary region Snodule, is acquired. If the proportion Psurface is greater than a predetermined threshold Tsurface, the determination unit 1021 determines that the extraction result of the second region is not suitable and that a correction process needs to be performed. The threshold Tsurface may set by the user in advance or may be determined in accordance with machine learning. If it is determined that a correction process needs to be performed, the process proceeds to step S2135. If it is determined that a correction process is not necessary, the process proceeds to step S2140.
In step S2135, the correction unit 1022 corrects the extraction result of the second region acquired in step S2120. In the following, the pulmonary nodule region extracted in step S2120 is referred to as an old pulmonary nodule region.
Examples of the process for correcting the extraction result of the pulmonary nodule include a process of setting the old pulmonary nodule region as a foreground region (seed region) and growing the old pulmonary nodule region by using the level-set method, the graph-cut method, or the like. The correction unit 1022 may correct the old pulmonary nodule region by using a technique having features different from those of the region extraction technique used in step S2120. For example, in step S2120, when the second extraction unit 1012 extracts a second region by using a technique based on the total optimization for a region, such as the Graph-cut technique, then, in step S2135, the correction unit 1022 may utilize a region growing technique capable of representing local features. The region growing technique capable of representing local features is, for example, a technique for performing local optimization based on local features for density values, that is, the mean, variance, maximum, and minimum values of density values or density value profile in a neighboring region of the pixel of interest.
In accordance with the correction of the second region in step S2135, the size of the region determined to be a non-extracted region is reduced. The processing of step S2130 is performed again based on the extraction result of the second region obtained in step S2135 to identify an attached region.
The processing of step S2140 is similar to the processing of step S1140 described with reference to
The processing of step S2150 is similar to the processing of step S1150 described with reference to
In the second embodiment, accordingly, a fourth region, which is a region where a region of a structure to be analyzed is located near any other structure, is identified to determine whether the structure to be analyzed is likely to be attached to any other structure. In the second embodiment, furthermore, it is possible to determine whether, on the basis of the result of identifying the fourth region, the region of the structure to be analyzed, that is, the result of extracting the second region, is suitable. If it is determined that the result of extracting the second region is not suitable, the result is corrected to more accurately extract the second region. Accordingly, the accuracy of a feature value for shape is enhanced, and the accuracy of CAD technology using the feature value is enhanced.
In the embodiments described above, a pulmonary nodule attached to the pleura on a chest CT image is to be processed. However, the type of modality for acquiring the target organ or lesion and an input image is not limited to that described above.
In the embodiments described above, the use of a partial image as illustrated in
In the embodiments described above, a region of the anatomical structure, such as the pleura, is used as a first region, and a region of a lesion, such as a pulmonary nodule, is used as a second region. However, the present invention is not limited to these embodiments. For example, both the first region and the second region may be regions of lesions, and an attached region can be identified by using a process similar to that described above.
The image processing apparatus 100 and the image processing apparatus 200 may further include a reasoning unit (not illustrated) for CAD. The reasoning unit (not illustrated) acquires a diagnostic name on the basis of a feature value acquired by the feature value acquisition unit 1030. The feature value acquisition unit 1030 may output a feature value to the reasoning unit (not illustrated) or may convert a feature value into another value corresponding to the feature value and then output the value to the reasoning unit. Another value corresponding to a feature value is, for example, information on a character string representing a state indicated by the feature value, or information on so-called findings.
The embodiments described above describe an example in which the shortest path length of an unattached region is set to −1 and the shortest path length of an attached region is set to 0 or the number of repetitions N of the region growing process. Accordingly, the identification unit 1020 can distinguish the unattached region and the attached region from each other by using the signs of the acquired shortest path lengths. The present invention is not limited to these embodiments, and any method capable of distinguishing the attached region and the unattached region from each other may be used.
The embodiments described above describe an example in which the identification unit 1020 acquires a shortest path length by using the region growing process. Path lengths obtained through the region growing process are typically geometric shortest path lengths, but are sometimes different from the lengths of geometric shortest paths. A shortest path length obtained through the region growing process is considered to satisfy the accuracy required in step S1130. To determine a more accurate shortest path length, a method based on the graph search algorithm may be used. The graph search algorithm is, for example, Dijkstra's algorithm or the A* search algorithm. In this case, the identification unit 1020 first performs a threshold process on the first region by using the threshold [T1, T2] and then generates a graph whose nodes correspond to pixels within the obtained region. Then, the identification unit 1020 applies Dijkstra's algorithm or the A* search algorithm to each of the pixels belonging to the boundary region Snodule by using the pleural region Vpleural as a search endpoint. In such a method, a shortest path is generated as a set of lines (edges) connecting nodes.
The embodiments described above describe an example in which an unattached region is identified on the basis of a shortest path length. However, the present invention is not limited to these embodiments. An unattached region may be acquired by, for example, identifying the attached region Sattached and subtracting the identified attached region Sattached from the boundary region Snodule. Further, the fourth region and the third region may not have an exclusive relationship, and may partially overlap.
The embodiments described above describe an example in which the lesion to be analyzed is attached to any other structure. However, the present invention is not limited to these embodiments. For example, an image processing apparatus according to an embodiment of the present invention may extract a region of a lesion that is not attached to any other structure and use the region as the analytical target. Further, an image processing apparatus according to an embodiment of the present invention may be connected to, for example, a modality for acquiring a medical image such as a CT image. In this case, the image processing apparatus may be a control device for controlling the modality.
The present invention can also be implemented using a process for providing a program that implements one or more functions in the embodiments described above to a system or device via a network or a storage medium and for reading and executing the program by one or more processors in a computer of the system or device. The present invention can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.
An image processing apparatus in each of the embodiments described above may be implemented as a standalone device or may be implemented as an arrangement where a plurality of devices are combined so as to be capable of communicating with each other to execute the processes described above, and both implementations are included in an embodiment of the present invention. The processes described above may be executed by a shared server device or a server group. An image processing apparatus and a plurality of devices making up an image processing system are desirably capable of performing communication at a predetermined communication rate, and do not need to be located in the same facility or even in the same country.
Embodiments of the present invention include an embodiment in which a software program that implements the functions of the embodiments described above is provided to a system or device and a computer of the system or device reads and executes code of the provided program.
Accordingly, the program code that is installed in a computer to allow the computer to implement processes according to the embodiments is in itself an embodiment of the present invention. Further, in accordance with instructions included in the program read by the computer, an OS or the like operating on the computer may perform some or all of the actual processes, and such processes may also implement the functions of the embodiments described above.
The embodiments described above may also be combined as appropriate, and such a combination is also included in an embodiment of the present invention.
The present invention is not restricted to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Accordingly, the following claims are appended to make public the scope of the present invention.
In an image processing apparatus according to an embodiment of the present invention, a feature value is acquired using a region that is included in a region to be analyzed and that is identified on the basis of the distance from any other region. Accordingly, the region can be analyzed taking into account that the region is located near any other region.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2016-218864 | Nov 2016 | JP | national |
This application is a Continuation of International Patent Application No. PCT/JP2017/035270, filed Sep. 28, 2017, which claims the benefit of Japanese Patent Application No. 2016-218864, filed Nov. 9, 2016, both of which are hereby incorporated by reference herein in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5185809 | Kennedy | Feb 1993 | A |
| 5881124 | Giger | Mar 1999 | A |
| 6246784 | Summers | Jun 2001 | B1 |
| 6549646 | Yeh | Apr 2003 | B1 |
| 9449247 | Yamada | Sep 2016 | B2 |
| 20030099384 | Zeng | May 2003 | A1 |
| 20030099389 | Zeng | May 2003 | A1 |
| 20030099390 | Zeng | May 2003 | A1 |
| 20040064029 | Summers | Apr 2004 | A1 |
| 20040086161 | Sivaramakrishna | May 2004 | A1 |
| 20050207630 | Chan | Sep 2005 | A1 |
| 20050286750 | Dehmeshki | Dec 2005 | A1 |
| 20070274583 | Sugiyama | Nov 2007 | A1 |
| 20080137921 | Simon | Jun 2008 | A1 |
| 20090129673 | Simon | May 2009 | A1 |
| 20150254842 | Brown | Sep 2015 | A1 |
| 20150348262 | Chen | Dec 2015 | A1 |
| 20150356730 | Grove | Dec 2015 | A1 |
| 20150379709 | Liang | Dec 2015 | A1 |
| 20160260211 | Gillies | Sep 2016 | A1 |
| 20170039737 | Madabhushi | Feb 2017 | A1 |
| 20170140534 | Chen | May 2017 | A1 |
| 20170256062 | Furukawa | Sep 2017 | A1 |
| 20180025250 | Chen | Jan 2018 | A1 |
| 20180181828 | Sakamoto | Jun 2018 | A1 |
| 20190046127 | Furukawa | Feb 2019 | A1 |
| 20190057503 | Nakamura | Feb 2019 | A1 |
| 20190156954 | Madabhushi | May 2019 | A1 |
| 20190258892 | Chen | Aug 2019 | A1 |
| 20200226746 | Schwartzbard | Jul 2020 | A1 |
| 20210118130 | Zhang | Apr 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 2007-312892 | Dec 2007 | JP |
| 2008-503294 | Feb 2008 | JP |
| 2011-135938 | Jul 2011 | JP |
| 2006000738 | Jan 2006 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20190258892 A1 | Aug 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2017/035270 | Sep 2017 | US |
| Child | 16402066 | US |
