Patent Grant
12033248
The present disclosure relates to a correction instruction region display apparatus, a correction instruction region display method, and a non-transitory computer readable recording medium storing a correction instruction region display program that display a correction instruction region that is used to correct a boundary of a region of interest extracted in a three-dimensional image.
In recent years, advances in medical equipment, such as a computed tomography (CT) apparatus and a magnetic resonance imaging (MRI) apparatus, have made it possible to perform diagnostic imaging using a higher quality three-dimensional image having higher resolution. A region of interest such as an organ and a lesion included in such a three-dimensional image has also been automatically extracted. However, in a case where the region of interest is automatically extracted, over-extraction and under-extraction may occur. In such a case, it is necessary to correct the boundary of the automatically extracted region of interest.
As a method of correcting the boundary, a method has been performed in which a two-dimensional tomographic image constituting a three-dimensional image is displayed, a cursor having a predetermined shape, such as a circle, is moved on the displayed tomographic image, and the over-extracted area is deleted from or the under-extracted area is added to the region of interest. However, in a case where the boundary of the region of interest is corrected for each tomographic image, the boundaries of the regions of interest in the cross-section perpendicular to the tomographic image may not be smoothly connected to each other.
In such a case, it is conceivable to three-dimensionally correct the boundary of the region of interest in the three-dimensional image by using a spherical three-dimensional cursor. However, in the cross-section perpendicular to the tomographic image, the boundary of the region of interest may not be spherical. Therefore, in a case where the boundary of the region of interest is corrected by using a spherical cursor, unintended over-correction or under-correction (hereinafter, referred to as over-correction or the like) may occur.
Meanwhile, a method has been proposed in which regions of interest are set in two tomographic images among a plurality of tomographic images constituting a three-dimensional image, the regions of interest set in the two tomographic images are interpolated into tomographic images that exist between the two tomographic images, and the region of interest is set (JP1998-57371A (JP-H10-57371A)). In addition, a method has also been proposed in which in response to deformation operation of a contour line of a region of interest in a designated tomographic plane, for a tomographic image constituting a three-dimensional image, deformation operation of the contour line in the other tomographic plane is performed (JP2012-14360A). The methods described in JP1998-57371A (JP-H10-57371A) and JP2012-14360A are used, so that the region of interest can be corrected even for a tomographic image that is not displayed.
However, in the method described in JP1998-57371A (JP-H10-57371A), a user does not know how the regions of interest are set in the tomographic images that exist between the two tomographic images until the set regions of interest are displayed. Further, in the method described in JP2012-14360A, the user also does not know how the contour line is deformed in the other tomographic plane until the other tomographic plane is displayed. For this reason, in the methods described in JP1998-57371A (JP-H10-57371A) and JP2012-14360A, unintended over-correction or the like in the region of interest may also occur.
The present disclosure has been made in view of the above circumstances, and an object thereof is to prevent, in a case where a region of interest for a tomographic image of one tomographic plane of a three-dimensional image is corrected, the occurrence of unintended over-correction or the like in a tomographic image of the other tomographic plane.
A correction instruction region display apparatus according to the present disclosure comprises: a first display control unit that displays a first tomographic image of a three-dimensional image consisting of a plurality of tomographic images on a display unit, and superimposes and displays, as a first cross-section correction instruction region, a cross-section of a three-dimensional correction instruction region on the first tomographic image, which is used to correct a boundary of a region of interest extracted from the three-dimensional image, on the first tomographic image; and a second display control unit that displays at least one second tomographic image adjacent to the first tomographic image, which includes the three-dimensional correction instruction region, on the display unit, and superimposes and displays, as a second cross-section correction instruction region, a cross-section of the three-dimensional correction instruction region on the second tomographic image, on the second tomographic image.
In the correction instruction region display apparatus according to the present disclosure, the second display control unit may display the second tomographic image and the second cross-section correction instruction region on the display unit, on the basis of a display instruction of the second tomographic image.
Further, in the correction instruction region display apparatus according to the present disclosure, the second display control unit may display a predetermined number of second tomographic images among a plurality of second tomographic images which include the three-dimensional correction instruction region, on the display unit.
Further, in the correction instruction region display apparatus according to the present disclosure, the second display control unit may display a partial region including the second cross-section correction instruction region in the second tomographic image, on the display unit.
Further, in the correction instruction region display apparatus according to the present disclosure, a first correction unit that corrects the boundary of the region of interest included in the first tomographic image in response to a correction instruction using the first cross-section correction instruction region for the boundary of the region of interest included in the first tomographic image may further be provided.
Further, in the correction instruction region display apparatus according to the present disclosure, a second correction unit that corrects the boundary of the region of interest included in the second tomographic image by using the second cross-section correction instruction region, in response to the correction instruction using the first cross-section correction instruction region for the boundary of the region of interest included in the first tomographic image may further be provided.
Further, in the correction instruction region display apparatus according to the present disclosure, the three-dimensional correction instruction region may be spherical.
Further, in the correction instruction region display apparatus according to the present disclosure, a region-of-interest extraction unit that extracts a region of interest from each of the plurality of tomographic images may further be provided.
A correction instruction region display method according to the present disclosure comprises: displaying a first tomographic image of a three-dimensional image consisting of a plurality of tomographic images on a display unit, and superimposing and displaying, as a first cross-section correction instruction region, a cross-section of a three-dimensional correction instruction region on the first tomographic image, which is used to correct a boundary of a region of interest extracted from the three-dimensional image, on the first tomographic image; and displaying at least one second tomographic image adjacent to the first tomographic image, which includes the three-dimensional correction instruction region, on the display unit, and superimposing and displaying, as a second cross-section correction instruction region, a cross-section of the three-dimensional correction instruction region on the second tomographic image, on the second tomographic image.
The correction instruction region display method according to the present disclosure may also be provided as a non-transitory computer readable recording medium storing a program to be executed by a computer.
Another correction instruction region display apparatus according to the present disclosure comprises: a memory that stores a command to be executed by a computer; and a processor configured to execute the stored command, in which the processor executes processing of displaying a first tomographic image of a three-dimensional image consisting of a plurality of tomographic images on a display unit, and superimposing and displaying, as a first cross-section correction instruction region, a cross-section of a three-dimensional correction instruction region on the first tomographic image, which is used to correct a boundary of a region of interest extracted from the three-dimensional image, on the first tomographic image, and displaying at least one second tomographic image adjacent to the first tomographic image, which includes the three-dimensional correction instruction region, on the display unit, and superimposing and displaying, as a second cross-section correction instruction region, a cross-section of the three-dimensional correction instruction region on the second tomographic image, on the second tomographic image.
According to the present disclosure, in a case where a region of interest for a tomographic image of one tomographic plane of a three-dimensional image is corrected, the occurrence of unintended over-correction or the like in a tomographic image of the other tomographic plane can be prevented.
Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.
The three-dimensional image capturing apparatus 2 is an apparatus that images an area to be diagnosed of a subject and that generates a three-dimensional image representing the area, and specific examples thereof include a CT apparatus, an MRI apparatus, and a positron emission tomography (PET) apparatus. The three-dimensional image generated by the three-dimensional image capturing apparatus 2 is transmitted to and stored in the image storage server 3. In the present embodiment, the three-dimensional image capturing apparatus 2 is a CT apparatus and generates a CT image including the area to be diagnosed of the subject, as a three-dimensional image G0. In addition, the three-dimensional image G0 consists of a plurality of tomographic images.
The image storage server 3 is a computer that stores and manages various data and comprises a large-capacity external storage device and database management software. The image storage server 3 communicates with another apparatus via a wired or wireless network 4 to transmit and receive image data and the like. Specifically, the image storage server 3 acquires various data including the image data of the three-dimensional image G0 generated by the three-dimensional image capturing apparatus 2 via the network, and stores and manages the acquired data on a recording medium such as a large-capacity external storage device. A storage format of the image data and the communication between the apparatuses via the network 4 are based on a protocol such as digital imaging and communications in medicine (DICOM).
The correction instruction region display apparatus 1 is an apparatus obtained by installing a correction instruction region display program of the present embodiment on one computer. The computer may be a workstation or a personal computer directly operated by a doctor who makes diagnosis, or may be a server computer connected to the workstation or the personal computer via a network 4. The correction instruction region display program is stored on a storage device of a server computer connected to the network 4 or on network storage so as to be accessible from the outside, and is downloaded and installed on a computer that the doctor uses according to a request. Alternatively, the correction instruction region display program is recorded on a recording medium, such as a digital versatile disc (DVD) or a compact disc read only memory (CD-ROM), and is distributed and installed on a computer from the recording medium.
The storage 13 includes a hard disk drive or the like, and stores the three-dimensional image G0 acquired from the image storage server 3 via the network 4 and various information including information necessary for processing.
Further, the correction instruction region display program is stored in the memory 12. The correction instruction region display program defines the following processing as a process to be executed by the CPU 11: image acquiring processing of acquiring a three-dimensional image G0; region-of-interest extracting processing of extracting a region of interest from each of the plurality of tomographic images constituting the three-dimensional image G0; first display controlling processing of displaying a first tomographic image of the plurality of tomographic images on a display unit 14, and superimposing and displaying, as a first cross-section correction instruction region, a cross-section of a three-dimensional correction instruction region on the first tomographic image, which is used to correct a boundary of the region of interest extracted from the three-dimensional image G0, on the first tomographic image; second display controlling processing of displaying at least one second tomographic image adjacent to the first tomographic image, which includes the three-dimensional correction instruction region, on the display unit 14, and superimposing and displaying, as a second cross-section correction instruction region, a cross-section of the three-dimensional correction instruction region on the second tomographic image, on the second tomographic image; first correcting processing of correcting the boundary of the region of interest included in the first tomographic image in response to a correction instruction using the first cross-section correction instruction region for the boundary of the region of interest included in the first tomographic image; and second correcting processing of correcting the boundary of the region of interest included in the second tomographic image by using the second cross-section correction instruction region, in response to the correction instruction using the first cross-section correction instruction region for the boundary of the region of interest included in the first tomographic image.
The CPU 11 executes the processing according to the program to make a computer function as an image acquisition unit 21, a region-of-interest extraction unit 22, a first display control unit 23, a second display control unit 24, a first correction unit 25, and a second correction unit 26.
The image acquisition unit 21 acquires the three-dimensional image G0 including the region of interest, from the image storage server 3 via an interface (not shown) connected to the network 4. The region of interest is an organ, a specific region within the organ, or a region of a structure, such as bone and cartilage, that the user is interested in, for example, as a diagnosis target. In a case where the three-dimensional image G0 is already stored in the storage 13, the image acquisition unit 21 may acquire the three-dimensional image G0 from the storage 13.
The region-of-interest extraction unit 22 extracts the region of interest from the three-dimensional image G0. For the extraction, the region-of-interest extraction unit 22 comprises a learned model that has performed machine learning so as to extract the region of interest from the three-dimensional image G0. The learned model includes a neural network that has performed deep learning so as to extract, as a region of interest, an organ, a specific region within the organ, or a region of a structure, such as bone and cartilage, which is a diagnosis target. Examples of the organ as a diagnosis target include a heart, liver, lungs, kidneys, and brain. In the present embodiment, for example, white matter of the brain is the region of interest. In a case where the input of the three-dimensional image G0 is received, the learned model outputs a determination result indicating whether or not each pixel of the three-dimensional image G0 corresponds to the region of interest. The region-of-interest extraction unit 22 extracts a region consisting of pixels determined to correspond to the region of interest, as the region of interest.
In addition to the neural network that has performed deep learning, the learned model may include, for example, a support vector machine (SVM), a convolutional neural network (CNN), and a recurrent neural network (RNN). Further, the region-of-interest extraction unit 22 is not limited to a unit provided with the learned model that has performed machine learning. For example, the region of interest may be extracted by template matching, threshold value processing, or the like.
Hereinafter, processing performed by the first display control unit 23, the second display control unit 24, the first correction unit 25, and the second correction unit 26 will be described.
Here, the extraction result by the region-of-interest extraction unit 22 may not be always accurate, and the region of interest 30 may not completely match with an actual region of interest 31 included in the first tomographic image Dk.
Next, the first display control unit 23 starts monitoring whether or not a display instruction of a cursor is given through the input unit 15 (Step ST4). In a case where affirmative determination is made in Step ST4, the first display control unit 23 displays the first tomographic image Dk with the three-dimensional cursor superimposed thereon, on the display unit 14 (display the cursor; Step ST5). In a case where negative determination is made in Step ST4, monitoring whether or not the display instruction of the cursor is given through the input unit 15 is repeated (Step ST4). In the present embodiment, the three-dimensional cursor is spherical. For this reason, in the present embodiment, as shown in
Subsequently, the second display control unit 24 determines whether or not a display instruction of the second tomographic image is given through the input unit 15 (Step ST6). In a case where affirmative determination is made in Step ST6, the second display control unit 24 displays a partial image of at least one second tomographic image adjacent to the first tomographic image, which includes the three-dimensional cursor, on the display unit 14 (display the second tomographic image; Step ST7). In a case where, for example, the input unit 15 has a keyboard, the display instruction of the second tomographic image can be given through the pressing of a control key on the keyboard, but the instruction is not limited thereto. Further, the second display control unit 24 superimposes and displays, as a second cursor, a cross-section of the three-dimensional cursor on the second tomographic image, on the partial image of the second tomographic image (Step ST8).
Here, since the three-dimensional cursor is spherical, as shown in
In a case where negative determination is made in Step ST6 or the process follows Step ST8, the first correction unit 25 receives a correction instruction using the three-dimensional cursor 44 by the user (Step ST9). In this case, the user moves the first cursor 40 by using the mouse of the input unit 15 to give the correction instruction so that the boundary 30L of the region of interest 30 matches with the boundary 31L of the actual region of interest 31. The first correction unit 25 corrects the boundary 30L of the region of interest 30 in the first tomographic image Dk by using the first cursor 40 in accordance with the correction instruction by the user (first correction; Step ST10). Further, the second correction unit 26 corrects the boundaries of the regions of interest included in the second tomographic images Dk+2, Dk+1, Dk−1, and Dk−2 by using the second cursors 41A to 41D, respectively, in response to the correction instruction of the boundary 30L of the region of interest 30 of the first tomographic image Dk (second correction; Step ST11).
Subsequently, determination is made whether or not a change instruction of the tomographic image to be corrected is input through the input unit 15 (Step ST12). In a case where affirmative determination is made in Step ST12, the first tomographic image displayed on the display unit 14 is changed (Step ST13), the process returns to Step ST4, and the processing after Step ST4 is repeated. In a case where negative determination is made in Step ST12, determination is made whether or not an end instruction is given (Step ST14). In a case where negative determination is made in Step ST14, the process returns to Step ST9, and the processing after Step ST9 is repeated. In a case where affirmative determination is made in Step ST14, the processing ends.
As described above, in the present embodiment, the first tomographic image Dk is displayed, and the cross-section of the three-dimensional cursor 44 on the first tomographic image Dk is superimposed and displayed on the first tomographic image Dk, as the first cursor 40. Further, the second tomographic images Dk−2, Dk−1, Dk+1, or Dk+2 adjacent to the first tomographic image Dk, in which the three-dimensional cursor 44 is included are displayed, and the cross-sections of the three-dimensional cursor 44 on the second tomographic images are superimposed and displayed as the second cursors 41A to 41D, on the second tomographic images Dk+2, Dk+1, Dk−1, and Dk−2, respectively. The second tomographic image arranged adjacent to the first tomographic image Dk may be at least one of the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2. Therefore, in a case where the boundary 30L of the region of interest 30 in the first tomographic image Dk is corrected by using the three-dimensional cursor 44, the user sees the displayed second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2, so that the user can confirm how the region of interest is corrected by the three-dimensional cursor 44 in the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 adjacent to the first tomographic image Dk. In addition, processing such as changing the size of the three-dimensional cursor 44 may be performed so that over-correction does not occur. Therefore, according to the present embodiment, in a case where a region of interest for a tomographic image of one tomographic plane of a three-dimensional image G0 is corrected, the occurrence of unintended over-correction or the like in a tomographic image of the other tomographic plane can be prevented.
Further, in the present embodiment, in a case where there is the display instruction of the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2, the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 are displayed. Therefore, in a case where the boundary 30L of the region of interest 30 is corrected, the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 are not displayed, so that resources such as the CPU 11 and the memory 12 are not consumed in order to display the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2. Accordingly, it is possible to prevent the correcting processing from becoming heavy as compared with a case where the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 are always displayed. Further, since the partial images Bk−2, Bk−1, Bk+1, and Bk+2 of the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 are displayed, the consumption of resources such as the CPU 11 and the memory 12 is reduced as compared with a case where the entire second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 are displayed. As a result, it is possible to prevent the processing for display from becoming heavy.
In the present embodiment, in a case where there is the display instruction of the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2, the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 are displayed, but the present disclosure is not limited thereto. In a case where there is the display instruction of the first tomographic image Dk, the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 may be displayed at the same time as the first tomographic image Dk.
Further, in the above-described embodiment, the partial images of the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 are displayed, but the present disclosure is not limited thereto. The second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 may be displayed as they are on the display unit 14. In this case, the first tomographic image Dk and the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 may be displayed side by side on the display unit 14, and the first tomographic image Dk and the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2 may be switched and displayed.
Further, in the above-described embodiment, partial images of the second tomographic images Dk−2, Dk−1, Dk+1, and Dk+2, two on each of the upper and lower sides of the first tomographic image Dk, are displayed, but the present disclosure is not limited thereto. Only the second tomographic images Dk−1 and Dk+1 on the upper and lower sides of the first tomographic image Dk, or only the second tomographic image, either one on the upper or lower side of the first tomographic image Dk, may be displayed.
Further, in the above-described embodiment, in a case where the size of the three-dimensional cursor 44 is enlarged, more tomographic images are included in the three-dimensional cursor. For example, as shown in
Further, in the above-described embodiment, the three-dimensional cursor is spherical, but the present disclosure is not limited thereto. A three-dimensional cursor having any shape, such as a rectangular parallelepiped, a cube, or a shape in which the bottom surfaces of two cones are connected to each other, may be used, as long as the three-dimensional cursor has a three-dimensional shape.
Further, in the above-described embodiment, the correction instruction region display apparatus 1 comprises the region-of-interest extraction unit 22, but the present disclosure is not limited thereto. The region of interest may be extracted by a separate apparatus connected to the correction instruction region display apparatus 1 via the network 4. Further, the three-dimensional image G0 to be acquired may be an image in which a region of interest is already extracted.
Further, in the above-described embodiment, for example, as a hardware structure of a processing unit that executes various processing such as processing performed by the image acquisition unit 21, the region-of-interest extraction unit 22, the first display control unit 23, the second display control unit 24, the first correction unit 25, and the second correction unit 26, the following various processors may be used. The various processors include, for example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), which is a processor having a changeable circuit configuration after manufacture, and a dedicated electrical circuit, such as an application specific integrated circuit (ASIC), which is a processor having a dedicated circuit configuration designed to perform specific processing, in addition to the CPU which is a general-purpose processor that executes software (programs) to function as various processing units as described above.
One processing unit may be constituted of one of the various processors or may be constituted of a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs and a combination of a CPU and an FPGA).
Further, the plurality of processing units may constitute one processor.
A first example of the configuration in which the plurality of processing units are constituted of one processor is an aspect in which one or more CPUs and software are combined to constitute one processor and the processor functions as a plurality of processing units. A representative example of the aspect is a computer such as a client and server. A second example of the configuration is an aspect in which a processor that implements all of the functions of a system including the plurality of processing units with one integrated circuit (IC) chip is used. A representative example of the aspect is a system on chip (SoC). As described above, as the hardware structure of various processing units, one or more of the various processors are used.
Further, as the hardware structure of the various processors, more specifically, an electrical circuit (circuitry) in which circuit elements such as semiconductor elements are combined may be used.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2019-104701 | Jun 2019 | JP | national |
This application is a Continuation of PCT International Application No. PCT/JP2020/018394 filed on May 1, 2020, which claims priority under 35 U.S.C § 119(a) to Japanese Patent Application No. 2019-104701 filed on Jun. 4, 2019. Each of the above application(s) is hereby expressly incorporated by reference, in its entirety, into the present application.
| Number | Name | Date | Kind |
|---|---|---|---|
| 8351670 | Ijiri et al. | Jan 2013 | B2 |
| 20080267481 | Nakamura | Oct 2008 | A1 |
| 20090257550 | Moriya | Oct 2009 | A1 |
| 20140286551 | Yoshida | Sep 2014 | A1 |
| 20160026266 | Douglas et al. | Jan 2016 | A1 |
| 20180353078 | Lee | Dec 2018 | A1 |
| 20190139223 | Nie et al. | May 2019 | A1 |
| 20210241016 | Hashimoto | Aug 2021 | A1 |
| 20210256741 | Sakuragi | Aug 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| H1057371 | Mar 1998 | JP |
| 2009106426 | May 2009 | JP |
| 2012014360 | Jan 2012 | JP |
| Entry |
|---|
| “International Search Report (Form PCT/ISA/210) of PCT/JP2020/018394,” mailed on Jun. 30, 2020, with English translation thereof, pp. 1-5. |
| “Written Opinion of the International Searching Authority (Form PCT/ISA/237)” of PCT/JP2020/018394, mailed on Jun. 30, 2020, with English translation thereof, pp. 1-8. |
| Number | Date | Country | |
|---|---|---|---|
| 20220076462 A1 | Mar 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2020/018394 | May 2020 | WO |
| Child | 17525942 | US |
