Patent Application
20220415481
This invention relates to the field of mesh topologies and more particularly to adapting a first, pre-defined mesh topology representing an organ to a second, different mesh topology of the organ.
Annotated medical images are widely used in scientific medical research, e.g. for evaluation and training of image segmentation techniques. A popular segmentation technique is model-based segmentation (MBS). With this technique, a triangulated mesh representing an organ boundary can be adapted to a medical image in a controlled way so that the general organ shape is preserved, thus regularizing the segmentation with prior anatomical knowledge. This segmentation model has to be trained using image data and corresponding ground truth meshes of the segmented organs. Usually, these meshes are manually created by clinicians or researchers. However, this manual process is very time consuming.
The process of generating ground truth meshes may be accelerated by using an existing model developed for the same organ, the existing model having been previously trained on similar image data. Alternatively, a ground truth mesh for the organ might already be available, having been previously delineated by clinicians for a different segmentation model. For both cases, the topology of the available ground truth meshes (either from previous segmentation or a different model) might not match the existing model topology (e.g. not match in number of vertices and triangles), and, for model-based segmentation, meshes typically have to have the same topology.
One way of getting the organ meshes in the same topology is to find a correspondence between the vertices in the two different meshes, which could be provided by well-known point matching methods such as iterative closest point (ICP) or coherent point drift (CPD). However, although these methods account for global transformations in the feature space between the two shapes, one still needs to incorporate spatial regularity into the mapping, such that neighboring points in one mesh correspond to neighboring points in the other mesh.
The invention is defined by the claims.
According to examples in accordance with an aspect of the invention, there is provided a method for adapting a first, pre-defined mesh topology representing an organ to a second, different mesh topology of the organ. The method comprises: identifying correspondence between the first and second mesh topologies based on spectral matching of the first and second mesh topologies; and aligning the first, predefined mesh topology with the second mesh topology based on the identified correspondence between the first and second mesh topologies.
Proposed embodiments provide concepts for adapting a first, pre-defined mesh topology to a second, different mesh topology. For instance, embodiments may be used to adapt a first, ground truth mesh for an organ to a second, new mesh topology of the organ. In particular, embodiments may employ spectral matching to align/adapt the first, pre-defined (i.e. ground truth) mesh to a second, new mesh topology. Put another way, it is proposed to use spectral matching for aligning an already existing mesh with a new topology of the same organ. Such approaches may help to relax requirements with respect to annotated datasets.
By way of example, proposed embodiments may provide a method for using spectral matching to find the correspondence between an already existing mesh topology (e.g. obtained either from segmentation using a pre-trained model or from previous ground truth mesh generation by a user) and a newly defined mesh topology. In this way, it may be possible to define a correspondence between meshes of an organ with different topologies, because a direct vertex correspondence between the two meshes is not required due to the conversion into the spectral space. Furthermore, such a proposed approach may ensure spatial regularity, due to the use of spectral embedding (e.g. computed from the graph Laplacian of the surface meshes).
Embodiments may be based on the idea to use spectral matching for aligning an already existing mesh with a new topology of the same organ. In such embodiments, it is proposed to use spectral matching to find the correspondence between an already existing mesh topology and a newly defined mesh topology. This may enable the conversion of previously-defined ground truth meshes to a new, desired mesh topology of a model (that can then be retrained for segmentation).
By way of example, identifying correspondence between the first and second mesh topologies based on spectral matching of the first and second mesh topologies may comprise: obtaining a first spectral graph derived from the first mesh topology; obtaining a second spectral graph derived from the second mesh topology; decomposing the first and second spectral graphs to determine spectral coordinates from the first and second spectral graphs; analyzing the determined spectral coordinates to identify matches between the first and second spectral graphs; and identifying correspondence between the first and second mesh topologies based on the identified matches. Embodiments may therefore employ conventional/known spectral graph generation and/or matching processes, thus leveraging widely known and/or available processes.
In some embodiments, aligning the first, predefined mesh topology with the second mesh topology may comprise aligning spectral components of the first and second mesh topologies. For instance, aligning spectral components may comprise processing the first mesh topology with a point transformation method on both meshes, such as a Coherent Point Drift (CPD) method or an Iterative Closest Point (ICP) method for example. Known point matching methods may therefore be employed, thus reducing the complexity and/or cost of embodiments for example.
In an exemplary embodiment, aligning the first, predefined mesh topology with the second mesh topology may comprise resampling the first mesh topology based on the identified correspondence between the first and second mesh topologies.
Because the proposed concepts may enable the conversion of previously-defined ground truth meshes to a new, desired mesh topology of a model (that can then be retrained for segmentation), it will be understood that embodiments of the invention may be leveraged for use in image segmentation. Thus, according to another aspect of the invention, there is provided a method for segmentation of a medical image of an organ, the method comprising: adapting a first mesh topology representing the organ to a second, different mesh topology of the organ according to a proposed embodiment; and performing model-based segmentation of the image using the adapted first mesh topology.
According to another aspect, there is provided a computer program product for adapting a first, pre-defined mesh topology representing an organ to a second, different mesh topology of the organ, wherein the computer program product comprises a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to perform all of the steps of a proposed embodiment.
Thus, there may also be provided a computer system comprising: a computer program product according to proposed embodiment; and one or more processors adapted to perform a method according to a proposed concept by execution of the computer-readable program code of said computer program product.
According to still another aspect of the invention, there is provided a system for adapting a first, pre-defined mesh topology representing an organ to a second, different mesh topology of the organ, the system comprising: an analysis component configured to identify correspondence between the first and second mesh topologies based on spectral matching of the first and second mesh topologies; and a mesh alignment component configured to align the first, predefined mesh topology with the second mesh topology based on the identified correspondence between the first and second mesh topologies.
The system may be remotely located from a user device for adapting a first mesh topology to a second, different mesh topology. In this way, a user (such as a medical professional) may have an appropriately arranged system that can receive information at a location remotely located from the system for adapting a first, mesh topology to a second, different mesh topology. Embodiments may therefore enable a user to adapt a first mesh topology to a second, different mesh topology using a local system (which may, for example, comprise a portable display device, such as a laptop, tablet computer, mobile phone, PDA, etc.). By way of example, embodiments may provide an application for a mobile computing device, and the application may be executed and/or controlled by a user of the mobile computing device.
The system may further include: a server device comprising the system for adapting a first mesh topology to a second, different mesh topology; and a client device comprising a user-interface. Dedicated data processing means may therefore be employed for the purpose of adapting a first mesh topology to a second, different mesh topology, thus reducing processing requirements or capabilities of other components or devices of the system.
The system may further include a client device, wherein the client device comprises the mesh alignment component and a display unit. In other words, a user (such as a doctor or medical professional) may have an appropriately arranged client device (such as a laptop, tablet computer, mobile phone, PDA, etc.) which processes received data in order to adapt a first mesh topology to a second, different mesh topology and generate a display control signal. Purely by way of example, embodiments may therefore provide a mesh-based annotation system that enables medical analysis of one or more subjects (e.g. patients) from a single, remote location, wherein communication between a subject and monitoring user (e.g. nurse or doctor) is provided and can have its functionality extended or modified according to proposed concepts, for example.
It will be understood that processing capabilities may therefore be distributed throughout the system in different ways according to predetermined constraints and/or availability of processing resources.
These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.
For a better understanding of the invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:
The invention will be described with reference to the Figs.
It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality.
It should be understood that the Figs. are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figs. to indicate the same or similar parts.
Proposed is an approach for enabling the adaptation of a first, pre-defined mesh topology representing an organ to a second, different mesh topology of the organ. Embodiments may thus be used for translating a mesh from a first, defined (e.g. ground truth with annotations) model space to another. This may enable the ground truth annotations of the first model space to be used to train a new, different model (e.g. that is designed to take a new problem).
Embodiments propose to use spectral matching to find the correspondence between a first, pre-defined (e.g. pre-existing) mesh topology of an organ (either obtained from segmentation of the organ using a pre-trained model or from previous ground truth mesh generation by a clinician) and second (e.g. newly defined) mesh topology. This may therefore enable conversion of a pre-defined ground truth mesh to a new mesh topology of a model that can then be re-trained for segmentation.
Spectral matching is widely known and therefore detailed description of spectral matching is omitted from this description. However, by way of brief explanation, the spectral decomposition of the graph Laplacian associated with complex shapes provides eigenfunctions (modes) which are invariant to isometries. Each vertex on the shape could be uniquely represented with a combinations of the eigenmodal values at each point, sometimes called spectral coordinates. Spectral matching consists of establishing the point correspondences by pairing vertices on different shapes that have the most similar spectral coordinates. Spectral graph theory thus offers a solution for matching surface meshes with different topologies in the spectral domain, and may therefore provide applications for matching different meshes to create atlases.
The inventors propose to leverage spectral matching in order to bring a previously defined mesh into a new model topology. In this way, proposed embodiments may enable the identification of correspondence between meshes of the same organs but with different topologies (e.g. brains, hearts, kidneys, livers or lungs of two different subjects/patients). In particular, due to conversion into the spectral space, a direct vertex correspondence between meshes need not be required. Furthermore, such an approach may ensure spatial regularity, due to the use of the spectral embedding computed from the graph Laplacian of the surface meshes for example.
Proposed embodiments may therefore be employed for segmentation of a medical image of an organ. Such a method may include: adapting a first mesh topology representing the organ to a second, different mesh topology of the organ according to a proposed embodiment. Model-based segmentation of the image may then be performed using the adapted first mesh topology. Semi-automatic ground truth segmentations may therefore be facilitated by the proposed concept(s). By employing previously developed models, the process may be accelerated (because a user may only need to implement small corrections, rather than manually creating a new mesh for example). Furthermore, multi-purpose data may be provided, and this may, for instance, be used for more projects where the same organ needs to be delineated but with different topology.
By way of example only, illustrative embodiments may be utilized in many different types of clinical, medical or subject-related environments, such as a hospital, doctor's office, medical-research facility, ward, care home, person's home, etc.
To aid understanding of the proposed concept(s), an exemplary embodiment of a system for adapting a first, pre-defined mesh topology representing an organ to a second, different mesh topology of the organ will now be described with reference to
More specifically, the system 100 comprises an analysis component 130 that is configured to undertake spectral matching of the first 110 and second 120 mesh topologies to identify correspondence between the first 110 and second 120 mesh topologies. A mesh alignment component 135 of the system is then configured to align the first, predefined mesh topology 110 with the second mesh topology 120 based on the correspondence between the first and second mesh topologies (as identified by the analysis component 130).
The aligned mesh topology 140 is output from the system 100 (e.g. via an output interface), the aligned mesh topology 140 being adapted to the second mesh topology 120. In this way, annotations of the first mesh topology 110 are carried through into the aligned mesh topology 140, but are also aligned with the second mesh topology.
By way of further illustration and description, an exemplary method according to an embodiment will now be described with reference to
The method 200 then comprises second step 260 of aligning the first, predefined mesh topology with the second mesh topology based on the identified correspondence between the first and second mesh topologies. Here, the step 260 of aligning the first, predefined mesh topology with the second mesh topology comprises aligning spectral components of the first and second mesh topologies. More specifically, in this exemplary embodiment, aligning spectral components comprises processing the first mesh topology with a point transformation method, such as a Coherent Point Drift method or an Iterative Closest Point (ICP) method for example. Alternatively, or additionally, aligning the first, predefined mesh topology with the second mesh topology may comprise resampling the first mesh topology based on the identified correspondence between the first and second mesh topologies.
From the above-described exemplary embodiments, it will be appreciated there proposed concepts may facilitate the generation of consistent mesh-based annotations for images using spectral matching. In particular, proposed embodiments employ spectral matching to find the correspondence between an already existing mesh topology and a newly defined mesh topology. Instead of direct vertex correspondence, it is proposed to use spectral embeddings computed from graph Laplacian of the surface meshes.
Thus, there is proposed a method for using spectral matching to find the correspondence between an already existing mesh topology (either obtained from segmentation of the organ using a pre-trained model or from previous ground truth mesh generation by a clinician) and a newly defined mesh topology. This may be achieved by using the spectral embeddings of the two shapes, calculated independently, and looking for a match between a defined mesh (i.e. ground truth) and a template (i.e. the new mesh topology). After a match has been defined, the ground truth mesh may then be resampled in the new topology. This is supported by the spectral matching which offers a point-to-point mesh correspondence.
The computer 300 includes, but is not limited to, PCs, workstations, laptops, PDAs, palm devices, servers, storages, and the like. Generally, in terms of hardware architecture, the computer 300 may include one or more processors 310, memory 320, and one or more I/O devices 370 that are communicatively coupled via a local interface (not shown). The local interface can be, for example but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The local interface may have additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communications. Further, the local interface may include address, control, and/or data connections to enable appropriate communications among the aforementioned components.
The processor 310 is a hardware device for executing software that can be stored in the memory 320. The processor 310 can be virtually any custom made or commercially available processor, a central processing unit (CPU), a digital signal processor (DSP), or an auxiliary processor among several processors associated with the computer 300, and the processor 310 may be a semiconductor based microprocessor (in the form of a microchip) or a microprocessor.
The memory 320 can include any one or combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), tape, compact disc read only memory (CD-ROM), disk, diskette, cartridge, cassette or the like, etc.). Moreover, the memory 320 may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory 320 can have a distributed architecture, where various components are situated remote from one another, but can be accessed by the processor 310.
The software in the memory 320 may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. The software in the memory 320 includes a suitable operating system (O/S) 350, compiler 340, source code 330, and one or more applications 360 in accordance with exemplary embodiments. As illustrated, the application 360 comprises numerous functional components for implementing the features and operations of the exemplary embodiments. The application 360 of the computer 300 may represent various applications, computational units, logic, functional units, processes, operations, virtual entities, and/or modules in accordance with exemplary embodiments, but the application 360 is not meant to be a limitation.
The operating system 350 controls the execution of other computer programs, and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. It is contemplated by the inventors that the application 360 for implementing exemplary embodiments may be applicable on all commercially available operating systems.
Application 360 may be a source program, executable program (object code), script, or any other entity comprising a set of instructions to be performed. When a source program, then the program is usually translated via a compiler (such as the compiler 340), assembler, interpreter, or the like, which may or may not be included within the memory 320, so as to operate properly in connection with the O/S 350. Furthermore, the application 360 can be written as an object oriented programming language, which has classes of data and methods, or a procedure programming language, which has routines, subroutines, and/or functions, for example but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP scripts, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, and the like.
The I/O devices 370 may include input devices such as, for example but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. Furthermore, the I/O devices 370 may also include output devices, for example but not limited to a printer, display, etc. Finally, the I/O devices 370 may further include devices that communicate both inputs and outputs, for instance but not limited to, a NIC or modulator/demodulator (for accessing remote devices, other files, devices, systems, or a network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, etc. The I/O devices 370 also include components for communicating over various networks, such as the Internet or intranet.
If the computer 300 is a PC, workstation, intelligent device or the like, the software in the memory 320 may further include a basic input output system (BIOS) (omitted for simplicity). The BIOS is a set of essential software routines that initialize and test hardware at startup, start the 0/S 350, and support the transfer of data among the hardware devices. The BIOS is stored in some type of read-only-memory, such as ROM, PROM, EPROM, EEPROM or the like, so that the BIOS can be executed when the computer 300 is activated.
When the computer 300 is in operation, the processor 310 is configured to execute software stored within the memory 320, to communicate data to and from the memory 320, and to generally control operations of the computer 300 pursuant to the software. The application 360 and the 0/S 350 are read, in whole or in part, by the processor 310, perhaps buffered within the processor 310, and then executed.
When the application 360 is implemented in software it should be noted that the application 360 can be stored on virtually any computer readable medium for use by or in connection with any computer related system or method. In the context of this document, a computer readable medium may be an electronic, magnetic, optical, or other physical device or means that can contain or store a computer program for use by or in connection with a computer related system or method.
The application 360 can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a “computer-readable medium” can be any means that can store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.
The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the
Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
A single processor or other unit may fulfill the functions of several items recited in the claims.
A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figs. illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the Figs.. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 19215636.2 | Dec 2019 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2020/085394 | 12/10/2020 | WO |
