Patent Application
20110182493
The present application hereby claims priority under 35 U.S.C. §119 on German patent application number EP10000730 filed Jan. 25, 2010, the entire contents of which are hereby incorporated herein by reference.
At least one embodiment of the invention generally relates to a method and/or a system for image annotation of images in particular medical images.
In many applications it is useful to annotate images such as medical images of patients. For example diagnosis and treatment planning for patients can be improved by comparing the patients images with clinical images of other patients with similar anatomical and pathological characteristics where the similarity is based on the semantic understanding of the image content. Further, a search in medical image databases can be improved by taking the content of the images into account. This requires the images to be annotated for example by labelling image regions of the image.
The conventional way to annotate images if that a user such as a doctor takes a look at medical images taken from a patient and speaks his comments into a dictaphone to be written by a secretary as annotation text data and stored along with the image in an image database. Another possibility is that the user or doctor himself types the annotation data in a word document stored along with the image in a database. The clinician or doctor is writing natural language reports to describe the image content of the respective image. This conventional way of annotating images has several drawbacks.
The conventional annotation method is time consuming and error prone. Furthermore, every doctor can use his own vocabulary for describing the image content so that the same image can be described by different doctors or users very differently with a different vocabulary.
Another disadvantage is that a user performing the annotation cannot use already existing annotation data so that the annotation of an image can take a lot of time and is very inefficient. Another drawback is that the natural language used by the doctor annotating the image is his own natural language such as German or English. This can cause a language barrier if the clinicians or doctors have different natural languages. For example annotation data in German can only be used by few doctors in the United States or Great Britain.
Furthermore, annotating is an interactive task consuming extensive clinician time and cannot be scaled to large amounts of imaging data in hospitals. On the other hand automated image analysis while being very scalable does not leverage standardized semantics and thus cannot be used across specific applications. Since the clinician is writing natural language reports to describe the image content of the respective image a direct link with the image content lacks. Often common vocabulary from biomedical and ontology is used, however the labelling is still manual, time consuming and therefore not accepted by users.
Accordingly, at least one embodiment of the present invention provides a method and/or a system for image annotation which overcomes at least one of the above-mentioned drawbacks and which provides an efficient way of annotating images.
At least one embodiment of the invention provides an image annotation system for annotation of images comprising:
The image annotation system according to at least one embodiment of the present invention increases the efficiency of annotation by using an image parser which can be run on an image parsing system.
The image annotation system can be used for annotation of any kind of images in particular medical images taken from a patient.
The image annotation system according to at least one embodiment of the present invention can be used also used for annotating other kinds of images such as images taken from complex apparatuses to be developed or images to be evaluated by security systems.
In a possible embodiment of the image annotation system according to the present invention the image database stores a plurality of two-dimensional or three-dimensional images.
In a possible embodiment of the image annotation system according to the present invention the image parser segments the image into disjoint image regions each being annotated with at least one class or relation of a knowledge database.
In a possible embodiment of the image annotation system according to the present invention the knowledge database stores linked ontologies comprising classes and relations.
In a possible embodiment of the image annotation system according to the present invention the image parser segments the image by means of trained detectors provided to locate and delineate entities of the image.
In a possible embodiment of the image annotation system according to the present invention annotation data of the image is updated by way of the user terminal by validation, removal or extension of the annotation data retrieved from the annotation database of the image parser.
In a possible embodiment of the image annotation system according to the present invention each user terminal has a graphical user interface comprising input means for performing an update of annotation data of selected image regions of the image or for marking image regions and output means for displaying annotation data of selected image regions of the image.
In a possible embodiment of the image annotation system according to the present invention the user terminal comprises context support means which associate automatically an image region marked by a user with an annotated image region, said annotated image region being located inside the marked image region or the marked region being located within the annotated image region or if no matching annotated image region can be found, it can be associated with the closest nearby annotated image region.
In a possible embodiment of the image annotation system according to the present invention the knowledge database stores Radlex-ontology data, foundational model of anatomy ontology data or ICD10-ontology data.
In a possible embodiment of the image annotation system according to the present invention the image database stores a plurality of two- or three-dimensional images, said images comprising:
magnetic resonance image data provided by a magnetic resonance detection apparatus,
computer tomography data provided by a computer tomograph apparatus,
x-ray image data provided by an x-ray apparatus,
ultrasonic image data provided by an ultrasonic detection apparatus or photographic data provided by a digital camera.
In a possible embodiment of the image annotation system according to the present invention the annotation data stored in the annotation database comprises text annotation data (classes and relation names coming from said ontologies) indicating an entity represented by the respective segmented image region of the image.
In a possible embodiment of the image annotation system according to the present invention the annotation data further comprises parameter annotation data indicating at least one physical property of an entity represented by the respective segmented image region of the image.
In an embodiment of the image annotation system according to the present invention the parameter annotation data comprises a chemical composition, a density, a size or a volume of an entity represented by the respective segmented image region of said image.
In a possible embodiment of the image annotation system according to the present invention the annotation data further comprises video and audio annotation data of an entity represented by the respective segmented image region of the image.
In a possible embodiment of the image annotation system according to the present invention the image database stores a plurality of two-dimensional or three-dimensional medical images which are segmented by means of trained detectors of said image parser into image regions each representing at least one anatomical entity of a human body of a patient.
In an embodiment of the image annotation system according to the present invention the anatomical entity comprises a landmark point, an area or a volume or organ within a human body of a patient.
In an embodiment of the image annotation system according to the present invention the annotated data of at least one image of a patient is processed by a data processor unit to generate automatically an image finding record of said image.
In an embodiment of the image annotation system according to the present invention the image finding records of images taken from the same patient are processed by the data processing unit to generate automatically a patient report of the patient.
In an embodiment of the image annotation system according to the present invention the image database stores a plurality of photographic data provided by digital cameras, wherein the photographic images are segmented by means of trained detectors of the image parser into image regions each representing a physical entity.
At least one embodiment of the invention further provides an image annotation system for annotation of medical images of patients, said system comprising:
At least one embodiment of the invention further provides an apparatus development system for development of at least one complex apparatus having a plurality of interlinked entities said development system comprising an image annotation system for annotation of images comprising:
At least one embodiment of the invention further provides a security system for detecting at least one entity within images, said security system having an image annotation system for annotation of images comprising:
At least one embodiment of the invention further provides a method for annotation of an image comprising the steps of:
At least one embodiment of the invention further provides an annotation tool for annotation of an image, said annotation tool loading at least one selected image from an image database and retrieving corresponding annotation data of segmented image region of said image from an annotation database for further annotation.
At least one embodiment of the invention further provides a computer program comprising instructions for performing such a method.
At least one embodiment of the invention further provides a data carrier which stores such a computer program.
In the following possible embodiments of the system and method for performing image annotation are described with reference to the enclosed figures:
Various example embodiments will now be described more fully with reference to the accompanying drawings in which only some example embodiments are shown. Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. The present invention, however, may be embodied in many alternate forms and should not be construed as limited to only the example embodiments set forth herein.
Accordingly, while example embodiments of the invention are capable of various modifications and alternative forms, embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit example embodiments of the present invention to the particular forms disclosed. On the contrary, example embodiments are to cover all modifications, equivalents, and alternatives falling within the scope of the invention. Like numbers refer to like elements throughout the description of the figures.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments of the present invention. As used herein, the term “and/or,” includes any and all combinations of one or more of the associated listed items.
It will be understood that when an element is referred to as being “connected,” or “coupled,” to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being “directly connected,” or “directly coupled,” to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between,” versus “directly between,” “adjacent,” versus “directly adjacent,” etc.).
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments of the invention. As used herein, the singular forms “a,” “an,” and “the,” are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the terms “and/or” and “at least one of” include any and all combinations of one or more of the associated listed items. It will be further understood that the terms “comprises,” “comprising,” “includes,” and/or “including,” when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be noted that in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed substantially concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.
Spatially relative terms, such as “beneath”, “below”, “lower”, “above”, “upper”, and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, term such as “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein are interpreted accordingly.
Although the terms first, second, etc. may be used herein to describe various elements, components, regions, layers and/or sections, it should be understood that these elements, components, regions, layers and/or sections should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or section from another region, layer, or section. Thus, a first element, component, region, layer, or section discussed below could be termed a second element, component, region, layer, or section without departing from the teachings of the present invention.
As can be seen from
The image acquisition apparatus 4 connected to the image parser 2 can be formed by a conventional digital camera or other image acquisition apparatuses, in particular a magnetic resonance detection apparatus, a computer tomograph apparatus, an x-ray apparatus or an ultrasonic machine. The magnetic resonance image data provided by a magnetic resonance scanning apparatus, the computer tomography data provided by a computer tomograph apparatus, the x-ray image data provided by an x-ray apparatus, the ultrasonic data provided by an ultrasonic machine and the photographic data provided by a digital camera are supplied to the image parser 2 of the image parsing system 6 and stored in the image database 3 for annotation.
The image database 3 can store a plurality of two-dimensional or three-dimensional images of the same or different type. The image parsing system 6 is connected via a network 7 to a knowledge database 8. The knowledge database 8 stores at least one ontology or several linked ontologies comprising classes and relations. Further, the image annotation system 1 according to the present invention comprises at least one user terminal 9-i which loads at least one selected image from the image database 3 and retrieves the corresponding annotation data of all segmented image regions of the image from the annotation database 5 for further annotation of the image. The user terminals can be a client computer that is connected to a local area or a wide area network 7. In a possible embodiment the user terminals 9-i and the knowledge database 8 and the image parsing system 6 are connected to the internet forming the network 7.
In the embodiment shown in
The image parsing system 6 as shown in
In a possible embodiment the user 11 terminal 9-i can comprise context support means which associate automatically an image region marked by a user with an annotated image region wherein the annotated image region can be located inside the marked image region or the marked image region can be located within the annotated image region or if no matching annotated image region can be found, it can be associated with the closest nearby annotated image region.
In a medical application the knowledge database 8 can store Radlex-ontology data, foundational model of anatomy ontology data or ICD10 ontology data. The knowledge database 8 can be connected as shown in
An ontology includes classes and relations. These are formed by predefined text data such as “heart”, i.e. does designate an entity. A relation, for instance, indicates whether one organ is located e.g. “above” another organ, for example, an organ A is located above organ B. Classes of ontologies are called also concepts and relations of ontologies are sometimes also called slots. By using such ontologies it is for example possible to use application programs which can automatically verify a correctness of a statement within a network of interrelated designations. Such a program can for instance verify or check whether an organ A can possibly be located above another organ B i.e. a consistency check of annotation data can be reformed. This consistency check can disclose inconsistencies or hidden inconsistencies between annotation data so that a feedback to the annotating person can be generated. Furthermore, it is possible by providing further rules or relations to generate additional knowledge data which can be added for instance in case of a medical ontology later. In a possible embodiment the system can by itself detect that an entity has a specific relation to another entity. For example, the system might find out that organ A has to be located above another organ B by deriving this knowledge or relation from other relations.
For a text annotation data primarily predefined texts of the ontologies can be used. By this multi-linguality or generation of further knowledge a broader use of the annotated images is possible. For example, it is possible that in the future a further ontology is added which describes a specific disease which is connected to the existing ontologies. In this case it is possible to find images of patients relating to this specific disease, which might have not been known at the time when the annotation was performed.
The image parser 2 segments an image into disjoint image regions each image being annotated with at least one class or relation of the knowledge database 8. The image parser 2 segments the image by means of trained detectors provided to locate and delineate entities of the respective image. The detectors can be trained by means of a plurality of images of the same entity such as an organ of the human body. For example, a detector can be trained by a plurality of images showing hearts of different patients so that the detector can recognize after the training a heart within a thorax picture of a patient.
The annotation data stored in an annotation database 5 can comprise text annotation data indicating an entity represented by the respective segmented image region of the image. In a possible embodiment the annotation data not only comprises text annotation data, e.g., defined texts coming from said ontologies, but comprises also parameter annotation data indicating at least one physical property of an entity represented by the respective segmented image region of the image. Such parameter annotation data can comprise for example a chemical composition, a density, a size or a volume of an entity represented by the respective segmented image region of the image. The annotation data in particular the parameter annotation data can either be input by the user such as the doctor 11 shown in
In a further possible embodiment the annotation data does not only comprise text annotation data or parameter annotation data but also video and audio annotation data of an entity represented by the respective segmented image region of the image.
In a possible embodiment the image database 3 stores a plurality of two- or three-dimensional images of a patient 10 which are segmented by means of trained detectors of the image parser 2 into image regions each representing at least one anatomical entity of the human body of the patient 10. These anatomical entities can for example comprise landmarks, areas or volumes or organs within a human body of the patient 10.
The annotated data of at least one image of a patient 10 such as shown in
The terms of the annotation data or annotated data are derived from ontologies stored in the knowledge database 8. The terms can be the names of classes within the ontology such as the Radlex ontology. Each entity such as an anatomical entity has a unique designation or corresponding term. In a possible embodiment a finding list is stored together with the image region information data in the annotation database 5.
In first step S1 an image retrieved from an image database 3 is parsed and segmented by means of trained detectors into image regions. Each segmented image region is annotated automatically with annotation data and stored in the annotation database 5.
In a further step S2 for an image selected from the image database 3 annotation data of all segmented image regions of the image is retrieved from the annotation database 5 for further annotation of the selected image.
The parsing of the image in step S1 is performed by the image parser 2 of the annotation system 1 as shown in
The user terminal 9-i as shown in
The efficiency of a manual annotation process can be increased by using automatisms realized by a context support unit 21. The context support unit 21 can automatically label image regions selected by the user 11. If the user 11 marks an image region within an already defined image region the context support unit 21 can automatically associate it with the annotation data of the outer image regions. This image region can be generated by the image parsing system 6 or specified by the user 11. In the same manner the context support unit 21 can associate a marked image region outside of any other image region with the nearest already annotated image region. The system 1 also enabled the user 11 to label arbitrary manually specified image regions. Since knowledge databases 8, for example in medical applications, can have a high volume a semantic filter unit 22 can be provided which schedules information about the current context from the context support unit 21, i.e. the current image regions. The semantic filter unit 14 can return a filtered, context related list of probable class and relation names coming ontology. In a possible embodiment the context support unit 21 and the semantic filter unit 14 do not directly query the knowledge database 8 but the use of a mediator instance, i.e. a knowledge access unit 23 which enables more powerful queries using high level inference strategies. In a possible embodiment for controlling the image parsing system 6, a maintenance unit 24 can be provided. The image annotation system 1 as shown in
For example the image parser 2 can segment the image by way of trained detectors for an organ A, B, C to locate and delineate these anatomical images. Accordingly, in this simple example shown in
The image annotation system 1 according to an embodiment of the present invention can also be used in the process of development of a complex apparatus or prototype comprising a plurality of interlinked electromechanical entities. Such a complex apparatus can be for example a prototype of a car or automobile. Accordingly, the image annotation system 1 can be used in a wide range of applications such as annotation of medical images but also in security systems or development systems.
The patent claims filed with the application are formulation proposals without prejudice for obtaining more extensive patent protection. The applicant reserves the right to claim even further combinations of features previously disclosed only in the description and/or drawings.
The example embodiment or each example embodiment should not be understood as a restriction of the invention. Rather, numerous variations and modifications are possible in the context of the present disclosure, in particular those variants and combinations which can be inferred by the person skilled in the art with regard to achieving the object for example by combination or modification of individual features or elements or method steps that are described in connection with the general or specific part of the description and are contained in the claims and/or the drawings, and, by way of combineable features, lead to a new subject matter or to new method steps or sequences of method steps, including insofar as they concern production, testing and operating methods.
References back that are used in dependent claims indicate the further embodiment of the subject matter of the main claim by way of the features of the respective dependent claim; they should not be understood as dispensing with obtaining independent protection of the subject matter for the combinations of features in the referred-back dependent claims. Furthermore, with regard to interpreting the claims, where a feature is concretized in more specific detail in a subordinate claim, it should be assumed that such a restriction is not present in the respective preceding claims.
Since the subject matter of the dependent claims in relation to the prior art on the priority date may form separate and independent inventions, the applicant reserves the right to make them the subject matter of independent claims or divisional declarations. They may furthermore also contain independent inventions which have a configuration that is independent of the subject matters of the preceding dependent claims.
Further, elements and/or features of different example embodiments may be combined with each other and/or substituted for each other within the scope of this disclosure and appended claims.
Still further, any one of the above-described and other example features of the present invention may be embodied in the form of an apparatus, method, system, computer program, computer readable medium and computer program product. For example, of the aforementioned methods may be embodied in the form of a system or device, including, but not limited to, any of the structure for performing the methodology illustrated in the drawings.
Even further, any of the aforementioned methods may be embodied in the form of a program. The program may be stored on a computer readable medium and is adapted to perform any one of the aforementioned methods when run on a computer device (a device including a processor). Thus, the storage medium or computer readable medium, is adapted to store information and is adapted to interact with a data processing facility or computer device to execute the program of any of the above mentioned embodiments and/or to perform the method of any of the above mentioned embodiments.
The computer readable medium or storage medium may be a built-in medium installed inside a computer device main body or a removable medium arranged so that it can be separated from the computer device main body. Examples of the built-in medium include, but are not limited to, rewriteable non-volatile memories, such as ROMs and flash memories, and hard disks. Examples of the removable medium include, but are not limited to, optical storage media such as CD-ROMs and DVDs; magneto-optical storage media, such as MOs; magnetism storage media, including but not limited to floppy disks (trademark), cassette tapes, and removable hard disks; media with a built-in rewriteable non-volatile memory, including but not limited to memory cards; and media with a built-in ROM, including but not limited to ROM cassettes; etc. Furthermore, various information regarding stored images, for example, property information, may be stored in any other form, or it may be provided in other ways.
Example embodiments being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the present invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10000730 | Jan 2010 | EP | regional |
