Patent Grant
11263732
The invention relates to an image processing apparatus and a method for applying a mask to an object in an image. The invention further relates to a workstation and imaging apparatus comprising the image processing apparatus set forth, and to a computer program product for causing a processor system to perform the method set forth.
In the fields of image evaluation and image display, it may be desirable to mask an object in an image. A reason for this is that the user may otherwise be distracted or hindered by the object when viewing the image. By masking the object, the visibility of the object in the image is reduced. As a result, the user can more easily direct his attention to other areas in the image, e.g., to another object or to adjacent areas of the object.
It is known to automatically mask an object in an image. For example, U.S. 2011/0123074 A1 discloses a system and method for altering the appearance of an artificial object in a medical image. An artificial object is first identified in the medical image, such as identifying a breast implant in a mammography image. The prominence of the artificial object is then reduced, e.g., by suppressing the brightness or masking the artificial object out altogether. The resulting medical image with the altered artificial object is then displayed to a user. It is said that, as a result, the medical image can be more accurately analyzed without requiring the user to adjust the image on his or her own.
A problem of the aforementioned system and method is that it is difficult for a user to determine whether the automatic masking of the object was erroneous.
It would be advantageous to have an improved apparatus or method enabling a user to more easily determine whether the automatic masking of the object was erroneous.
To better address this concern, a first aspect of the invention provides an image processing apparatus for applying a mask to an object, comprising an input for obtaining an image, a processor for (i) detecting the object in the image, and (ii) applying the mask to the object in the image for obtaining an output image, and the processor being arranged for said applying the mask to the object by (j) establishing an object contour of the object, (jj) generating, based on the object contour, a mask being smaller than the object, and (jjj) positioning the mask over the object for masking a body of the object while keeping clear a border area of the object.
In a further aspect of the invention, a workstation and an imaging apparatus is provided comprising the image processing apparatus set forth.
In a further aspect of the invention, a method is provided for applying a mask to an object, comprising obtaining an image, detecting the object in the image, and applying the mask to the object in the image for obtaining an output image by (j) establishing an object contour of the object, (jj) generating, based on the object contour, a mask being smaller than the object, and (jjj) positioning the mask over the object for masking a body of the object while keeping clear a border area of the object.
In a further aspect of the invention, a computer program product comprising instructions for causing a processor system to perform the method set forth.
The input obtains an image which comprises an object. As a result, when displaying the image, the object is visible to a user. The processor detects the object in the image, and applies a mask to the object. The mask is image data intended for reducing the visibility of the object in the image. By applying the mask to the object, the image data of the mask is inserted over, or affects the visibility of the image data of the object. As a result, when displaying the output image, the visibility of the object in the output image is reduced. The processor is arranged for applying the mask to the object in the following manner. Firstly, an outline of the object is determined. The outline indicates a size of the object.
Based on the outline of the object, the mask is generated as being smaller than the object. As a result, the mask does not cover the entire object. Finally, the mask is positioned such over the object that a border area of the object is kept visible, whereas a bulk of the object is covered by the mask. As a result, the visibility of the bulk of the object in the output image is reduced, while the visibility of the border area of the object is not affected.
By masking the body of the object while keeping clear the border area of the object, the user is less distracted or hindered by the object as the bulk of the object is reduced in visibility. However, the border area of the object is kept clear of the mask, and is thus not reduced in visibility or otherwise modified as a result of applying the mask to the object.
The invention is partially based on the recognition that it is convenient for a user to obtain automatic masking of an object in an image, but that the automatic masking of the object may be erroneous due to, e.g., failure to detect the object correctly. As a result, the mask may cover adjacent areas of the object that may be of relevance to the user. The inventors have recognized that a user, when viewing the output image, is typically unable to determine whether the automatic masking of the object was erroneous and thus whether adjacent areas of the object were covered. A reason for this is that determining whether the mask is applied correctly to the object involves comparing the mask to the object, which is now hindered by the object being reduced in visibility by said masking of the object.
By keeping clear the border area of the object in the automatic masking of the object, the user may, from seeing the border area of the object, infer that the mask is applied such that no adjacent areas of the object are covered. Similarly, when no border area of the object is visible, the user may infer that the automatic masking of the object was erroneous, and thus that adjacent areas of the object may be covered. Moreover, the effect of masking is generally maintained as the body of the object is masked and thus reduced in visibility.
Advantageously, relevant information in adjacent areas of the object is preserved. Advantageously, relevant information in the border area of the object is preserved. Advantageously, a user is more likely to rely on the automatic masking of the object knowing that he may infer from the output image whether said automatic masking was erroneous. Advantageously, it is not needed for the automatic masking of the object to be 100% reliable, as errors in said automatic masking that may result in adjacent areas of the object being covered are easily noticeable by the user. Advantageously, when the image is a medical image, diagnostic information in the border area of the object is kept free, i.e., not masked.
Optionally, the processor is arranged for generating the mask by reducing the object contour in size along a direction inwards the object for obtaining a mask contour of the mask. By reducing the object contour in size in an inwards direction, the mask contour is similarly shaped as the object contour while having a smaller size. The mask therefore fits inside the object contour. As a result, when positioning the mask at a suitable position over the object, a border area of the object will automatically be kept clear.
Optionally, the processor is arranged for reducing the object contour in size by applying a morphological erosion technique to the object contour. Morphological erosion techniques are particularly well suited for reducing the object contour in size.
Optionally, the processor is arranged for detecting, in the image, an object gradient constituting a gradual transition between the object and its surroundings for keeping clear the object gradient and the border area of the object between the mask and the object gradient. The transition between the object and its surroundings in the image may be a gradual transition. Hence, the contour of the object is not constituted by a line being infinitely thin, but rather by an area in the image corresponding to said gradual transition. A reason for this may be that the resolution of an imaging device used in acquiring the image was limited, e.g. due to a limited size of detector elements. The object gradient corresponds neither fully to the object nor its surroundings, but rather is a gradual transition between both. The border area of the object is thus located inwards of the object gradient. The object gradient may contain relevant information for the user. By detecting the object gradient, it is known where in the image the gradual transition between the object and its surroundings is located. Accordingly, the object gradient and the border area of the object are both kept clear.
Optionally, the processor is arranged for establishing the border area of the object having a displayed width between 2 mm and 10 mm when displaying the output image on a display. The processor thus establishes the border area of the object as having a width, when measured on the display used for displaying the output image, between 2 mm and 10 mm. Said range constitutes a suitable compromise between showing a large enough border area of the object so as to be visible to a user, and masking an as large as possible portion of the object for reducing the overall visibility of the object.
Optionally, the image processing apparatus further comprises a user input for enabling a user determine a zoom factor for zooming in or out of the output image, and the processor is arranged for generating the mask based on the zoom factor for maintaining a displayed width of the border area of the object when displaying the output image on a display based on the zoom factor. Zooming in or out of the output image typically results in a displayed width of structure increasing or decreasing proportionally with the zoom factor. This may be undesired for the border area. For example, when zooming out, the displayed width of the border area may decrease such that the border area is not clearly visible to the user anymore. Similarly, when zooming in, the displayed width of the border area may become unnecessarily large, resulting in the user being distracted or hindered by the border area. By using the zoom factor to maintain the displayed width of the border area, e.g., at a constant value or within a limited range, it is avoided that, as a result of zooming, the border area is not clearly visible to the user anymore or becomes unnecessarily large.
Optionally, the processor is arranged for, when applying the mask to the object, generating a mask gradient in the output image for establishing a gradual transition between the mask and the object. As such, a gradual transition is introduced between the mask and the object in the output image. Advantageously, the user is not distracted or hindered by a sharp transition between the object and the mask in the output image.
Optionally, the processor is arranged for generating the mask gradient by blending of a border area of the mask with the object. Blending the border area of the mask with the object is particularly well suited for generating the mask gradient.
Optionally, the processor is arranged for (i) detecting, in the image, an object gradient constituting a gradual transition between the object and its surroundings, and (ii) generating the mask gradient as differing in width and/or shape from the object gradient for visually differentiating the object gradient from the mask gradient in the output image. The user may therefore visually distinguish the object gradient from the mask gradient. Advantageously, the user is less likely to erroneously identify the mask gradient as the object gradient. Advantageously, the user may easily identify the mask gradient in the output image for easily identifying synthetically introduced parts of the output image.
Optionally, the image processing apparatus further comprises a user input for enabling a user to determine a width and/or shape of the mask gradient. The user can thus manually determine the width and/or shape of the mask gradient.
Optionally, the processor is arranged for masking the body of the object by reducing a brightness and/or contrast of the body of the object. By reducing a brightness and/or contrast of the body of the object, the visibility of the body of the object is reduced while still providing visual information concerning the body of the object, e.g., edges comprised therein. Advantageously, a user may identify the object despite the body of the object being masked.
It will be appreciated by those skilled in the art that two or more of the above-mentioned embodiments, implementations, and/or aspects of the invention may be combined in any way deemed useful.
Modifications and variations of the workstation, the imaging apparatus, the method, and/or the computer program product, which correspond to the described modifications and variations of the image processing apparatus, can be carried out by a person skilled in the art on the basis of the present description.
A person skilled in the art will appreciate that the method may be applied to multi-dimensional image data, e.g. to two-dimensional (2-D), three-dimensional (3-D) or four-dimensional (4-D) images. A dimension of the multi-dimensional image data may relate to time. For example, a three-dimensional image may comprise a time domain series of two-dimensional images. The image may be a medical image, acquired by various acquisition modalities such as, but not limited to, standard X-ray Imaging, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound (US), Positron Emission Tomography (PET), Single Photon Emission Computed Tomography (SPECT), and Nuclear Medicine (NM). However, the image may also be of any other type, e.g., a geological or astrophysical image.
The invention is defined in the independent claims. Advantageous embodiments are defined in the dependent claims.
These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. In the drawings,
The apparatus 110 may further comprise a user input 140 for obtaining selection data 144 from the user. For that purpose, the user input 140 may be connected to a user interface means (not shown in
The operation of the apparatus 110 will be explained in reference to
Applying the mask 171 to the object 124 is performed by firstly establishing an object contour of the object 124. The object contour corresponds to the outline of the object 124 in the image. The object contour is visible in the example shown in
It will be appreciated that the above techniques are known per se from the fields of image analysis and object detection. Moreover, it is noted that various other techniques from said fields may be advantageously used for detecting the object 124 and/or establishing the object contour. In particular, prior knowledge of the object 124 that is to be detected in the image and/or the object contour that is to be established may be advantageously used. For example, the aforementioned lowest-cost circular path may be well-suited for establishing a contour of objects that have an approximately circular shape. For example, when the image is a medical image of a breast, i.e., the medical image is a mammography image, and the object 124 is a breast implant, i.e., an artificial object inserted into the patient, knowledge that the breast implant is approximately circular and located predominantly at a side of the image may be used to improve the reliability, i.e., avoid erroneous results, of detecting the object 124 and/or establishing the object contour.
Having established the object contour, the processor 130 generates, based on the object contour, a mask 171 being smaller than the object 124. Generating the mask 171 as being smaller than the object 124 may comprise reducing the object contour in size along a direction inwards the object 124, thereby obtaining a mask contour of the mask 171. Reducing the object contour in size may comprise establishing the mask contour within the object 124 at a given distance from the object contour and in a direction that is locally orthogonal with respect to the object contour. The distance may be in pixels. Reducing the object contour in size may also comprise applying a morphological erosion technique to the object contour, as is known per se from the field of morphological image processing.
Having obtained the mask contour, the mask 171 itself may be generated by ‘filling in’ the mask contour with mask values, e.g., luminance, chrominance and/or an opacity values. The mask values may be identical. As a result, the mask may be ‘filled in’ with identical mask values. As a result of the above, an explicit representation of the mask 171 may be obtained, e.g., in a memory of the apparatus 110. It will be appreciated, however, that the mask 171 may also be generated implicitly, e.g., being defined by the mask contour and a single mask value, e.g., a luminance, chrominance and/or opacity value.
Having generated the mask 171, the processor 130 positions the mask 171 over the object 124 in order to mask a body of the object 124 while keeping clear a border area 126 of the object 124. The result is shown in
In general, the processor 130 may be arranged for establishing the border area 126 of the object 124 having a displayed width 180 when displaying the output image 162 on the display 150. The displayed width 180 may be a constant value or be set within a limited range. For example, the border area 126 of the object 124 may be established as having a displayed width 180 between 2 mm and 10 mm. The displayed width 180 may be 5 mm. Here, the displayed width 180 is a width of the border area 126 when measured on the display 150. The displayed width 180 typically corresponds to a width that allows a user to perceive the border area 126 on the display 150 and/or allows the user to perceive the object contour as being separated from the mask contour. The displayed width 180 may depend on, e.g., a size of the display 150, a viewing distance of the user to the display 150, etc. Accordingly, the mask 171 may be generated as being smaller than the object 124 by an amount that results in the border area 126 of the object 124 having said displayed width 180. It will be appreciated that the processor 130 may arranged for establishing the border area 126 of the object 124 having a pixel width in the output image 162, with said pixel width having been previously determined to correspond to the displayed width 180 on the display 150. For that purpose, a conversion factor between pixel width and displayed width 180 may be established as a function of, e.g., a resolution of the output image 162 and a size of the display 150.
The processor 130 may be arranged for generating the mask 171 based on the zoom factor for maintaining the displayed width 180 of the border area 126 of the object 124 when displaying the output image 162 on the display 150 based on the zoom factor. A result of the above is shown in
The processor 130 may be arranged for disabling applying the mask 171 to the object 124 when the output image 162 is zoomed out above a certain threshold. The threshold may correspond to a zoom factor in which the object 124 is below a given size when displayed on the display 150. At the given size, the object 124 may not be regarded as hindering the user anymore. Hence, masking the object 124 may be disabled. Moreover, in general, the processor 130 may be arranged for disabling applying the mask 171 to the object 124 when the object is below said given size when displayed on the display 150.
In general, the processor 130 may be arranged for determining or modifying the width and/or shape of the mask gradient 173 based on a zoom factor used for displaying the output image 163, or based on a zoom factor used for displaying a portion thereof, e.g., in a virtual magnifying glass. Moreover, the processor 130 may be arranged for determining or modifying the width and/or shape of the mask gradient 173 based on a location of the mask gradient 173 in the output image 163. For example, when the mask gradient 173 is located at the edge of the output image 163, the width of the mask gradient 173 may be zero.
The user input 140 may be arranged for enabling a user to determine a width and/or shape of the mask gradient 173. The user interface commands 142 may be indicative of the width and/or shape of the mask gradient 173. Said width and/or shape may be received by the processor 130 from the user input 140 in the form of the selection data 144. The shape may be adapted to the actual appearance of the mask 172 and/or the object 124. For example, when the user determines the mask gradient 173 to be ‘S’-shaped, a maximum and/or a minimum of the ‘S’-shape may set to correspond in appearance with mask 172 and/or the object 124. For example, the shape may be scaled or adjusted for obtaining a gradual transition between a luminance of the mask 172 and a luminance of the object.
For obtaining the result shown in
In general, it is noted that the term image refers to a multi-dimensional image, such as a two-dimensional (2-D) image or a three-dimensional (3-D) image. Here, the term 3-D image refers to a volumetric image, i.e., having three spatial dimensions. The image is made up of image elements. The image elements may be so-termed picture elements, i.e., pixels, when the image is a 2-D image. The image elements may also be so-termed volumetric picture elements, i.e., voxels, when the image is a volumetric image. The term value in reference to an image element refers to a displayable property that is assigned to the image element, e.g., a value of a pixel may represent a luminance and/or chrominance of the pixel, or may indicate an opacity or translucency of a voxel within the volumetric image.
It will be appreciated that the invention also applies to computer programs, particularly computer programs on or in a carrier, adapted to put the invention into practice. The program may be in the form of a source code, an object code, a code intermediate source and an object code such as in a partially compiled form, or in any other form suitable for use in the implementation of the method according to the invention. It will also be appreciated that such a program may have many different architectural designs. For example, a program code implementing the functionality of the method or system according to the invention may be sub-divided into one or more sub-routines. Many different ways of distributing the functionality among these sub-routines will be apparent to the skilled person. The sub-routines may be stored together in one executable file to form a self-contained program. Such an executable file may comprise computer-executable instructions, for example, processor instructions and/or interpreter instructions (e.g. Java interpreter instructions). Alternatively, one or more or all of the sub-routines may be stored in at least one external library file and linked with a main program either statically or dynamically, e.g. at run-time. The main program contains at least one call to at least one of the sub-routines. The sub-routines may also comprise function calls to each other. An embodiment relating to a computer program product comprises computer-executable instructions corresponding to each processing step of at least one of the methods set forth herein. These instructions may be sub-divided into sub-routines and/or stored in one or more files that may be linked statically or dynamically. Another embodiment relating to a computer program product comprises computer-executable instructions corresponding to each means of at least one of the systems and/or products set forth herein. These instructions may be sub-divided into sub-routines and/or stored in one or more files that may be linked statically or dynamically.
The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a storage medium, such as a ROM, for example, a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example, a hard disk. Furthermore, the carrier may be a transmissible carrier such as an electric or optical signal, which may be conveyed via electric or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted to perform, or used in the performance of, the relevant method.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
This application is a national stage application under 35 U.S.C. § 371 of International Application No. PCT/IB2012/056402 filed on Nov. 14, 2012 and published in the English language on Jun. 6, 2013 as International Publication No. WO/2013/080071, which claims priority to U.S. Application No. 61/563,990 filed on Nov. 28, 2011, the entire disclosures of which are incorporated herein by reference.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/IB2012/056402 | 11/14/2012 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2013/080071 | 6/6/2013 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4787393 | Fukukita | Nov 1988 | A |
| 5103254 | Bell | Apr 1992 | A |
| 5687251 | Erler | Nov 1997 | A |
| 5768413 | Levin | Jun 1998 | A |
| 5896463 | Kuhn | Apr 1999 | A |
| 6094508 | Acharya | Jul 2000 | A |
| 6137898 | Broussard | Oct 2000 | A |
| 6352509 | Kawagishi et al. | Mar 2002 | B1 |
| 7623728 | Avinash | Nov 2009 | B2 |
| 7702149 | Ohkubo et al. | Apr 2010 | B2 |
| 7826683 | Phillips | Nov 2010 | B2 |
| 8180133 | Omi | May 2012 | B2 |
| 8401285 | Rezaee | Mar 2013 | B1 |
| 8717355 | Newton | May 2014 | B2 |
| 8797619 | Kido et al. | Aug 2014 | B2 |
| 9547908 | Kim | Jan 2017 | B1 |
| 20020093670 | Luo | Jul 2002 | A1 |
| 20060173317 | Lee | Aug 2006 | A1 |
| 20070055178 | Verre | Mar 2007 | A1 |
| 20070188510 | Kokemohr | Aug 2007 | A1 |
| 20080002872 | Gatesoupe | Jan 2008 | A1 |
| 20090016580 | Yamamichi | Jan 2009 | A1 |
| 20090099563 | Ciaccio | Apr 2009 | A1 |
| 20090273608 | Buchanan | Nov 2009 | A1 |
| 20100046837 | Boughorbel | Feb 2010 | A1 |
| 20100061613 | Choi et al. | Mar 2010 | A1 |
| 20100150419 | Dutta | Jun 2010 | A1 |
| 20100246989 | Agrawal | Sep 2010 | A1 |
| 20100271466 | Newton | Oct 2010 | A1 |
| 20110002519 | Tomisaki et al. | Jan 2011 | A1 |
| 20110123074 | Nie | May 2011 | A1 |
| 20110123075 | Nie et al. | May 2011 | A1 |
| 20110190632 | Kim | Aug 2011 | A1 |
| 20110200238 | Garud et al. | Aug 2011 | A1 |
| 20110206250 | McGinnis | Aug 2011 | A1 |
| 20110274336 | Nielsen | Nov 2011 | A1 |
| 20130077844 | Vija | Mar 2013 | A1 |
| 20140334680 | Serlie et al. | Nov 2014 | A1 |
| 20150078640 | Guo | Mar 2015 | A1 |
| Number | Date | Country |
|---|---|---|
| 102007057013 | May 2009 | DE |
| S61161583 | Jul 1986 | JP |
| 2002374403 | Dec 2002 | JP |
| 2009106335 | May 2009 | JP |
| Entry |
|---|
| Bakic, Predrag R., et al. “Mammogram synthesis using a 3D simulation. I. Breast tissue model and image acquisition simulation.” Medical physics 29.9 (2002): 2131-2139. (Year: 2002). |
| Meng, X. et al., “Enhancement Strategy for Region of Interest Based on Attentional Shroud”, Computer Engineering, vol. 37 No. 4, Abstract. |
| Number | Date | Country | |
|---|---|---|---|
| 20140334680 A1 | Nov 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61563990 | Nov 2011 | US |
