In the following, a method for delineating an object in an image is disclosed. Such object delineation is useful for video editing applications targeting at segmenting an object within an image for various purposes, such as applying special video processing on the object itself or for cloning it into other images. Corresponding device is also disclosed.
This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
Methods are known for image cloning in the video processing domain. Image cloning can be described as creating a new image by duplicating a source image region into the target image. The more seamless it is, the more natural it looks. A common method to perform image cloning is a simple process of cut-and-paste with interactive image editing tools such as PhotoShop from Adobe. It is however generally labor intensive as it is difficult for users to select the source region exactly, i.e. draw a boundary that matches exactly the contours of the objects to be inserted. Moreover, it is difficult to modify the appearance of the cloned region to match the target scene. To solve the problem of seamlessly inserting an object into an image or a video, alpha blending methods have been proposed: foreground pixels are superimposed on background pixels with their opacities given by the alpha matte. Other methods, called gradient methods have also been proposed: unlike alpha blending methods which perform direct blending operation on intensities in the target image, gradient domain methods reconstruct intensities in the pasted area by solving partial differential Poisson equations in the gradient domain, as proposed by Perez et al. in “Poisson image editing” in the Proceedings of ACM SIGGRAPH'03. Poisson cloning works best when the error along the boundary of the cloned region is nearly constant, or changes smoothly. When this is not the case, as for instance when the boundary must cross an object, there is a visible smudging of the error from the boundary into the cloned area. More recently Bie et al. have proposed a method in “user-aware image cloning” in the International Journal of Computer Graphics Volume 29 Issue 6-8. Their method reduces the effect of artifacts, by analyzing the diversity of the boundary around the object of interest and correlating the diversity of the boundary to a probability of presence of an artifact. These techniques however fail when the presence of artifacts are not fully correlated with the boundary diversity around the object of interest.
Using prior art image editing tools, complex object delineation remains a painful task in situations where artifacts are present in the neighborhood of the object to delineate. Prior art techniques generally fail in precisely automatically delineating complex objects in presence of artifacts.
A salient idea is to delineate an object in an image via a touch screen device, possibly a pressure sensitive device, by obtaining a band around the object to delineate wherein some characteristics of the band depend on the presence of artifacts in the vicinity of the object to delineate. These varying characteristics, such as the width or the pressure, are generated by a user creating the band, so as to obtain a transparency factor around the object being delineated.
To this end, a method, performed by a touch screen device, for delineating an object in an image is disclosed. The method comprises obtaining, from a user input, a band around an object wherein at least one characteristic of the band depends on the presence of artifacts in the neighborhood of the object to delineate.
According to a preferred embodiment, the characteristic of the band is its width. According to another particularly advantageous embodiment, the touch screen device is a pressure sensitive touch screen device, and the characteristic of the band comprises the pressure.
According to a particularly advantageous variant, the band comprises an inner boundary, and the method further comprises applying a contour detector to the inner boundary to fit the object to delineate.
According to another particularly advantageous variant, the band comprises an inner and an outer boundary, and the method further comprises shifting the inner boundary in the direction of the outer boundary from a constant value.
According to another particularly advantageous variant, the method further comprises obtaining a transparency factor for each pixel of the band, from a nonlinear function applied to the relative position of the pixel between the inner boundary and the outer boundary.
According to another particularly advantageous variant, the method further comprises obtaining an intermediate boundary located between the inner boundary and the outer boundary, wherein the distance between the intermediate boundary and any of the inner or outer boundary depends on the at least one characteristic.
According to another particularly advantageous variant, the method further comprises obtaining a transparency factor for each pixel of the band, from a nonlinear function applied to the relative position of the pixel between the inner boundary and the outer boundary, wherein the nonlinear function is centered to the intermediate boundary.
In a second aspect, a touch screen device for delineating an object in an image is also disclosed, wherein the touch screen device comprises means for obtaining, from a user input, a band around the object wherein at least one characteristic of the band depends on the presence of artifacts in the neighborhood of the object to delineate. In a preferred embodiment, the touch screen device is a pressure sensitive touch screen device, and the at least one characteristic comprises the pressure.
In another preferred embodiment, the characteristic of the band is its width.
In another preferred embodiment, the band comprises an inner boundary, and the device further comprises means for applying a contour detector to the inner boundary to fit the object to delineate.
In another preferred embodiment, the band further comprises an outer boundary, and the device further comprises means for shifting the inner boundary in the direction of the outer boundary from a constant value.
In another preferred embodiment, the device further comprises means for obtaining a transparency factor for each pixel of the band, from a nonlinear function applied to the relative position of the pixel between the inner boundary and the outer boundary.
In another preferred embodiment, the device further comprises means for obtaining an intermediate boundary located between the inner and the outer boundary, wherein the distance between the intermediate boundary and any of the inner or outer boundary depends on the at least one characteristic.
In another preferred embodiment, the device further comprises means for obtaining a transparency factor for each pixel of the band, from a nonlinear function applied to the relative position of the pixel between the inner and the outer boundary, wherein the nonlinear function is centered to the intermediate boundary.
In a third aspect, a device comprising at least one processor adapted to compute the steps of the delineating method in any of its embodiments is also disclosed.
In a fifth aspect, a computer program product comprising instructions of program code for execution by at least one processor to perform the disclosed methods, is also disclosed.
While not explicitly described, the present embodiments may be employed in any combination or sub-combination. For example, the disclosed method, device, or computer program are not limited to the described characteristics of the band.
Besides, any characteristic or variant described for the delineating method is compatible with a device intended to process the disclosed method, with a computer-readable storage medium storing program instructions, and with a computer program product.
In the drawings, an embodiment of the present invention is illustrated. It shows:
According to a specific and non-limitative embodiment of the invention, the processing device 1 comprises an input 10 configured to receive the images comprising the objects to delineate. The images are obtained from a source. According to different embodiments of the invention, the source belongs to a set comprising:
The inputs 10 and 12 are linked to a processing module 14 configured to delineate an object in an image by obtaining a band with varying characteristics around an object to delineate, and according to the drawing data captured from a drawing mean via the input 12. At least one varying characteristic of the band depends on the presence of artifacts in the neighborhood of the object to delineate. For example the drawing data comprise a finger trajectory on a touch screen device and the at least one varying characteristic, such as for example the width are obtained from the finger trajectory. The band comprises an inner boundary and an outer boundary wherein the inner boundary is the boundary closer to the object to delineate. The processing module 14 is also configured to obtain a transparency factor for each pixel of the band as a nonlinear function applied to the relative position of the pixel between the inner boundary and the outer boundary. The resulting band, and its corresponding set of transparency factors are sent to an output 18 such as a display mean. In a variant the band is sent to a display mean together with the original image. In another variant, the delineated object is merged with another image via for example alpha blending by using the set of transparency factors of the band. More generally any alpha compositing method applied to the delineated object and another background image using the set of transparency factors is compatible with the invention. According to a particular embodiment the display mean is external to the device and the output 18 sends the data to display to an external display mean. According to different embodiments of the invention, the display mean, internal or external, belongs to a set comprising:
In a variant, the band and its corresponding set of transparency factors are stored in a memory. As an example, such information is stored in a remote or in a local memory, e.g. a video memory or a RAM, a hard disk.
According to an exemplary and non-limitative embodiment of the invention, the processing device 1 further comprises a computer program stored in the memory 120. The computer program comprises instructions which, when executed by the processing device 1, in particular by the processor 110, make the processing device 1 carry out the processing method described with reference to
According to exemplary and non-limitative embodiments, the processing device 1 is a device, which belongs to a set comprising:
In a second embodiment, a band is obtained in an image from a finger trajectory on touch screen, equipped with pressure sensing multi-touch and pressure imaging sensors, where the finger pressure on the surface varies depending whether the user wants to indicate the presence of an artifact in the neighborhood of the object to delineate. For example the pressure of the finger indicates the user's confidence in his drawing. In a first variant a high pressure indicates a high level of confidence, meaning there is no artifact in the close neighborhood. A low pressure indicates a low level of confidence, meaning there is an artifact in the close neighborhood of the object. In another example a high pressure indicates a low level of confidence while a low pressure indicates a high level of confidence. Advantageously, the varying width of the band drawn on the sensitive touch screen is an additional varying characteristic of the band that is combined with the varying measured pressure.
The band obtained in the step S2 comprises an inner boundary 401 and an outer boundary 402 as shown in
In the step S4 an intermediate boundary 51, 53, as shown in
In a second variant, the intermediate boundary 51 is located between the median curve and the outer boundary. More precisely, when the varying characteristic indicates the presence of an artifact in the neighborhood, the intermediate boundary is close to the median boundary. At the opposite, when the varying characteristic indicates the absence of artifacts in the neighborhood, the intermediate boundary is close to the outer boundary. In absence of artifacts, the background around the object is relatively uniform and a larger band is preferred so as to provide better visual results while inserting the delineated object in a target image. Indeed, the transparency factor over the band (as described further) varies more smoothly over a larger band. In the presence of an artifact, being closed to the median curve is the best compromise for being away from both the artifact and the delineated object itself. More formally, let C be the intermediate boundary and C(t) the spatial coordinates of the point of the intermediate boundary at curvilinear coordinate t. As for the first variant, L(t) is the distance between the inner and outer boundary at position t. This corresponds to the first embodiment described above where a varying characteristic of the band is its width.
Let also
be the maximum width.
be the minimum width of the band. C(t) is expressed as:
Accordingly if L(t)=Lmax, meaning that the band is at its largest width, C(t)=C1(t). Similarly if L(t)=Lmin, meaning that the band is at its thinnest width, C(t)=C0(t). Between these extreme, the intermediate boundary is located between the outer and median boundary, and its distance to the outer boundary depends on the varying characteristic, here the width of the band.
In the second embodiment described above, where a varying characteristic of the band is the pressure detected by the pressure sensitive touch screen, L(t) is the pressure. More generally, any metric L(t) obtained from the combination of the pressure and the width of the band to obtain the intermediate boundary, according to the equations described above is compatible with the invention.
A transparency factor for each pixel of the band is obtained in the step S6 from a function applied to the relative position of the pixel between the inner boundary and the outer boundary. In a first variant, the function is a linear function 60 as shown in
where a, b, c are parameters of the function. The sigmoid function allows a better differentiation than a linear function, between different areas of the band for obtaining the transparency. For pixels located close to the delineated object their corresponding transparency will be closed to none (ie fully visible), while for pixels located far away from the delineated object their corresponding transparency will be closed to full (the corresponding background, in which the delineated object is inserted, is fully visible). Alpha compositing using transparency factors obtained from a sigmoid function provide better visual results.
Advantageously, the sigmoid function applied to x is centered to the intermediate boundary, in other words the sigmoid function inflection point 621 as shown in
In a third variant, the function for obtaining the transparency factor is a ramp function 610 as shown in
Number | Date | Country | Kind |
---|---|---|---|
14307087.8 | Dec 2014 | EP | regional |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/EP2015/078832 | 12/7/2015 | WO | 00 |