Patent Grant
10127643
The present invention relates to an inpainting device and method using the segmentation of a reference region, and more particularly to an inpainting device and method using the segmentation of a reference region that can perform inpainting without segmenting an inpainting region and after segmenting an inpainting region in multiple stages and that can provide information about the difference between the results of respective types of inpainting to the user.
With the development of digital signal processing technology, virtual advertising that inserts a virtual image into a video has been popularized, and indirect advertising has established itself as an effective advertising means. Accordingly, interest has been rapidly increasing in virtual indirect advertising that combines a virtual advertisement with an indirect advertisement and naturally inserts an advertisement image into a previously generated video.
In connection with virtual indirect advertising, inpainting technology that naturally inserts a virtual image into a previously generated video or that eliminates part of an image of a previously generated video and performs natural restoration to achieve harmonization with a background is fundamental.
In inpainting technology, a reference region to be used for restoration is required to restore a target region. When a reference region is designated in a non-limiting manner, as in a case where the reference region is designated as an overall video frame, a problem arises in that computational load rapidly increases. When overall inpainting is performed regardless of the correction between a reference region and a region to be restored even when the reference region is limited, a problem arises in that an unintentional inpainting result occurs.
Accordingly, there is a growing need for new inpainting technology that performs inpainting without segmenting an inpainting region and after segmenting an inpainting region in multiple stages and that provides information about the difference between the results of respective types of inpainting to the user, thereby overcoming a problem in which an awkward inpainting result occurs when inpainting is performed in a single mode.
Related preceding technology includes Korean Patent Application Publication No. 10-2013-0056407 published on May 30, 2013 (entitled “Inpainting System and Method for H.264 for Error Concealment Image”).
An object of the present invention is to perform inpainting without segmenting an inpainting region and after segmenting an inpainting region in multiple stages and provide information about the difference between the results of respective types of inpainting to the user, thereby overcoming a problem in which an awkward inpainting result occurs when inpainting is performed in a single mode.
Furthermore, an object of the present invention is to provide various types of inpainting result information to a user, receive feedback from the user, and incorporate the feedback into a final inpainting result, thereby performing more natural inpainting.
In order to accomplish the above objects, the present invention provides an inpainting device using the segmentation of a reference region, including: a region determination unit configured to determine an inpainting target region present within video image content and a reference region composed of pixels present within a preset distance from the boundary pixels of the inpainting target region; an inpainting group setting unit configured to set inpainting groups composed of respective pairs of segmented reference regions obtained by segmenting the reference region using pixel values within the reference region and segmented target regions obtained by segmenting the target region so as to correspond to the segmented reference region; an inpainting unit configured to perform segmentation inpainting adapted to perform inpainting on a segmented target region using the segmented reference region on an inpainting group basis, and non-segmentation inpainting adapted to perform inpainting on the target region using the reference region; and an information provision unit configured to visually provide the result of the segmentation inpainting, the result of the non-segmentation inpainting, and information about the difference between the results of the segmentation inpainting and the non-segmentation inpainting to the user.
In this case, the inpainting group setting unit may further include a group set generation unit configured to differently generate group sets based on the number of inpainting groups; and the group set generation unit may generate a first group set in which the number of inpainting groups has been set to the first number of segmentations and generate a second group set in which the number of inpainting groups has been set to the second number of segmentations different from the first number of segmentations.
In this case, the segmentation inpainting may include first segmentation inpainting adapted to be performed on the first group set and second segmentation inpainting adapted to be performed on the second group set.
In this case, the result of the segmentation inpainting may include the results of the first segmentation inpainting and the second segmentation inpainting.
In this case, the information provision unit may further visually display information about the difference between the results of the first segmentation inpainting and the second segmentation inpainting.
In this case, the group set generation unit additionally may generate the second group set in which the number of inpainting groups is equal to or smaller than the reference number of groups when the number of inpainting groups of the first group set exceeds the preset reference number of groups.
In this case, the group set generation unit may generate the second segmentation set by merging any two or more of the inpainting groups of the first group set when the first number of segmentations exceeds the preset reference number of segmentations.
In this case, the inpainting group setting unit may calculate the edge values of pixels within the reference region, may determine whether a variation in edge values is below a preset reference variation, and may group the pixels within the reference region and set the grouped pixels as the segmented reference region based on the result of the determination.
In this case, the inpainting group setting unit may determine whether the variation in edge values is below the preset reference variation, and may group the pixels within the reference region and set the grouped pixels as the segmented reference region based on the result of the determination.
In this case, the inpainting group setting unit may determine whether the pixel values within the reference region fall within a preset range of values similar to a reference pixel value, and may set pixels falling within the similar range as an identical inpainting group.
In this case, the inpainting group setting unit may segment the target region using the pattern of the segmentation boundary line of the reference region.
Furthermore, the present invention provides an inpainting method using the segmentation of a reference region, including: determining an inpainting target region present within video image content and a reference region composed of pixels present within a preset distance from the boundary pixels of the inpainting target region; setting inpainting groups composed of respective pairs of segmented reference regions obtained by segmenting the reference region using pixel values within the reference region and segmented target regions obtained by segmenting the target region so as to correspond to the segmented reference region; performing segmentation inpainting adapted to perform inpainting on a segmented target region using the segmented reference region on an inpainting group basis and non-segmentation inpainting adapted to perform inpainting on the target region using the reference region; and visually providing the result of the segmentation inpainting, the result of the non-segmentation inpainting, and information about the difference between the results of the segmentation inpainting and the non-segmentation inpainting to the user.
In this case, setting the inpainting groups may further include differently generating group sets based on the number of inpainting groups; generating the group sets may include generating a first group set in which the number of inpainting groups has been set to the first number of segmentations and generating a second group set in which the number of inpainting groups has been set to the second number of segmentations different from the first number of segmentations.
In this case, the segmentation inpainting may include first segmentation inpainting adapted to be performed on the first group set and second segmentation inpainting adapted to be performed on the second group set.
In this case, the result of the segmentation inpainting may include the results of the first segmentation inpainting and the second segmentation inpainting.
In this case, providing the information about the difference may include further visually displaying information about the difference between the results of the first segmentation inpainting and the second segmentation inpainting.
In this case, generating the group sets may include additionally generating the second group set in which the number of inpainting groups is equal to or smaller than the reference number of groups when the number of inpainting groups of the first group set exceeds the preset reference number of groups.
In this case, generating the group sets may include generating the second segmentation set by merging any two or more of the inpainting groups of the first group set when the first number of segmentations exceeds the preset reference number of segmentations.
According to the present invention, inpainting is performed without segmenting an inpainting region and after segmenting an inpainting region in multiple stages, and information about the difference between the results of respective types of inpainting is provided to the user, thereby overcoming a problem in which an awkward inpainting result occurs when inpainting is performed in a single mode.
Furthermore, according to the present invention, various types of inpainting result information are provided to a user, feedback is received from the user, and the feedback is incorporated into a final inpainting result, thereby performing more natural inpainting.
Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, detailed descriptions of well-known functions and configurations that may make the gist of the present invention unnecessarily obscure will be omitted in the following description and the attached drawings. Furthermore, it should be noted that the same components are designated by the same reference symbols throughout the drawings as much as possible.
The terms and words used in the following specification and the claims should not be restrictively interpreted as having common and dictionary meanings, but should be interpreted as having meanings and concepts in conformity with the technical spirit of the present invention based on the principle that an inventor can appropriately define the concepts of terms in order to describe his or her invention in the best way. Accordingly, since the embodiments described in the present specification and the configurations shown in the drawings merely correspond to the preferred embodiments of the present invention and do not cover all the technical spirit of the present invention, it should be appreciated that there may be various equivalents and modifications that may replace the configurations at the time at which the present application is filed. Furthermore, the terms “first,” “second” and so on are used to describe various components, and are intended to distinguish one component from other components but are not intended to limit these components.
An inpainting device using the segmentation of a reference region according to an embodiment of the present invention is described below.
Referring to
The region determination unit 110 determines an inpainting target region present within video image content and a reference region composed of pixels present within a preset distance from the boundary pixels of the inpainting target region.
In this case, the inpainting target region refers to a region on which inpainting will be performed within the video image content.
In this case, the inpainting target region may be determined by the input of a user.
The inpainting group setting unit 120 sets inpainting groups composed of respective pairs of segmented reference regions obtained by segmenting the reference region using pixel values within the reference region and segmented target regions obtained by segmenting the target region so as to correspond to the segmented reference region.
In this case, the inpainting group setting unit 120 may calculate the edge values of pixels within the reference region, may determine whether a variation in edge values is below a preset reference variation, and may group the pixels within the reference region and set the grouped pixels as the segmented reference region based on the result of the determination.
In this case, the inpainting group setting unit 120 may determine whether the variation in edge values is below the preset reference variation, and may group the pixels within the reference region and set the grouped pixels as the segmented reference region based on the result of the determination.
In this case, the reference variation may be a reference value that is preset so as to determine pixels having large variations in edge values among the pixels within the reference region.
In this case, the inpainting group setting unit 120 may determine whether the pixel values within the reference region fall within a preset range of values similar to a reference pixel value, and may set pixels falling within the similar range as the same inpainting group.
In this case, the reference pixel value may be a reference value that is preset so as to determine pixels having similar pixel values among the pixels within the reference region.
In this case, the reference pixel value may be a reference value that is preset to a pixel value having a predetermined or higher distribution using a distribution histogram of pixel values.
In this case, the inpainting group setting unit 120 may segment the target region using the pattern of the segmentation boundary line of the reference region.
For example, the inpainting group setting unit 120 may segment the target region into rectilinear patterns when the segmentation boundary line of the reference region is a rectilinear pattern, and may also segment the target region into zigzag patterns when the segmentation boundary line of the reference region is a zigzag pattern.
In this case, the inpainting group setting unit 120 may include a group set generation unit configured to differently generate group sets based on the number of inpainting groups.
In this case, the group set is a set in which a method of segmenting the same target region and reference region in various manners has been set.
In this case, the group set generation unit may generate a first group set in which the number of inpainting groups has been set to the first number of segmentations, and may generate a second group set in which the number of inpainting groups has been set to the second number of segmentations different from the first number of segmentations.
For example, the group set generation unit may generate a first group set in which the number of inpainting groups has been set to 2 segmentations and a second group set in which the number of inpainting groups has been set to 3 segmentations.
In this case, the group set generation unit may additionally generate a second group set in which the number of inpainting groups is equal to or smaller than the reference number of groups when the number of inpainting groups of the first group set exceeds the preset reference number of groups.
In this case, the reference number of groups may be the upper limit value of the number of inpainting groups that is preset so as to limit segmentation into an excessively large number of inpainting groups.
For example, the group set generation unit may additionally generates a second group set in which the number of inpainting groups is equal to or smaller than 6 when the number of inpainting groups of the first group set exceeds 6, i.e., the preset reference number of groups.
In this case, the group set generation unit may generate the second segmentation set by merging any two or more of the inpainting groups of the first group set when the first number of segmentations exceeds the preset reference number of segmentations.
For example, the group set generation unit may generate a second group set having a total of 6 inpainting groups by merging any two of the inpainting groups of the first group set when the reference number of groups is 6 and the number of inpainting groups of the first group set is 7.
In this case, the group set generation unit may merge groups in which the representative pixel values of respective inpainting groups are similar.
The inpainting unit 130 performs segmentation inpainting adapted to perform inpainting on a segmented target region using the segmented reference region on an inpainting group basis, and non-segmentation inpainting adapted to perform inpainting on a target region using the reference region.
In this case, the non-segmentation inpainting is inpainting that is performed without segmenting the reference region and the target region.
In this case, the segmentation inpainting may include first segmentation inpainting adapted to be performed on the first group set, and second segmentation inpainting adapted to be performed on the second group set.
In this case, the result of the segmentation inpainting may include the result of the first segmentation inpainting and the result of the second segmentation inpainting.
The information provision unit 140 visually provides the result of the segmentation inpainting, the result of the non-segmentation inpainting, and information about the difference between the results of the segmentation inpainting and the non-segmentation inpainting to the user.
In this case, the information provision unit may further visually display information about the difference between the results of the first segmentation inpainting and the second segmentation inpainting.
As described above, the inpainting device using the segmentation of a reference region according to the embodiment of the present invention may perform inpainting without segmenting the inpainting region and after segmenting the inpainting region in multiple stages, and may provide information about the difference between the results of respective types of inpainting to the user. Furthermore, more natural inpainting can be performed by providing various types of inpainting result information to the user, receiving feedback from the user and incorporating the feedback into a final inpainting result.
The determination of regions according to an embodiment of the present invention is described below.
Referring to
It can be seen that the pixels corresponding to a target on which inpainting will be performed are pixels within a target region and pixels inside a target region boundary line 210.
It can be seen that the pixels which are referred to during inpainting are pixels between the target region boundary line 210 and a reference region boundary line 220 that is a preset distance away from the target region boundary line 210 in an outward direction.
In this case, the inpainting target region or target region boundary line 210 may be determined by the input of the user.
The setting of inpainting groups according to an embodiment of the present invention is described below.
Referring to
Referring to
That is, at the step of setting inpainting groups according to the present invention, a region corresponding to a portion outside a desk in
In this case, at the step of setting inpainting groups according to the present invention, it may be determined whether the pixel values within the reference region 320 fall within a preset range of values similar to a reference pixel value, and pixels falling within the range of similar values may be set as the same segmented reference region.
In this case, the reference pixel value may be a reference value that is preset so as to determine pixels having similar pixel values among pixels within the reference region.
For example, when the pixels corresponding to the portion outside the desk in
In this case, the reference pixel value may be a reference value that is preset to a pixel value having a predetermined or higher distribution using a distribution histogram of pixel values.
For example, in
In this case, at the step of setting inpainting groups according to the present invention, the edge values of the pixels within the reference region 320 may be calculated, it may be determined whether a variation in edge values is below a preset reference variation, and the reference region may be segmented based on the result of the determination.
In this case, the reference variation may be a reference value that is preset so as to determine pixels having large variations in edge values among the pixels within the reference region.
For example, in
Referring to
In this case, at the step of setting inpainting groups according to the present invention, the target region 310 may be segmented using the pattern of a boundary line between the segmented reference regions 421 and 422.
For example, at the step of setting inpainting groups according to the present invention, when the boundary line between the segmented reference regions 421 and 422 is a rectilinear pattern, the target region 310 may be segmented into segmented target regions 511 and 512 having a boundary line based on the rectilinear pattern.
That is, at the step of setting inpainting groups according to the present invention, when a boundary line between the segmented reference regions 421 and 422 has a zigzag pattern, the target region 310 may be segmented into the segmented target regions 511 and 512 having a boundary line based on the zigzag pattern.
Referring to
For example, at the step of setting inpainting groups according to the present invention, when the segmented reference region 421 and segmented target region 511 corresponding to each other, the inpainting group 601 composed of a pair of the segmented reference region 421 and segmented target region 511 may be set.
That is, as described in the series of descriptions of
The generation of group sets according to an embodiment of the present invention is described below.
At the step of generating group sets according to the present invention, group sets may be differently generated based on the number of inpainting groups.
In this case, the group set is a set in which a method of segmenting the same target region and reference region in various manners has been set.
Referring to
Referring to
As described above, it can be seen that, at the step of generating group sets according to the present invention, various inpainting group settings that are generated when the reference value, such as a similar pixel value range, varies at the step of setting inpainting groups may be generated as respective group sets.
An inpainting method using the segmentation of a reference region according to an embodiment of the present invention is described below.
Referring to
In this case, the inpainting target region refers to a region on which inpainting will be performed within the video image content.
In this case, the inpainting target region may be determined by the input of a user.
Furthermore, in the inpainting method using the segmentation of a reference region according to the embodiment of the present invention, inpainting groups composed of respective pairs of segmented reference regions obtained by segmenting the reference region using pixel values within the reference region and segmented target regions obtained by segmenting the target region so as to correspond to the segmented reference region are set at step S920.
In this case, at step S920, the edge values of pixels within the reference region may be calculated, it may be determined whether a variation in edge values is below a preset reference variation, and the pixels within the reference region may be grouped and set as the segmented reference region based on the result of the determination.
In this case, at step S920, it may be determined whether the variation in edge values is below the preset reference variation, and the pixels within the reference region may be grouped and set as the segmented reference region based on the result of the determination.
In this case, the reference variation may be a reference value that is preset so as to determine pixels having large variations in edge values among the pixels within the reference region.
In this case, at step S920, it may be determined whether the pixel values within the reference region fall within a preset range of values similar to a reference pixel value, and pixels falling within the similar range may be set as the same inpainting group.
In this case, the reference pixel value may be a reference value that is preset so as to determine pixels having similar pixel values among the pixels within the reference region.
In this case, the reference pixel value may be a reference value that is preset to a pixel value having a predetermined or higher distribution using a distribution histogram of pixel values.
In this case, at step S920, the target region may be segmented using the pattern of the segmentation boundary line of the reference region.
For example, at step S920, the target region may be segmented into rectilinear patterns when the segmentation boundary line of the reference region is a rectilinear pattern, and the target region may be also segmented into zigzag patterns when the segmentation boundary line of the reference region is a zigzag pattern.
In this case, step S920 may further include the step of differently generating group sets based on the number of inpainting groups.
In this case, the group set is a set in which a method of segmenting the same target region and reference region in various manners has been set.
In this case, at the step of generating group sets, a first group set in which the number of inpainting groups has been set to the first number of segmentations may be generated, and a second group set in which the number of inpainting groups has been set to the second number of segmentations different from the first number of segmentations may be generated.
For example, at the step of generating group sets, a first group set in which the number of inpainting groups has been set to 2 segmentations may be generated, and a second group set in which the number of inpainting groups has been set to 3 segmentations may be generated.
In this case, at the step of generating group sets, when the number of inpainting groups of the first group set exceeds the preset reference number of groups, a second group set in which the number of inpainting groups is equal to or smaller than the reference number of groups may be additionally generated.
In this case, the reference number of groups may be the upper limit value of the number of inpainting groups that is preset so as to limit segmentation into an excessively large number of inpainting groups.
For example, at the step of generating group sets, when the number of inpainting groups of the first group set exceeds 6, i.e., the preset reference number of groups, a second group set in which the number of inpainting groups is equal to or smaller than 6 may be additionally generated.
In this case, at the step of generating group sets, when the first number of segmentations exceeds the preset reference number of segmentations, the second segmentation set may be generated by merging any two or more of the inpainting groups of the first group set.
For example, at the step of generating group sets, when the reference number of groups is 6 and the number of inpainting groups of the first group set is 7, a second group set having a total of 6 inpainting groups may be generated by merging any two of the inpainting groups of the first group set.
In this case, at the step of generating group sets, groups in which the representative pixel values of respective inpainting groups are similar may be merged.
Furthermore, in the inpainting method using the segmentation of a reference region according to the embodiment of the present invention, segmentation inpainting adapted to perform inpainting on a segmented target region using the segmented reference region on an inpainting group basis is performed, and non-segmentation inpainting adapted to perform inpainting on a target region using the reference region is performed at step S930.
In this case, the non-segmentation inpainting is inpainting that is performed without segmenting the reference region and the target region.
In this case, the segmentation inpainting may include first segmentation inpainting adapted to be performed on the first group set, and second segmentation inpainting adapted to be performed on the second group set.
In this case, the result of the segmentation inpainting may include the result of the first segmentation inpainting and the result of the second segmentation inpainting.
Furthermore, in the inpainting method using the segmentation of a reference region according to the embodiment of the present invention, the result of the segmentation inpainting, the result of the non-segmentation inpainting, and information about the difference between the results of the segmentation inpainting and the non-segmentation inpainting are visually provided to the user.
In this case, at step S540, information about the difference between the results of the first segmentation inpainting and the second segmentation inpainting may be further visually displayed.
As described above, in the inpainting method using the segmentation of a reference region according to the embodiment of the present invention, inpainting may be performed without segmenting the inpainting region and after segmenting the inpainting region in multiple stages, and information about the difference between the results of respective types of inpainting may be provided to the user. Furthermore, more natural inpainting can be performed by providing various types of inpainting result information to the user, receiving feedback from the user and incorporating the feedback into a final inpainting result.
The steps shown in
The inpainting method using the segmentation of a reference region according to the present invention may be implemented as a program or smart phone app that can be executed by various computer means. In this case, the program or smart phone app may be recorded on a computer-readable storage medium. The computer-readable storage medium may include program instructions, data files, and data structures solely or in combination. Program instructions recorded on the storage medium may have been specially designed and configured for the present invention, or may be known to or available to those who have ordinary knowledge in the field of computer software. Examples of the computer-readable storage medium include all types of hardware devices specially configured to record and execute program instructions, such as magnetic media, such as a hard disk, a floppy disk, and magnetic tape, optical media, such as compact disk (CD)-read only memory (ROM) and a digital versatile disk (DVD), magneto-optical media, such as a floptical disk, ROM, random access memory (RAM), and flash memory. Examples of the program instructions include machine code, such as code created by a compiler, and high-level language code executable by a computer using an interpreter. These hardware devices may be configured to operate as one or more software modules in order to perform the operation of the present invention, and the vice versa.
The inpainting device and method using the segmentation of a reference region according to the present invention are not limited to the configurations and methods of the above-described embodiments, but some or all of the embodiments may be configured to be selectively combined such that the embodiments can be modified in various manners.
According to the present invention, inpainting is performed without segmenting an inpainting region and after segmenting an inpainting region in multiple stages, and information about the difference between the results of respective types of inpainting is provided to the user, thereby overcoming a problem in which an awkward inpainting result occurs when inpainting is performed in a single mode. In particular, when the trend in which interest in virtual indirect advertising that increases an advertising effect by inserting an image of an advertising item into a drama or sports broadcast has been increasing in line with a recent increase in the scale of the advertising market and the fact that inpainting technology that naturally inserts a virtual image into a previously generated video or that eliminates part of an image of a previously generated video and performs natural restoration to achieve harmonization with a background is fundamental are taken into consideration, the utilization of the inpainting device and method using the segmentation of a reference region according to the present invention will be high.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2014-0009197 | Jan 2014 | KR | national |
This present application is a continuation application of a pending U.S. patent application Ser. No. 14/896,888 filed on Dec. 8, 2015 which is a national stage filing under 35 U.S.C § 371 of the PCT application number PCT/KR2014/009924 filed on Oct. 22, 2014, which is based upon and claims the benefit of priority to Korean Patent Application No. 10-2014-0009197 filed on Jan. 24, 2014 in the Korean Intellectual Property Office. The disclosures of the above-listed applications are hereby incorporated by reference herein in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 20040164996 | Criminisi et al. | Aug 2004 | A1 |
| 20080238942 | Sun et al. | Oct 2008 | A1 |
| 20110103706 | Jeong et al. | May 2011 | A1 |
| 20120001902 | Lee et al. | Jan 2012 | A1 |
| 20120224781 | Tai et al. | Sep 2012 | A1 |
| 20130083965 | Joung et al. | Apr 2013 | A1 |
| 20130182184 | Senlet et al. | Jul 2013 | A1 |
| 20130266208 | Lim et al. | Oct 2013 | A1 |
| 20140112548 | Huang et al. | Apr 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 1020100088282 | Aug 2010 | KR |
| 1020110046904 | May 2011 | KR |
| 1020120003232 | Jan 2012 | KR |
| 1020120014876 | Feb 2012 | KR |
| 1020130036514 | Apr 2013 | KR |
| 1020130056407 | May 2013 | KR |
| Entry |
|---|
| Subin Lee et al., Image Inpainting by Band Matching, Seamless Cloning and Area Sub-Division, Journal of Korea Multimedia Society, Feb. 2008, pp. 153-162, vol. 11, No. 2. |
| International Search Report for PCT/KR2014/009924 dated Jan. 16, 2015. |
| Number | Date | Country | |
|---|---|---|---|
| 20180158178 A1 | Jun 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14896888 | US | |
| Child | 15887670 | US |
