The present invention relates to an image processing device, an image processing method, and a program, and more specifically relates to a technique for reducing noise in a digital image.
Priority is claimed on Japanese Patent Application No. 2017-227701, filed Nov. 28, 2017, the content of which is incorporated herein by reference.
In recent years, solid-state imaging sensors such as CCD (Charged-Coupled Device) and CMOS (Complementary Metal-Oxide-Semiconductor) image sensors have rapidly emerged, resulting in the generation of an enormous number of digital images. These digital images may include various types of noise, such as block noise due to compression, or noise caused by an imaging sensor, such as a CCD or CMOS image sensor. Because digital images can be easily processed by computer and the like, various types of noise reduction processing for reducing the noise included in digital images have been proposed (for example, see Patent Document 1).
[Patent Document 1] Japanese Unexamined Patent Application, First Publication No. 2007-42124
The technique disclosed in Patent Document 1 includes processing that performs non-linear conversion processing on a plurality of band-limited images representing each of a plurality of different frequency band components of a digital image. Consequently, the calculation amount is large. For example, the processing may take a long time to apply to a large amount of frame data such as that of a moving image.
The present invention has been made in view of these aspects. An exemplary object of the present invention is to provide a noise suppression technique having a high noise reduction effect with respect to a calculated amount.
A first exemplary aspect of the present invention is an image processing device. The device includes: a region setting unit configured to set a region including a target pixel selected from among a plurality of pixels that form an image; a modeling unit configured to model, with a curved surface, a pixel value distribution of a plurality of pixels that form the region; and a pixel value changing unit configured to replace a pixel value of the target pixel by a value of a position that corresponds to the target pixel on the modeled curved surface.
The pixel value of the plurality of pixels that form the image may hold three values that correspond to the respective axes of a three-dimensional color space. The modeling unit may model, with the curved surface, a value distribution for each of the three values corresponding to the respective axes of the three-dimensional color space for the plurality of pixels that form the set region. The pixel value changing unit may replace the three values of the pixel value of the target pixel by values of the position that corresponds to the target pixel on the curved surface modeled for each of the three values.
A target pixel selection unit may be further included which selects a plurality of different target pixels while scanning the image. The region setting unit may set, for each of the plurality of target pixels selected by the target pixel selection unit, a region that includes the respective target pixels.
The modeling unit may model the pixel value distribution of the plurality of pixels that form the set region with an Nth order curved surface (where N is an integer greater than or equal to 2).
The region setting unit may increase the number of the plurality of pixels that form the region as a magnitude of noise in the image increases.
A second exemplary aspect of the present invention is an image processing method. The image processing method includes: setting a region including a target pixel selected from among a plurality of pixels that form an image; modeling, with a curved surface, a pixel value distribution of a plurality of pixels that form the region; and replacing a pixel value of the target pixel by a value of a position that corresponds to the target pixel on the modeled curved surface.
A third exemplary aspect of the present invention is a program. The program causes a computer to execute processing including the steps of: setting a region including a target pixel selected from among a plurality of pixels that form an image; modeling, with a curved surface, a pixel value distribution of a plurality of pixels that form the region; and replacing a pixel value of the target pixel by a value of a position that corresponds to the target pixel on the modeled curved surface.
According to the present invention, it is possible to provide a noise suppression technique having a high noise reduction effect with respect to a calculated amount.
An image processing device according to an exemplary embodiment is a device for reducing noise in a processing target image. Here, the processing target image may be a color image having the three color components of R (red), G (green), and B (blue). Alternatively, it may be a gray scale image such as a radiation image. Furthermore, the image may be a still image such as a photograph, or a moving image. When the processing target image is a moving image, the image of each frame that forms the moving image represents a processing target image. Hereunder, the present specification will be described assuming that the processing target image is a color still image.
(1) The image processing device according to the exemplary embodiment acquires a processing target image I.
(2) The image processing device decomposes the processing target image I into three image planes, which correspond to the respective axes of a three-dimensional color space. In
(3) The image processing device sequentially selects a single image plane from among the three image planes.
(4) The image processing device selects target pixels G from the pixels that form the image plane, and sets regions A that include the target pixels G The image processing device selects a plurality of different target pixels G while scanning the image plane, and sets the respective regions A so as to include the target pixels G The set regions A represent unit regions for the image processing device to execute noise reduction processing.
(5) The image processing device models, with a curved surface, the pixel value distribution of the pixels that form a region A. Specifically, the image processing device uses the least-squares method to obtain, as a model of the pixel value distribution, a quadratic surface approximating the pixel value distribution of the plurality of pixels that form the region A.
(6) The image processing device replaces the pixel value Gr of the target pixel G by a values Gm of the position that corresponds to the target pixel G on the modeled curved surface.
The processing target image I handled by the image processing device according to the exemplary embodiment is a digital image. A digital image includes various types of noise, such as block noise due to compression, or noise caused by an imaging sensor, such as a CCD or CMOS image sensor. These types of noise generally take random values. On the other hand, the image processing device according to the exemplary embodiment approximates the pixel value distribution with a smooth curved surface. This makes it possible to restore a smooth pixel distribution from a pixel distribution whose smoothness has been lost due to the superimposition of noise. As a result, the image processing device according to the exemplary embodiment is capable of reducing the noise of the processing target image I.
The storage unit 10 represents a ROM (Read Only Memory) that stores a BIOS (Basic Input Output System) and the like of a computer that realizes the image processing device 1, or a RAM (Random Access Memory) that serves as the work area of the image processing device 1, and a large-capacity storage device such as a hard disk drive (HDD) or a solid state drive (SSD) that stores an OS (Operating System), an application program, and various types of information which are referenced when the application program is executed.
The control unit 11 is a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of the image processing device 1, and executes a program stored in the storage unit 10 to function as an image acquisition unit 110, an image decomposition unit 111, a target pixel selection unit 112, a region setting unit 113, a modeling unit 114, and a pixel value changing unit 115.
The image acquisition unit 110 acquires the processing target image I, which is the processing target for noise reduction. The image decomposition unit 111 decomposes the processing target image I acquired by the image acquisition unit 110 into three image planes, which correspond to the respective axes of a three-dimensional color space.
The target pixel selection unit 112 sequentially selects a plurality of different target pixels G while scanning a single image plane that has been sequentially selected from among the three image planes that form the processing target image I. The region setting unit 113 sets regions A that include the target pixels G selected by the target pixel selection unit 112 from among the pixels that form the image plane. Specifically, the region setting unit 113 selects, for each of the plurality of target pixels selected by the target pixel selection unit 112, a region A that includes the target pixel G.
The modeling unit 114 models, with a curved surface, the pixel value distribution of the plurality of pixels that form a region A. Specifically, the modeling unit 114 uses the least-squares method to model the pixel value distribution of the plurality of pixels that form a region A with an Nth order curved surface (where N is an integer greater than or equal to 2). The details of the pixel value distribution obtained by the modeling unit 114 will be described later.
The pixel value changing unit 115 replaces the pixel value of a target pixel G by a value of the position that corresponds to the target pixel G on the curved surface modeled by the modeling unit 114. A simple arithmetic operation is sufficient for modeling the pixel distribution using the least-squares method. Consequently, the image processing according to the exemplary embodiment is capable of reducing the noise of the processing target image I with a small calculation cost.
If the processing target image I acquired by the image acquisition unit 110 is a color image generated by, for example, a digital still camera, the pixel values of the pixels that form the processing target image I have three values which correspond to the respective axes of a three-dimensional color space. Generally, the pixel values of the pixels that form the processing target image I have pixel values that correspond to each of the colors R, C, and B. However, depending on the image compression method, the pixel values of the pixels that form the processing target image I may have pixel values that correspond to brightness and color difference signal components.
In any case, the modeling unit 114 models, with a curved surface, a value distribution for each of the three values corresponding to the respective axes of the three-dimensional color space for the pixels that form the region A which has been set by the region setting unit 113. The pixel value changing unit 115 replaces each of the three values by values of the position that corresponds to the target pixel G on the curved surface modeled for each of the three values. As a result, the image processing device 1 is capable of achieving noise reduction, even when the processing target image I is a color image.
Next, the pixel distribution modeling processing performed by the image processing device 1 will be described.
In the example shown in
A case will be described below in which a 5×5 pixel region A is used. However, it will be apparent to those skilled in the art upon reading the present specification that a region A having an arbitrary size, such as 3×3 pixels, 7×7 pixels, 9×9 pixels, 11×11 pixels, 3×5 pixels, or 7×3 pixels, may be used. Alternatively, an optimal numerical value (number of pixels) may be selected as the size of the region A to obtain a desired processing result. For example, for a region A in a processing target image I in which the magnitude of the noise is large, the region setting unit 113 may increase the number of pixels that form the region A.
For example, when an image is compressed in units of partial regions of the image having a predetermined size, block noise is generated having about the same size as the size of the partial regions. The region setting unit 113 sets the number of pixels that form the region A so that the size of the region A is larger than the size of the block noise. Consequently, when the modeling unit 114 executes modeling processing of the pixel distribution, it is possible to suppress the influence of noise in the region A from being excessively reflected in the model. As a result, the modeling unit 114 is capable of improving the modeling accuracy of the pixel distribution.
The region setting unit 113 may increase the number of pixels that form the region A as the number of pixels that form the processing target image I increases. Compared with a case where the size of the region A is held constant regardless of the number of pixels that form the processing target image I, the ratio between the number of pixels that form the processing target image I and the number of pixels that form the region A is leveled. As a result, the image processing device 1 is capable of stabilizing the processing result of the noise reduction.
The modeling unit 114 models the pixel values of the pixels that form the region A with a quadratic surface. That is to say, the modeling unit 114 models the pixel values S of the pixels that form the region A using formula (1), which is a function S(x,y) of the XY coordinates of the pixels.
S(x,y)=m1x2+m2x+m3y2+m4y+m5 (1)
Here, mi (i=1, . . . , 5) are modeling parameters.
Formula (2) is obtained when formula (1) is written down with the X coordinate of the pixel j as xj, the Y coordinate as yj, and S(xj,yj) as sj.
Formula (3) is obtained by expressing formula (2) using a matrix.
When formula (3) is written with the left hand side as a vector s, the first term on the right hand side as a matrix X, and the second term on the right hand side as a vector m, formula (3) takes the form of formula (4) below.
s=Xm (4)
Suppose the actual pixel value of a pixel j is represented by dj, and the vector d is a vertical vector having dj as an element. When the pixel value distribution of the region A is modeled with a quadratic surface, the vector d on the right side of formula (3) is expressed in the form shown in formula (5).
d=Xm (5)
In formula (5), the left side is known because it represents the pixel values of the area A. Furthermore, the first term on the right side is known because it is based on the coordinates of each pixel. The second term on the right side is unknown because it is a modeling parameter. Here, an error vector e representing the modeling error is defined by the following formula (6).
e=d−Xm (6)
Formula (5) is an overdetermined problem because the number of data is larger than the number of modeling parameters that are unknown. At this time, a vector mopt that minimizes eTe, which is the two-norm of the error vector e, is known as a least-squares solution, and is represented by the following formula (7).
m
opt=(XTX)−1XTd (7)
Here, T represents the transpose of a matrix, and −1 represents an inverse matrix.
As shown in
The calculation in formula (8) enables the modeling unit 114 to obtain optimal modeling parameters in the context of minimizing the two-norm in formula (6). As shown in
The target pixel G (j=13) represents the origin in the region A. Consequently, the coordinates (x13,y13) are (0,0). Therefore, the value Gm=S(0,0) of the position that corresponds to the target pixel G on the modeled curved surface is given by formula (9) below.
Gm=S(0,0)=m102+m20+m302+m4×0+m5=m5 (9)
As a result, the value Gm of the position that corresponds to the target pixel G on the modeled curved surface is m5, which is one of the modeling parameters.
As shown in
As described above, the pixel value changing unit 115 only needs to be able to acquire the value of m5 among the five modeling parameters. Therefore, the following formula (10) is obtained when formula (8) is expanded to extract m5.
Note that the vector v=(−0.074286, 0.011429, 0.040000, 0.011429, 0.074286, 0.011429, 0.097143, 0.125714, 0.097143, 0.011429, 0.040000, 0.125714, 0.154286, 0.125714, 0.040000, 0.011429, 0.097143, 0.125714, 0.097143, 0.011429, 0.074286, 0.011429, 0.040000, 0.011429, −0.074286)T.
The vector v can be calculated by substituting the coordinates of the pixels shown in
The storage unit 10 may hold the vector v. In this case, when the pixel value distribution of the plurality of pixels that form the region A is modeled using a quadratic surface, the pixel value changing unit 115 acquires the vector by referring to the storage unit 10. As a result, the calculation of an inverse matrix and the like can be omitted. Consequently, the image processing device 1 according to the exemplary embodiment is capable of accelerating the noise reduction processing of the processing target image I.
Generally, when noise is superimposed on image data, there is an increase in the variation of the pixel values. As shown in
The image acquisition unit 110 acquires the processing target image I, which is the target of the noise reduction processing (S2). If the processing target image I is a color image, the image decomposition unit 111 decomposes the processing target image I into an image plane for each color space (S4). The target pixel selection unit 112 sequentially selects the image planes one by one (S6).
The target pixel selection unit 112 selects, while scanning the selected image plane, one pixel among the plurality of pixels that form the image plane as the target pixel G (S8). The region setting unit 113 sets a region A which includes the target pixel G that has been selected by the target pixel selection unit 112 (S10). The modeling unit 114 models the pixel values of each of the pixels that form the region A according to a quadratic surface (S12). The pixel value changing unit 115 replaces the pixel value Gr of the target pixel G by a value Gm of the position that corresponds to the target pixel G on the quadratic curved surface modeled by the modeling unit 114 (S14).
The image processing device 1 repeats the processing from step S8 to step S14 until the target pixel selection unit 112 has finished selecting the target pixels G from the image plane (S16:No). The image processing device 1 returns to step S6 and repeats the processing from step S6 to step S16 when the target pixel selection unit 112 finishes selecting the target pixels G in one image plane (S16:Yes), but has not yet selected all of the image planes (S18:No).
When the target pixel selection unit 112 has selected all of the image planes (S18:Yes), the processing in the flowchart ends.
As described above, according to the image processing device 1 of the exemplary embodiment, it is possible to provide a noise suppression technique having a high noise reduction effect with respect to a calculated amount.
The present invention has been described above using an exemplary embodiment. However, the technical scope of the present invention is not limited to the scope described in the exemplary embodiment above, and various modifications and changes can be made within the scope of the gist thereof. For example, the specific mode in which the device is distributed and integrated is not limited to the above exemplary embodiment. Further, part or all of the device may be functionally or physically configured by distributing and integrating arbitrary units. Moreover, new embodiments that are generated by arbitrary combinations of a plurality of exemplary embodiments are also included in the exemplary embodiment of the present invention. The effects of the new exemplary embodiments generated by such combinations have the effects of the original exemplary embodiment. Such modifications will be described below.
A case where the modeling unit 114 models, with a quadratic curved surface, the pixel value distribution of the plurality of pixels that form the region A has been described above. However, the order of the curved surface modeled by the modeling unit 114 is not limited to the second order, and may be a third order or higher. As the order of the curved surface modeled by the modeling unit 114 increases, a larger change in the pixel values of the pixels that form the processing target image I can be modeled. Therefore, when foresight information is obtained that indicates the image of the processing target image I contains many high-frequency components, the modeling unit 114 may model the pixel value distribution of the plurality of pixels that form the region A with a third or higher order surface.
For example, when the modeling unit 114 models the pixel value distribution of the pixels that form the area A on a quartic surface, the equation corresponding to formula (1) is written as shown in formula (11).
S(x,y)=m1x4+m2x3+m3x2+m4x+m5y4+m6y3+m7y2+m8y+m9 (11)
Furthermore, the formula corresponding to formula (3) is written as shown in formula (12).
When formula (12) is written with the left hand side as a vector s, the first term on the right hand side as a matrix X, and the second term on the right hand side as a vector m, formula (12) takes the same form as formula (4). Therefore, the least-squares error solution mopt in formula (12) takes the same form as formula (7). As a result, formula (7) takes the same form regardless of the order of the curved surface used to model the pixel value distribution of the plurality of pixels that form the region A.
When the modeling unit 114 models the pixel value distribution of the pixels that form the area A on a quartic surface, the equation corresponding to formula (8) takes the form of formula (13).
In formula (13), the vector corresponding to the vector v in formula (9) becomes v=(−0.04, −0.04, 0.16, −0.04, −0.04, −0.04, −0.04, 0.16, −0.04, −0.04, 0.16, 0.16, 0.36, 0.16, 0.16, −0.04, −0.04, 0.16, −0.04, −0.04, −0.04, −0.04, 0.16, −0.04, −0.04)T.
The storage unit 10 may hold a calculation result of the vector v for different orders in advance. In this case, the pixel value changing unit 115 refers to the storage unit 10 to acquire the vector v corresponding to each order. As a result, even when the pixel value distribution of the plurality of pixels that form the region A is modeled by a different order, the calculation of the inverse matrix and the like by the modeling unit 114 can be omitted.
A case where noise is reduced by modeling the processing target image I with a quadratic surface has been described above. In addition, the noise reduction processing may be further duplicated by modeling, with a quadratic surface, the image obtained by performing a single iteration of the noise reduction processing. As a result, noise reduction can be executed more effectively.
According to the present invention, it is possible to provide a noise suppression technique having a high noise reduction effect with respect to a calculated amount.
Number | Date | Country | Kind |
---|---|---|---|
2017-227701 | Nov 2017 | JP | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/JP2018/026383 | 7/12/2018 | WO | 00 |