EFFICIENT HARDWARE GUIDED FILTERING METHOD FOR USE IN MULTI-LABEL PROBLEM

Abstract

The present invention provides an efficient hardware guided filtering method for use in solving a multi-label problem. The method includes the following steps: inputting an input guidance of a multi-label image; defining an efficient hardware guided filtering (HGF) model; calculating a vector by a customized matrix inversion operation; inputting guidance through a mapping program for adding up result of each channel to form a polynomial guidance, and introducing nonlinearity into the linear model; and obtaining a filtering result in an efficient hardware mode by element-wise calculation and box filtering.

Description

FIELD OF THE INVENTION

The present invention relates to the field of computer vision techniques in a multi-channel guiding the filtered image, particularly an efficient hardware guided filtering method for use in a multi-label problem.

BACKGROUND OF THE INVENTION

Since 2010, guided filtering (GF) has been used to many problems in computer vision and graphics such as image redirection, color transfer and video defogging. Among them, a multi-label system may be one of the most suitable applications for GF to fully utilize its efficiency and effects, because the heavy calculations in the multi-label system urgently require a fast filtering tool.

Due to linear complexity and edge preservation abilities, GF is considered to be the best choice among all candidate filters in a multi-label system. However, a shortcoming of GF is that the color image guided filtering algorithm is not efficient. It is observed that the running time increases significantly according to the size of the matrix. Specifically, matrix inversion is a time-consuming operation. Therefore, it is inefficient to apply GF to a multi-label system with multi-channel guidance, especially for a large number of channels.

In order to reduce the execution time, the most direct method of GF is to start a group of threads to invert the matrix at the same time. However, this strategy is not efficient on current hardware. This is because: (1) Both CPU and GPU rely on Single Instruction Multiple Data (SIMD) architecture to improve performance; (2) Branch instructions are inevitable for traditional matrix inversion methods such as LU algorithm; (3) The SIMD architecture cannot run branch instructions at the fastest speed, because these instructions need to decompose each vector into elements and process them sequentially on the architecture. In order to avoid branch instructions, the matrix can be inverted according to an analytical solution of the matrix inversion. GF uses the fastest OpenCV to implement this strategy to invert the 3×3 matrix, and successfully reduces the running time of inverting 106 matrices to less than 100 ms. However, the implementation complexity of the analysis solution increases as the size of the matrix increases. When the size of the matrix becomes larger, this method can no longer be implemented manually.

SUMMARY OF THE INVENTION

An objective of the present invention is to provide an efficient hardware guided filtering method for use in a multi-label problem, where the method includes:

Step 1, inputting the input guidance of a multi-label image;

Step 2, defining an efficient hardware guided filtering (HGF) model;

Step 3, calculating a vector {right arrow over (w_p)} by a customized matrix inversion operation;

Step 4, inputting guidance through a mapping program for adding up result of each channel to form a polynomial guidance, and introducing nonlinearity into the linear model;

Step 5, obtaining a filtering result in an efficient hardware mode by element-wise calculation and box filtering.

The invention adopts element-wise arithmetic calculation and box filtering, having the following advantages: (1) Introducing nonlinearity to GF by inputting synthetic polynomial multi-channel guidance to overcome the shortcomings of GF linear model; (2) Linear models are usually not suitable for the input data due to their simplicity, so they tend to produce excessively smooth results; (3) Through the hardware efficient matrix inversion algorithm, the additional running time accompanying the nonlinear model can be reduced to an acceptable level.

The present invention will be further described below in conjunction with the accompanying drawings of the specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of the present invention.

FIG. 2 is a flowchart of HGF using input filtering from a multi-label system.

DETAILED DESCRIPTION

With reference to FIG. 1, an efficient hardware guided filtering method for use in a multi-label problem includes a total of four processes: (1) Definition of HGF; (2) Calculation of vector {right arrow over (w_p)}; (3) Synthetic polynomial guidance; (4) Efficient hardware implementation.

(1) Defining HGF Includes the Following Steps:

Step 1. multi-point estimation: to calculate the estimation of a group of points in local support. Specifically, HGF estimates the coefficient {right arrow over (w_p)} of the filter model (1) by minimizing linear ridge regression (2), where Y represents the input image.

$\begin{array}{cc}Z\left(q\right)={\sum }_{i=1}^n\overrightarrow{w_p}\left(i\right)G_i\left(q\right)+\overrightarrow{w_p}\left(0\right),\forall q\in {\Omega }_p& \left(1\right)\\ \underset{\overrightarrow{w_p}}{\min }\lambda {\overrightarrow{w_p}}_2^2+{\sum }_{q\in {\Omega }_p}{\left(Y\left(q\right)-{\sum }_{i=1}^n\overrightarrow{w_p}\left(i\right)G_i\left(q\right)-\overrightarrow{w_p}\left(0\right)\right)}^2& \left(2\right)\end{array}$

Step 2. Aggregation: Fusion of each point available for multi-point estimates.

Equation (2) is used to optimize equation (1) to minimize {right arrow over (w_p)} input HGF, getting a set of values Z_p′(q)=Σ_i=1ⁿ {tilde over (w)}_i,p G_i(q)+{right arrow over (w)}_0,p, q∈Ω_p for a given window Ω_p; HGF aggregates these values together, and regards their average value

$\frac{1}{{\Omega }_q}{\sum }_{p\in {\Omega }_q}{Z}_p^{\prime }\left(q\right)$

as the final filtering result Z(q).

$\begin{array}{cc}\begin{array}{c}Z\left(q\right)=\frac{1}{{\Omega }_q}{\sum }_{p\in {\Omega }_q}{\sum }_{i=1}^n\overrightarrow{w_p}\left(i\right)G_i\left(q\right)+\overrightarrow{w_p}\left(0\right)\\ ={\sum }_{i=1}^n{\overrightarrow{w}}_q^a\left(i\right)G_i\left(q\right)+{\overrightarrow{w}}_q^a\left(0\right)\end{array}& \left(3\right)\end{array}$

(2) Calculating the Vector {right arrow over (w_p)} Includes the Following Steps:

Step 3. To calculate the vector {right arrow over (w_p)} according to equation (4) including the matrix inversion operation (λE+X_P^TX_p)⁻¹. E represents the identity matrix, the matrix X_p=[{right arrow over (c)}_0,p, . . . , {right arrow over (c)}_n,p] of input image pixel p. For the i^th vector {right arrow over (c)}_i,p of P, {right arrow over (c)}_i,p=[G_i(q₁), . . . , G_i(q_|Ωp|)]^T (0≤i≤n+1) is added to the record value G_i(q_k) of the k^th output pixel, wherein q_k∈Ω_p, Ω_p represents the neighboring region centered on the pixel P; and |Ω_p| represents the total number of pixels in Ω_p, and the image is G_i.

{right arrow over (w_p)}=(λE+X_P^TX_p)⁻¹X_p^T{right arrow over (c)}_n,p  (4)

Step 4. Replacing λE+X_P^TX_p by λE+Σ_i=0ⁿ {right arrow over (c)}_i,p {right arrow over (c)}_i,p^T, equation (4) is re-expressed as equation (5).

{right arrow over (w_p)}=X_P^T(λE+X_P^TX_p)⁻¹{right arrow over (c)}_n+1,p=[{right arrow over (c)}_0,p^T, . . . ,{right arrow over (c)}_n,p^T]^T(λE+Σ_i=0ⁿ{right arrow over (c)}_i,p{right arrow over (c)}_i,p^T)⁻¹{right arrow over (c)}_n+1,p  (5)

Step 5, if λE+Σ_i=0ⁿ {right arrow over (c)}_i,p {right arrow over (c)}_i,p^T is invertible in step 4, then equation (6) is valid. Where α_ij,p=α_ij,pⁿ, and can be calculated through equation (7) iterative calculation, k is from 1 to n, wherein

$\begin{array}{cc}{\alpha }_{00}^0=-{\left(\lambda +G_{00,p}\right)}^{-1}{G}_{ij,p}={\overrightarrow{c}}_{i,p}^T{\overrightarrow{c}}_{j,p}{F}_{ij,p}^k={\sum }_{m,n=0}^{k-1}{\alpha }_{im,p}^{k-1}{\alpha }_{nj,p}^{k-1}{G}_{mk,p}{G}_{kn,p}{\gamma }_p^k=-{\left(1+{\lambda }^{-1}{G}_{kk,p}+{\sum }_{m,n=0}^{k-1}{\alpha }_{mn,p}^{k-1}{G}_{km,p}{G}_{nk,p}\right)}^{-1}{\left(\lambda E+{\sum }_{i=0}^n{\overrightarrow{c}}_{i,p}{\overrightarrow{c}}_{i,p}^T\right)}^{-1}={\lambda }^{-1}E+{\sum }_{i,j=0}^n{\alpha }_{ij,p}{\overrightarrow{c}}_{i,p}{\overrightarrow{c}}_{j,p}^T& \left(6\right)\\ {\alpha }_{ij,p}^k=\{\begin{array}{cc}{\gamma }_p^k{F}_{ij,p}^k+{\alpha }_{\mathrm{ij},p}^{k-1}& i<k,j<k\\ {\lambda }^{-1}{\gamma }_p^k\left({\sum }_{n=0}^{k-1}{\alpha }_{in,p}^{k-1}{G}_{nj,p}\right)& i<k,j=k\\ {\lambda }^{-1}{\gamma }_p^k\left({\sum }_{m=0}^{k-1}{\alpha }_{mj,p}^{k-1}{G}_{im,p}\right)& i=k,j<k\\ {\lambda }^{-2}{\gamma }_p^k& i<k,j=k\end{array}& \left(7\right)\end{array}$

Step 6, to put the equation (6) into equation (5): the k^th element {right arrow over (w_p)}(k) of {right arrow over (w_p )} converts into a linear combination G_ij,p as in equation (8).

$\begin{array}{cc}\overrightarrow{w_p}\left(k\right)={{\overrightarrow{c}}_{k,p}^T\left(\mathrm{\lambda E}+{\sum }_{i=0}^n{\overrightarrow{c}}_{i,p}{\overrightarrow{c}}_{i,p}^T\right)}^{-1}{\overrightarrow{c}}_{n+1,p}={\overrightarrow{c}}_{k,p}^T\left({\lambda }^{-1}E+{\sum }_{i,j=0}^n{\alpha }_{ij,p}{\overrightarrow{c}}_{i,p}{\overrightarrow{c}}_{j,p}^T\right){\overrightarrow{c}}_{n+1,p}={\lambda }^{-1}{I}_{kn+1,p}+{\sum }_{i,j=0}^n{\alpha }_{ij,p}{I}_{ki,p}{I}_{jn+1,p}& \left(8\right)\end{array}$

Step 7, there is a vector inner product result of point p: G_ij,p={right arrow over (c)}_i,p^T {right arrow over (c)}_j,p=Σ_k=1^|Ωp|G_i(q_k)G_j(q_k)=Σ_q∈Ωp G_i(q)G_j(q), and the box filtering result G_ij(p)=Σ_q∈Ωp G_iG_j(q)=Σ_q∈Ωp G_i(q)G_j(q), so if the neighboring region Ω_p of p is a box window, since there is G_ij,p=G_ij(p), the box filter is applied to the element to generate the image to form G_ij, {right arrow over (w_p)}(k) is calculated according to the linear combination of G_ij to form {right arrow over (w_p)}. The above steps 4), 5), 6), and 7) only consist of arithmetic calculations and box filtering, which completely eliminates matrix inversion operations in the calculation process.

(3) The Synthetic Polynomial Guidance Includes the Following Steps:

With reference to FIG. 2, in step 8, equation (9) shows a polynomial model guidance I with gray input, where d is the degree of the polynomial function. Assuming G_i=Iⁱ, the equivalence between the linear model (1) and the polynomial model (9) can be found, when the input guidance is multi-channel, the mapping program G_(i-1)d+j=I_i^j is applied directly to each channel independently, where I_i represents the i^th channel of the multi-channel guide I, and n is the number of channels. After that, the results of each channel of the input multi-channel guide I are superimposed to form a polynomial guidance. Mathematically, the linear model (1) in this case is equivalent to the nonlinear polynomial model (10). Therefore, the nonlinearity is successfully assigned to the generalized linear model (1) of HGF.

Z(q)=Σ_i=1^d={right arrow over (w_p)}(i)Iⁱ(q)+{right arrow over (w_p)}(0),∀q∈Ω_p  (9)

Z(q)=Σ_i=1ⁿΣ_j=1^d{right arrow over (w_p)}((i−1)d+j)I_i^j(q)+{right arrow over (w_p)}(0)  (10)

(4) Efficient Hardware Implementation Includes the Following Steps:

Step 9, equation (8) reveals the effective hardware method for calculating {right arrow over (w_p)}, guaranteeing that the k^th element {right arrow over (w_p)}(k) of {right arrow over (w_p)} is a linear combination of the box filtering result G_ij(p). Specifically, (X) represents the box filtering result of image X, W_i and α_ij record the values {right arrow over (w_p)}(k) and α_ij,p of any p in the image region (W_i(p)={right arrow over (w_p)}(i), α_ij(p)=α_ij,p).

Step 10, extending equation (8) to the following box filtering result G_ij (11) and element-wise arithmetic calculation (12), G₀ represents all-ones matrix, G_i(1≤i≤n) represents the i^th channel of G guided by the synthetic polynomial n channel, G_n+1 is another representation method of the input image Y.

G _ij=(G_iG_j)  (11)

W _i=λ⁻¹G_kn+1+Σ_i,j=0ⁿα_ijG_kiG_jn+1  (12)

Updating the formula α_ij,p^k can also be modified to the element-wise arithmetic calculation of matrix (13), where α₀₀⁰=−(λ+G_00,p)⁻¹, F^k=Σ_m,n=0^k-1 α_im^k-1 α_nj^k-1 G_mk G_kn, and γ^k=−(1+λ⁻¹G_kk+Σ_m,n=0^k-1 α_mn^k-1 G_km G_nk)⁻¹.

$\begin{array}{cc}{\alpha }_{\mathrm{ij}}^k=\{\begin{array}{cc}{F}_p^k+{\alpha }_{\mathrm{ij}}^{k-1}& i<k,j<k\\ {\lambda }^{-1}{\gamma }^k\left({\sum }_{n=0}^{k-1}{\alpha }_{in}^{k-1}{G}_{\mathrm{nk}}\right)& i<k,j=k\\ {\lambda }^{-1}{\gamma }^k\left({\sum }_{m=0}^{k-1}{\alpha }_{\mathrm{mj}}^{k-1}{G}_{km}\right)& i=k,j<k\\ {\lambda }^{-2}{\gamma }^k& i=j=k\end{array}& \left(13\right)\end{array}$

Step 11, HGF calculates the filtering result Z according to the average value of the coefficient {right arrow over (w_p)}, defining the average operator (X)=(X)/(G₀), expressing the element-wise arithmetic calculation form of equation (3) as equation (14):

Z=Σ _i=1 ⁿ (W_i)G_i+(W₀)  (14)

Step 12, observing the equation (11), (12), (13), (14), all equations involve only two calculation types: one is the element-wise arithmetic calculation of the matrix, and the other is the box filtering of the image, element-wise arithmetic calculation is a typical data parallel task. It applies element functions to the actual set of input data. The arithmetic calculation can be directly assigned to the core of the CPU or the threads of the GPU for parallel calculation. Many software or libraries support element-wise arithmetic calculations, such as Matlab, ViennaCL and Arrayfire.

Step 13, the value of the smoothed image produced by the box filtering is equal to the sum of its neighboring pixels in the input image. There is no need to manually implement box filtering, Intel's two libraries, the NPP of OpenCV and Nvidia are already available.

Claims

1. An efficient hardware guided filtering method for use in a multi-label problem comprising: step 1: inputting the input guidance of a multi-label image;step 2: defining an efficient hardware guided filtering (HGF) model;step 3: calculating a vector {right arrow over (wp)} by means of a customized matrix inversion operation;step 4: inputting guidance through a mapping program for adding up the results of each channel to form a polynomial guidance, and introducing nonlinearity into the linear model; andstep 5: obtaining a filtering result in an efficient hardware mode by means of element-wise calculation and box filtering.

2. The method according to claim 1, wherein the step 2 comprises the following steps: step 201: defining the HGF by: Z(q)=Σi=1n{right arrow over (wp)}(i)Gi(q)+{right arrow over (wp)}(0),∀q∈Ωp  (1)wherein {right arrow over (wp)}(i) is ith coefficient; {right arrow over (wp)}(0) is an initial coefficient; Gi(q) is a recorded value of a pixel q;step 202: minimizing linear ridge regression (2) to estimate equation (1), wherein coefficient {right arrow over (wp)} of HGF is obtained by:

3. The method according to claim 1, wherein the step 3 comprises the following steps: step 301: calculating the vector {right arrow over (wp)} according to the following equation (4) including the customized matrix inversion operation (λE+XPTXp)−1: {right arrow over (wp)}=(λE+XPTXp)−1XpT{right arrow over (c)}n,p  (4)wherein E is an identity matrix; Xp=[{right arrow over (c)}0,p, . . . , {right arrow over (c)}n,p] is a matrix of input image pixel p, and {right arrow over (c)}n,p is the nth vector of pixel p, andwherein with respect to ith vector {right arrow over (c)}i,p of {right arrow over (c)}i,p=[Gi(q1), . . . , Gi(q|Ωp|)]T (0≤i≤n+1) is added to a recorded value Gi(qk) of the kth output pixel q, wherein qk∈Ωp, Ωp represents a neighboring region centered on the pixel P; and |Ωp| represents the total number of pixels in Ωp;step 302: replacing λE+XPTXp by λE+Σi=0n {right arrow over (c)}i,p {right arrow over (c)}i,pT to obtain equation (5), wherein if the equation is invertible then the equation (6) is valid:

4. The method according to claim 1, wherein the step 4 comprises: when the input guidance is multi-channel, the mapping program G(i-1)d+j=Iij is applied directly to each channel independently, and results of each channel of the input multi-channel guide I are superimposed to form a polynomial guidance, nonlinearity is assigned to a generalized linear model (1) of the HGF to obtain a nonlinear polynomial model (10): Z(q)=Σi=1nΣj=1d{right arrow over (wp)}((i−1)d+j)Iij(q)+{right arrow over (wp)}(0)  (10)wherein d is a degree of the polynomial function; Ii represents the ith channel of the multi-channel guide I, and n is the number of channels

5. The method according to claim 1, wherein the step 5 comprises the following steps: step 501: representing a box filtering result of image X as (X), Wi and αij recording values of {right arrow over (wp)}(k) and αij,p at any p point in the image region;step 502: extending equation for calculating {right arrow over (wp)}(k) to those which only use the box filtering result Gij (11) and element-wise arithmetic calculation (12)