Method for image visual effect improvement of video encoding and decoding

Abstract

The present invention relates to method for image visual effect improvement of video encoding and decoding. The method at an encoder comprises enhancement process of image boundary information, adaptive brightness transformation process, and adaptive enhancement process of chrominance information. The method at a decoder comprises adaptive enhancement of boundary information based on the block statistical characteristic, adaptive brightness transformation process, and adaptive enhancement process of the chrominance information. Based on characteristics of the image source at the encoder and the decoder, the present invention uses brightness space processing technology to eliminate blocking artifacts, enhance high frequency information and improve brightness information distribution, so as to improve brightness information of the image in one aspect, and uses the chroma space processing technology to enhance color effect of the image in the other aspect, and finally, the objective of improving visual effect of the image is achieved.

Description

FIELD OF THE INVENTION

The present invention relates to video encoding and decoding methods, more particularly, to a method for image visual effect improvement of video encoding and decoding.

BACKGROUND OF THE INVENTION

Based on human vision system, color can be described by brightness, hue and saturation. Usually, hue and saturation are generally referred to as chroma, which is used to represent the category and depth of color. In the video encoding process, for different frames, regions people cared about are dynamically changed, which requires that the algorithm is able to adjust transformation function according to the change of the video sequences, so that brightness distribution of the image can be improved according to demand in various scenes. The visual quality of the image can be improved by a constant transformation function of brightness with the parameters obtained by considerable statistical experiments. However, if the same approach used in the ordinary scenes is carried out in some specific scenes (such as a wholly dark scene), visual quality of the image will be decreased.

For color information of an object, people always hope that, the more colorful the better. Considering the requirement of visual comfort, the bigger the transform intensity is, the more color of the image with insufficient chroma information is improved. Skin color of human beings is between yellow and red. If the same model is used for the whole region, taking relatively large adjusting values, uncomfortable feeling to skin color will be generated, and taking relatively small adjusting values, the requirement of enhancing color information of objects in other color gamut will be restricted. If the algorithm is dependent on the detection of skin color regions, firstly, computational complexity is increased, and secondly there isn't a detection algorithm for skin color regions with 100% accuracy, thirdly many problems such as balance transition brought in by incorrect judgment of discrete point field will occur. Although people are more sensitive to luminance than to chrominance, preprocessing should be employed to enhance the color of the image, since chroma information carried by the image sequence (such as image captured by a camera) processed by the video encoder is insufficient at some time. Most conventional color processing methods are based on RGB or HSV color model, while a separate representation mode of luminance and chrominance, i.e., YUV, is used in video encoding. Although transformation between different models can be realized through color space transformation technology, computational complexity bought in by transformation and invert transformation is also considerable.

Image quality will be decreased in varying degrees after encoding. Problems, such as blocking artifacts brought in by block-based encoding and decoding strategy, attenuation and losing of high frequency information and so on, are present in the image sequence after decoding. In order to eliminate blocking artifacts without losing of boundary high frequency information, and take characteristics of block-based encoding and decoding strategy into account that the blocking artifacts always present at the boundary between blocks, a method for block-based boundary adaptive enhancement is employed.

SUMMARY OF THE INVENTION

In order to improve visual effect of video sequences at an encoder, the present invention provides a method for image visual effect improvement of video encoding and decoding, wherein a boundary information enhancement technology is used to increase the amount of high frequency information contained in the image, and adaptive enhancement technologies for luminance and chrominance respectively are provided for improving the brightness information distribution of the image and enhancing chroma information of the image.

The method according to present invention comprises the following steps at the encoder:

S11: extracting image boundary information and enhancing a boundary information operation, the step further comprising:

• • S111: Extracting boundary information h(x,y) of an image f(x,y),h(x,y)=γ(f(x,y))
  • S112: Obtaining a boundary enhanced image g(x,y) by transforming the extracted boundary information h(x,y) and the image f(x,y),g(x,y)=φ(f(x,y),h(x,y))

wherein f (x,y) is a brightness value of the original image at the encoder, γ(x) is a boundary information extracting function, φ(f(x,y),h(x,y)) is a transformation function selected according to characteristics of the original image and the boundary information;

S12: Adaptive luminance transforming to improve luminance distribution:g′(x,y)=ψ(f(x,y),α(k)|k=1,2, . . . ,K),

wherein g′(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is a transformation function, wherein α(k) is a set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters;

S13: Adaptively enhancing the chrominance information, which is performed in the UV color space,(u′(x,y),v′(x,y))=w*φ(u(x,y),v(x,y),α_u,α_v,β_u,β_v)

wherein φ(u(x,y),v (x,y),α_u,α_v,β_u,β_v) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α_u and α_v, a chroma adjusting step is determined by β_u and β_v.

Image quality will be decreased in varying degrees after encoding. Problems, such as blocking artifacts brought in by block-based encoding and decoding strategy, attenuation and losing of high frequency information and so on, are present in the image sequence after decoding. In consideration of a need for improving visual effect of the image at the decoder, a method for image visual effect improvement of video encoding and decoding is provided in the present invention.

The method according to present invention comprises the following steps at the decoder:

S21: selecting a processing mode

$\mathrm{t\_}0=\sum _{x,y\in 0_{}}{\hslash }_0\left(f\left(x,y\right)\right)$ according to block statistical characteristic,

• • if (t_—0>Thres_—1) mod e=Flat region processing method
  • else mod e=Complex region processing methodthen implementing operations for eliminating blocking artifacts and enhancing boundary information based on the determined processing mode;

wherein f(x,y) is a original image value at the decoder, t_—0 is a statistical variable name of a statistical region , (j=0, 1, 2) is a statistical characteristic function, _j is a statistical region corresponding to , and Thres_—1 is a threshold for determining whether the current processing region is a flat region or a complex region;

S22: Adaptively transforming the brightness and improving brightness distribution of the image:g(x,y)=ψ(f(x,y),α(k)|k=1,2, . . . ,K),

wherein f(x,y) is a brightness value of the original image at the decoder, g(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K), is a transformation function, wherein α(k) is a set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters;

S23: Adaptively enhancing the chroma information, wherein the chroma information adaptive enhancement is performed in a UV chroma space,(u′(x,y),v′(x,y))=w*φ(u(x,y),v(x,y),α_u,α_v,β_u,β_v)

wherein φ(u(x,y),v (x,y),α_u,α_v,β_u,β_v) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α_u and α_v, a chroma adjusting step is determined by β_u and β_v.

Through the above mentioned method, adaptive adjustment can be applied to eliminate the blocking artifacts, and enhance the image luminance and chrominance information, in such a way the object of improving the objective effect and subjective effect of the coded and decoded images can be achieved. When the adaptive boundary information enhancement technology according to present invention is employed at the decoder, the effect of separation method in enhancing boundary information and eliminating blocking artifacts can be maintained while the processing speed is improved, and the objective effect and subjective effect of the image also can be improved remarkably.

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings illustrate preferred, but not exclusive embodiments of the inventions:

FIG. 1 is a flow diagram of a method at the encoder according to an embodiment of the present invention;

FIG. 2 is a flow diagram of boundary information enhancement shown in FIG. 1;

FIG. 3 is a flow diagram of adaptive brightness transformation shown in FIG. 1;

FIG. 4 is a flow diagram of adaptive enhancement of chroma information shown in FIG. 1;

FIG. 5 is a flow diagram of a method at the decoder according to an embodiment of the present invention;

FIG. 6 is a contrast diagram showing an original decoded image at the decoder and two images processed by the conventional separation method and the adaptive boundary enhancement respectively using test source in WMV format;

FIG. 7 is a contrast diagram showing effects of an original decoded image and a processed image by the present invention using test source in WMV format.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring FIG. 1, FIG. 1 is a flow diagram of the method for image visual effect improvement of video encoding and decoding at the encoder according to an embodiment of the present invention. The method comprises the following steps at the encoder:

1. The present step implements boundary information enhancement process, and further comprises the following steps:

• • a1: Extracting boundary information h(x,y) of an image f(x,y),h(x,y)=γ(f(x,y))
  • b1: Obtaining a boundary enhanced image g(x,y) by transforming the extracted boundary information h(x,y) and the image f(x,y),g(x,y)=φ(f(x,y),h(x,y))

wherein: f(x,y) is a brightness value of the original image at the encoder, φ(f(x,y),h(x,y)) is a transformation function selected according to characteristics of the original image and its boundary information, γ(x) is a boundary information extracting function, wherein different methods for extraction can be employed according to different applications requirements. With respect to derivative method, for example, a first order derivative, a second order derivative and so on can be employed, such as gradient module extracting method:

$\mathrm{grad}\left(f\left(x,y\right)\right)=\underset{l}{\max }\left(\frac{\partial f\left(x,y\right)}{\partial l}\right)$

2. In video encoding process, for different frames, regions people cared about are dynamically changed, which requires that the algorithm is able to adjust transformation function according to the change of the video sequences, so that brightness distribution of the image can be improved according to demand in various scenes.

The visual quality of the image can be improved by a constant transformation function of brightness with the parameters obtained by considerable statistical experiments. However, if the same approach used in the ordinary scenes is carried out in some specific scenes (such as a wholly dark scene), visual quality of the image will be decreased.

The present step implements adaptive brightness transformation and improvement of image brightness distribution. The principle of adaptive brightness transformation is that, the set of parameters of the transformation function is adaptively updated according to a statistical characteristic of brightness value of the image before being transformed, so that the transformation function is adjusted dynamically along with different image characteristics, and thus the processing method is optimized:g(x,y)=ψ(f(x,y),α(k)|k=1, 2, . . . , K),

wherein f(x,y) is a brightness value of the original image at the decoder, g(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is the transformation function, wherein α(k) is the set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters.

As shown in FIG. 3, step 2 further comprises the following steps:

a2: Given a characteristic space of the current frame image is

$\xi =\bigcup _{k=1}^M{\xi }_k,$ ξ_i∩ξ_j=φ,i≠j, and a whole statistical characteristic of the image is obtained by statistic of the brightness information,

for (k=0; k<M; k++) if (f(x,y)εξ_k) Calculating a statistical characteristic φ_k(f(x,y)) of ξ_k.

Finally, the statistical characteristic of the current frame image is obtained:{φ_k(f(x,y))|k=1, 2, . . . , M};

wherein ξ_k and φ_k(f(x,y)) are image characteristic subspace, the statistical characteristic of ξ_k respectively;

b2: The threshold is adjusted according to visual characteristic together with the regional statistical characteristic, and the image is divided into different regions

$\Omega =\bigcup _{k=1}^N{\Omega }_k.$ The statistical characteristic threshold P_H is adjusted to be P_H′ according to a statistical relationship between the global area and the regions,P′_H=ratio*η(P_H, Φ₁, Φ₂, . . . , Φ_N)

wherein Φ_k is a statistical characteristic of Ω_k,Φ_k={φ₁(Ω_k), φ₂(Ω_k), . . . , φ_M(Ω_k)}

P_H′, P′_H are the threshold obtained through the whole statistical information and the adjusted threshold respectively;

c2: Parameter values of the transformation function is obtained based on the statistical characteristic;α(k)=(P′_H)k=1, 2, . . . K

(x) is a adjusting function of the parameter α(k) of the transformation function ψ(x,α(k)|k=1, 2, . . . , K);

d2: By using brightness transformation function ψ(f(x,y),α(k)|k=1, 2, . . . , K), brightness transformation is implemented and distribution of the image brightness information is improved;

wherein f(x,y) is the brightness value of the original image at the decoder, g(x,y) is the adjusted brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is the transformation function, wherein α(k) is the set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), K is the number of the parameters.

3. Chroma information is adaptively enhanced, wherein the chroma information adaptive enhancement is performed in a UV chroma space,(u′(x,y),v′(x,y))=w*φ(u(x,y),v(x,y),α_u,α_v,β_u,β_v)

wherein φ(u(x,y),v (x,y),α_u,α_v,β_u,β_v) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α_u and α_v, a chroma adjusting step is determined by β_u and β_v.

As shown in FIG. 4, following steps are further comprised:

a3: Saturation information of the UV space κ is obtained through a statistic of the UV characteristic of the current image frame;

b3: Adjusting parameters are calculated with the color saturation information;α_u=γ_u(κ)β_u=γ_u(κ)α_v=γ_v(κ)β_v=γ_v(κ)

c3: By statistical experiments in the UV space model, empirical value range of skin color distribution is obtained, and the weight function w=η(θ) is determined, wherein θ is the empirical value range of skin color, θε[θ₁,θ₂].

w=η(θ), η(θ) is a continuous function having only one minimum value, and w_min=η((θ₁+θ₂)/2).

d3: Chroma transformation is implemented using the chroma transformation function (u′(x,y),v′(x,y))=w*φ(u(x,y),v(x,y),α_u,α_v,β_u,β_v), and the chroma information of the image is enhanced;

wherein φ(u(x,y),v(x,y),α_u,α_v,β_u,β_v) is the transformation function, w is the weight function, the UV chroma deviation position is determined by α_u and α_v, and the chroma adjusting step is determined by β_u and β_v.

Although transformation between different models can be realized through color space transformation technologies, computational complexity bought in by the transformation and invert transformation is also considerable. Considering the document format processed by the encoder, format conversation time should be reduced. The present invention implements a color information process directly in the UV chroma space.

Referring FIG. 5, FIG. 5 is a flow diagram of the method for image visual effect improvement of video encoding and decoding at the decoder according to an embodiment of the present invention. The method comprises the following steps at the decoder:

10. Selecting a processing mode

$\mathrm{t\_}0=\sum _{x,y\in 0_{}}{\hslash }_0\left(f\left(x,y\right)\right)$ according to block statistical characteristic

• • if (t_—0>Thres_—1) mod e=Flat region processing method
  • else mod e=Complex region processing method

Then, operations for eliminating blocking artifacts and enhancing boundary information are implemented based on the determined processing mode.

For processing of flat regions:t_—1_j=(f(x,y)|f(x,y)ε)

• • if (t_—1_j>Thres_—2)weighted lowpass filtering
  • else protecting the boundary information

wherein t_—1j is a statistical characteristic variable name of the j-th flat region , Thres_—2 is a threshold for the currently processed flat region processed by the selected different processing methods.

For processing of complex regions:t_—2_j=(f(x,y)|f(x,y)ε)

• • if (t_—2_j>Thres_—3)f(x_M,y_N|(x_m,y_n)εblock_j, m=1, 2, . . . , M, n=1, 2, . . . , N)−=κ;f(x₁,y₁|(x_m,y_n)εblock_j+1, m=1, 2, . . . , M, n=1, 2, . . . , N)+=κ;
  • else enhancing the boundary information.

wherein f(x,y) is a original image value at the decoder, t_—0 is a statistical variable name of a statistical region , (j=0, 1, 2) is a statistical characteristic function, is a statistical region corresponding to , and Thres_—1 is a threshold for determining whether the current processing region is a flat region or a complex region; t_—2j is a statistical characteristic variable name of the j-th complex region , Thres_—3 is a threshold for the currently processed complex region processed by the selected different processing methods.

The following table shows comparative experiment between the adaptive boundary information enhancement process of the present step and the separation method process in the prior art. The experiment uses images with a source size of 320×240, and same decoders are used. Comparison of objective effects is as follows:

	processing method
		Adaptive	Increase of
	Separation	boundary information	processing
Test	method	enhancement method	speed
sequence	Speed (fps)
Test1_WMV	187	280	49.7%
Test3_FLV	181	253	39.8%
Test4_FLV	182	249	36.8%
Test2_FLV	188	252	34.0%

It can be seen from the above table that, processing speed can be significantly improved by the adaptive boundary information enhancement method in accordance with the present invention.

FIG. 6 is a contrast diagram showing an original decoded image at the decoder and two images processed by the conventional separation method and the adaptive boundary enhancement respectively using test source in WMV format. Through comparison of the objective effect, the objective quality of the image is significantly improved by the image processing method of adaptive boundary enhancement according to the present invention.

20. Brightness of the image is adaptively transformed, and the brightness distribution of the image is improved. The principle of adaptive brightness transformation is that, the set of parameters of the transformation function is adaptively updated according to a statistical characteristic of brightness value of the image before being transformed, so that the transformation function is adjusted dynamically along with different image characteristics, and thus the processing method is optimized:g(x,y)=ψ(f(x,y),α(k)|k=1, 2, . . . , K),

wherein f(x,y) is a brightness value of the original image at the decoder, g(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is the transformation function, wherein α(k) is the set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters.

30. Chroma information is adaptively enhanced, wherein the chroma information adaptive enhancement is performed in a UV chroma space,(u′(x,y),v′(x,y))=w*φ(u(x,y),v(x,y),α_u,α_v,β_u,β_v)

wherein φ(u(x,y),v (x,y),α_u,α_v,β_u,β_v) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α_u and α_v, a chroma adjusting step is determined by β_u and β_v.

FIG. 7 is a contrast diagram showing effects of an original decoded image and a processed image by the present invention using test source in WMV format. After implementing the step 10, step 20 and step 30 of the above-mentioned method, blocking artifacts are eliminated, luminance and chrominance information of the image are enhanced by adaptive adjustment, so that the object of improving objective effect and subjective effect of the decoded image is achieved.

The foregoing description of the exemplary embodiments of the invention has been presented only for the purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching without departing from the protection scope of the present invention.

Claims

1. A method for image visual effect improvement of video encoding and decoding, comprising the following steps at an encoder: extracting image boundary information and enhancing a boundary information, the step further comprising: extracting boundary information h(x,y) of an image f(x,y), h(x,y)=γ(f(x,y));andobtaining a boundary enhanced image g(x,y) by transforming the extracted boundary information h(x,y) and the image f(x,y), g(x,y)=φ(f(x,y), h(x,y)),wherein f(x,y) is a brightness value of the original image at the encoder, γ(f(x,y) is a boundary information extracting function, φ(f(x,y),h(x,y)) is a transformation function selected according to characteristics of the original image and the boundary information;adaptive luminance transforming to improve luminance distribution: g′(x,y)=ψ(f(x,y),α(k)|k=1, 2, . . . , K),wherein g′(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is a transformation function, wherein α(k) is a set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters;adaptively enhancing the chrominance information, which is performed in the UV color space, (u′(x,y),v′(x,y))=w*φ(u(x,y),v(x,y),αu,αv ,βu,βv )wherein φ′(u(x,y),v (x,y),αu,αv ,βu,βv ) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by αu and αv ,a chroma adjusting step is determined by βu and βv , u(x,y),v (x,y) are UV coordinate value before adaptively transformation, and (u ′(x,y),v ′(x,y) are UV coordinate value after adaptively transformation;wherein the step of adaptive luminance transforming to improve luminance distribution further comprising following steps: given a characteristic space of the current frame image being

2. The method according to claim 1, wherein the step of adaptively enhancing the chromiance information further comprises. obtaining saturation information of the UV space κ through a statistic of the UV characteristic of the current image frame by constructing a UV chroma space model mod(u(x,y),v (x,y))=√{square root over (u(x,y)2+v (x,y)2)}{square root over (u(x,y)2+v (x,y)2)},

3. The method according to claim 1, wherein γ(f(x,y)) is a boundary information extracting function, and a gradient module extracting method is employed:

4. A method-for image visual effect improvement of video encoding and decoding, comprising the following steps at a decoder: selecting a processing mode