The present invention relates to an image coding apparatus, a system comprising an image encoding apparatus and an image decoding apparatus, and a method for still image coding. The present invention also relates to a computer-readable storage medium storing program code, the program code comprising instructions for carrying out such a method.
Still image compression technologies can be used e.g. in still image codecs such as H.264/AVC, H.265/HEVC and JPEG. Improvement of the decoded image can be achieved by post filtering. There are two base classes of post filtering: encoder side post filtering and decoder side post filtering.
In case of encoder side post filtering, filter parameters are estimated on the encoder side and are then transferred to the decoder side. The decoder recovers post filter parameters after transmission and applies filter parameters for decoded image filtering before delivery to the end user. In this case filter design and filter parameter estimation depend on the still image encoder architecture.
In case of decoder side post filtering post filter parameters are estimated from the decoded image and so there is no still image encoder modification.
The most familiar encoder side post filter is the Adaptive Loop Filter. This filter was proposed on MPEG standardization meeting for use in the Next Generation Video codec.
A still image decoder recovers post filter parameters from a transmitted bitstream and uses these parameters for decoded image filtering before end user delivery.
According to an Adaptive Loop Filter (ALF) method, each reconstructed (decoded) image is divided into a set of small macroblocks (super-pixels) and then each macroblock is filtered by an adaptive post filter.
An ALF can comprise three blocks: A parameter estimation block, a pixel domain filtering block and an application map block. In the encoder, the parameter estimation block estimates a 2D impulse response for a Wiener filter in pixel domain for each macroblock based on matching between original and reconstructed images. Then these impulse responses are quantized and transferred to an entropy encoder. The entropy encoder inserts encoded filter parameters into an output bitstream transferred from encoder to decoder side. In the decoder, the entropy decoder recovers the Wiener filter impulse response for each macroblock and transfers filter parameters to the pixel domain filtering block for use in the filtering process. In the decoder and the encoder the pixel domain filtering block performs reconstructed image filtering in pixel domain according to filter parameters from the parameter estimation block. In the decoder the filtered image is used for decoded image improvement before end user delivery. Filtering can be applied not for full reconstructed frame but only for part of image. In the encoder the application map block chooses which super-pixels should be filtered based on a rate-distortion optimization (RDO) process. The application map is encoded by the entropy encoder and written into the output bitstream on the encoder side. In the decoder the entropy decoder recovers the filter application map. Then this map is used to turn on/off filtering.
The ALF can have the following base features for still image encoding:
Despite advantages of the ALF structure, there is still a need for improvements in image coding.
The objective of the present invention is to provide an image coding apparatus, a system comprising an image encoding apparatus and an image decoding apparatus, and a method for still image coding, wherein the image coding apparatus, the system and the method overcome one or more of the problems of the prior art.
A first aspect of the invention provides an image coding apparatus, comprising:
The image coding apparatus can be an encoder or a decoder for still images. It is understood that filtering the reconstructed image does not necessarily refer to filtering the entire image. For example, in some embodiments only some blocks of the reconstructed image may be filtered.
The image coding apparatus of the first aspect has the advantage that the filter parameters can be derived on the decoder-side and do not need to be encoded into the bitstream, thus an improved coding efficiency can be achieved.
Preferably, the parameter determination unit is configured to determine the one or more image filter parameters based only on the reconstructed image and not on the original image on the encoder side. Since the parameter determination unit thus does not require the original image, the parameters can be determined in the same way based on the reconstructed image, both at the encoder side and at the decoder side (where the original image is not available).
In a first implementation of the image coding apparatus according to the first aspect, the parameter determination unit is configured to estimate the original image from the reconstructed image and determine the first parameters based on the estimated original image.
Original signal estimating allows roughly removing quantization noise and better estimating original signal parameters (for example original signal spectrum). Thus, an improved determination of the first parameters can be performed based on the estimated original image compared to determining the first parameters based on the reconstructed image.
In a second implementation of the image coding apparatus according to the first aspect as such or according to the first implementation of the first aspect, the parameter determination unit is configured to determine a quantization noise value from the codec signaling information and determine the second parameters based on the derived quantization noise value.
This has the advantage that the filter parameters can be determined with higher accuracy
Preferably, the quantization noise value can be derived analytically from the codec signaling information.
In a third implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the parameter determination unit is configured to analytically derive the quantization noise value from a quantization parameter QP of the codec signaling information.
This has the advantage that the quantization noise value can be determined efficiently and with high accuracy.
In a fourth implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the image coding apparatus further comprises an application map unit configured to determine where filtering should be implemented.
Preferably, the application map unit is implemented at the encoder side, and corresponding signaling information is transferred to the decoder side.
In a fifth implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the parameter determination unit is configured to determine the filter parameters by:
In a sixth implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the parameter determination unit is configured to determine, for each of the blocks, the filter parameters based on the transformed patches by:
wherein Ω is a column number in matrix spatial frequencies v, w correspond to a j-th row of matrix tfvwi and N is a quantization noise value derived from the codec signaling information.
In a seventh implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the mixed-domain filtering unit is configured to, for each of a set of blocks of the reconstructed image:
Preferably, the set of blocks of the reconstructed image covers the entire reconstructed image. The blocks can represent a segmentation of the reconstructed image (i.e., they are non-overlapping) or they can cover the reconstructed image in an overlapping manner.
In an eighth implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the 2D transformation is a Haar wavelet transform.
In a ninth implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the mixed-domain filtering unit is configured to perform, for each of the blocks, the collaborative filtering based on the transformed patches by:
In a tenth implementation of the image coding apparatus according to the first aspect as such or according to any of the preceding implementations of the first aspect, the 1D transformation is a Hadamard transform.
In an eleventh implementation of the image coding apparatus of the second aspect,
The above-mentioned flags can be e.g. encoded by the encoder into the bitstream.
A second aspect of the invention refers to a system comprising an image encoding apparatus according to the first aspect or one of its implementations and an image decoding apparatus according to the first aspect or one of its implementations, wherein the parameter determination unit of the image encoding apparatus is configured to determine the filter parameters in the same way as the parameter determination unit of the image decoding apparatus.
This system has the advantage that no transmission of filter parameters from the encoder side to the decoder side is required. Thus, a coding efficiency can be improved.
A third aspect of the invention refers to a method for still image coding, the method comprising:
The methods according to the third aspect of the invention can be performed by the image coding apparatus according to the first aspect of the invention. Further features or implementations of the method according to the third aspect of the invention can perform the functionality of the image coding apparatus according to the first aspect of the invention and its different implementation forms.
A fourth aspect of the invention refers to a computer-readable storage medium storing program code, the program code comprising instructions for carrying out the method of the third aspect.
To illustrate the technical features of embodiments of the present invention more clearly, the accompanying drawings provided for describing the embodiments are introduced briefly in the following. The accompanying drawings in the following description are merely some embodiments of the present invention, modifications on these embodiments are possible without departing from the scope of the present invention as defined in the claims.
The image coding apparatus 100 comprises an image reconstruction unit 110, a parameter determination unit 120 and a mixed-domain filtering unit 130.
The image reconstruction unit 110 is configured to reconstruct an image.
The parameter determination unit 120 is configured to determine one or more filter parameters, based on one or more first parameters which are based on the reconstructed image and one or more second parameters which are based on codec signaling information.
The mixed-domain filtering unit 130 is configured to filter in a frequency domain and a pixel domain the reconstructed image based on the determined filter parameters to obtain a filtered image.
The image coding apparatus 100 can be an encoder and/or a decoder.
The method comprises a first step 310 of reconstructing an image.
The method comprises a second step 320 of determining one or more filter parameters based on one or more first parameters which are based on the reconstructed image and one or more second parameters which are based on codec signaling information.
The method comprises a third step 330 of filtering in a frequency domain and in a pixel domain the reconstructed image based on the determined filter parameters to obtain a filtered image.
The reconstructed (decoded) image can be divided into a set of small macroblocks and then each macroblock can be filtered by a filter as described below.
The filter 400 comprises three blocks: a parameter estimation block 410, a mixed domain collaborative filtering block 420 and an application map block 430.
Similar to ALF, the parameter estimation block 410 calculates filter parameters. But in contrast to ALF, the filter parameters are calculated without knowledge of the source image. The filter parameter estimation is based on two groups of input parameters. The first group of parameters is estimated based on the reconstructed image and the second group of parameters is derived from codec signaling information which are already transferred from the encoder to the decoder. According to the above described procedure the filter parameters can be estimated on the decoder side and so filter parameters should not be transferred from the encode to decoder side. Similar to ALF, a parameter estimation block calculates the filter impulse response, but the parameter estimation block 410 of filter 400 estimates a frequency impulse response, because base filtering is performed in frequency domain. The frequency domain implementation allows building a more efficient non-linear filter. The frequency impulse response is an example of filter parameters.
In contrast to ALF, which performs local filtering in pixel domain, non-local collaborative filtering is performed in a mixed domain (spatial frequency and pixel domain). Such approach allows more efficient usage of spatial redundancy. Initial filtering is performed in frequency domain and the final averaging is performed in pixel domain.
Similar to ALF, during the RDO process the application map block determines areas where filtering should be applied. If the coding gain from removing quantization noise from the decoded image is significantly more than degradation of the filtered decoded image, then filtering is applied. Otherwise, the unfiltered reconstructed image is used as output for the end user.
The filter 400 can be used as decoder side post filter, too. In this case the application map block 430 can be excluded from the post filter design and there is no additional bit budget for application map transmission. Of course, in this case coding gain will be decreased, but it will be significant also.
The application map block 430 generates application map data that is provided to an entropy coder 440.
For the particular case of Wiener collaborative filter the frequency impulse response (filter parameter in frequency domain) can be determined using the following procedure. At the first step a StackTransform( )procedure is performed for each frequency domain cluster Fi.
The following scanning rule is used: each row of matrix Ti consists of frequency component from different patch of the same frequency domain cluster Fi with the same spatial frequencies [v, w ]:
On the last step 920 of the StackTransform( )procedure the output matrix TFi is created. Each row of this matrix is a 1D transform 920 of a corresponding row of Ti matrix.
A frequency impulse response for a Wiener collaborative filter is determined by the following equation:
wherein Ω is a column number in matrix tfvwi, spatial frequencies v, w correspond to a j-th row of matrix tfvwi and N is a quantization noise value derived from the codec signaling information.
As in the filter parameter estimator the image partitioning block 1010 creates a set of macroblocks which cover the reconstructed image. Then, for each reference macroblock from this set k closest blocks are found using a MSE metric by the block matching block 1020. In a next step, found spatial patches are combined in a pixel domain cluster corresponding to the reference macroblock. The 2D transform block 1030 performs a 2D transform over each patch in a chosen pixel domain cluster and produces a frequency domain cluster which comprises 2D spectra of corresponding pixel domain patches. The collaborative frequency domain filter 1040 performs collaborative filtering of 2D spectra of pixel domain patches using filter parameters calculated in the previous step. The inverse 2D transform block 1050 returns filtered frequency domain patches to the pixel domain. Then, pixel based collaborative filtering 1060 performs final averaging of pixel domain patches corresponding to the reference macroblock.
P
i=BlockMatch(S, bi)={bi, Pi0, pi2, . . . , pik-1},
where S is the reconstructed image and pij is a j-th patch corresponding to the bi reference macroblock. In the next stage for each patch from pixel domain cluster Pi a 2D transform is performed. A frequency domain cluster Fi includes 2D spectra of pixel domain patches from pixel domain cluster Pi is used for collaborative filtering. In a general case the collaborative filter in frequency domain performs frequency domain patch averaging and produces filtered frequency domain patches
R
i=FreqCollaborativeFiltering (Fi, Gi)
corresponding to pixel domain patches. An inverse 2D transform returns filtered in frequency domain patches Ri to pixel domain and produces filtered pixel domain patches {tilde over (P)}i. On the last processing stage frequency domain filtered patches {tilde over (P)}0, {tilde over (P)}1,. . . , {tilde over (P)}m are averaged in pixel domain based on a procedure SameBlockAvg(), which will be described below.
Collaborative filter in pixel domain averages the same patch (patch with fixed spatial coordinates) along all clusters which include this patch. This allows decreasing noise and introducing low edge distortion.
If a coding gain from removing quantization noise from the decoded image is significantly more than degradation of the filtered decoded image, then filtering is applied. Otherwise, a reconstructed image is used as output for an end user. The application map block decisions are encoded with an entropy encoder 1430 and transferred from encoder to decoder side.
As outlined above, embodiments of the invention overcome one or more of the following disadvantages of the Adaptive Loop Filter:
Further embodiments of the present invention may include:
The foregoing descriptions are only implementation manners of the present invention, the scope of the present invention is not limited to this. Any variations or replacements can be easily made through person skilled in the art. Therefore, the protection scope of the present invention should be subject to the protection scope of the attached claims.
This application is a continuation of International Application No. PCT/RU2016/000917, filed on Dec. 23, 2016, the disclosure of which is hereby incorporated by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/RU2016/000917 | Dec 2016 | US |
Child | 16448956 | US |