Patent Application
20180035113
The present disclosure relates to a method of reconstructing signal amplitudes for video coding and compression. More specifically, it relates to methods for signaling whether a Sample Adaptive Offset (SAO) process is used in video coding and processing systems such as within the High Efficiency Video Coding (HEVC) standard.
The HEVC standard, currently published as ISO/IEC 23008-2 MPEG-H Part 2 and ITU-T H.265, introduced several new video coding tools designed to improve video coding efficiency over previous video coding standards and technologies such as MPEG-2, MPEG-4 Part 2, MPEG-4 AVC/H.264, VC1, and VP8. One of these tools is the SAO, which is a filtering mechanism that may be performed after deblock filtering. SAO groups reconstructed pixels into categories and reduces distortion by applying an offset to pixel values based on a classification process. Under the HEVC standard, SAO may be applied for some samples and not applied for other samples. Whether SAO is applied for a particular sample may be signaled in a bitstream. SAO parameters used for one coding tree block (LCU) may also be used for a neighboring LCU, if appropriate.
The conventional SAO signaling protocol defined by the HEVC standard does not specify how flags are signaled for an LCU whose SAO is turned off. By perceiving a need in the art for efficiently using flags, i.e., bits, used for signaling SAO, the inventors have developed methods for efficiently signaling SAO.
Methods and systems of the present disclosure provide techniques for signaling a state of sample adaptive offset (SAO) for a given coding unit. In an embodiment, a method may determine whether the SAO state for a given coding unit is different from an SAO state of a first neighbor in a first scanning direction. If the SAO state for the given coding unit is different from the SAO state of the first neighbor, the method may determine whether an SAO state is off for the first neighbor. If the first neighbor's SAO state is off, the method may signal the SAO state for the given coding unit with a single flag.
In another embodiment, a method may signal a state of SAO for coding units of a frame. The method may be performed iteratively for a plurality of coding units within a frame. The method may include determining whether the respective coding unit has a first neighbor in a first scanning direction. When the respective coding unit does not have a first neighbor, the method may code an SAO state of the respective coding unit according to an SAO state of a coding unit in a second scanning direction in relation to the respective coding unit. Otherwise, the method may code the SAO state of the respective coding unit according to an SAO state of the coding unit in a first direction in relation to the respective coding unit. The method may provide that a protocol for representing SAO state of the coding units at an interior of the frame includes a field for a flag representing state of neighbors in the first scanning direction of the interior coding units but the protocol does not include such a flag for coding units at an edge with respect to the first scanning direction of the frame.
According to an embodiment, a method may signal a sample adaptive offset (SAO) state by signaling SAO data for each coding units in a frame according to a variable field signaling protocol. For a coding unit in an interior of the frame (e.g., not on an edge), when SAO signaling is present for a previously-coded coding unit adjacent to a present coding unit and the SAO state of the previously-coded coding unit agrees with the SAO state of the present coding unit, the signaling comprises providing a flag indicating that the SAO state of the present coding unit is the same as the SAO state of the first previously-coded coding unit. For a coding When SAO signaling is present for a previously-coded coding unit adjacent to the present coding unit in an alternate second direction and an SAO state of the previously-coded coding unit agrees with the SAO state of the present coding unit, the signaling comprises providing a pair of flags, where the first flag indicates that the SAO state of the present coding unit does not agree with the SAO state of the first previously-coded coding unit and a second flag indicates that the SAO state of the present coding unit is the same as the SAO state of the second previously-coded coding unit. When the SAO states of both of the adjacent coding units is not the same as the current coding unit, SAO state data is provided in a four field syntax unit including a pair of flags indicating that the SAO state of the present coding unit does not agree with the SAO state of the first previously-coded coding unit and the SAO state of the second previously-coded coding unit, the syntax unit also including a pair of fields with SAO state values. The techniques described herein may provide savings in the number of bits used for signaling an SAO state compared with conventional techniques. For example, the techniques may reduce the number of fields or flags used for signaling an SAO state.
In
The network 130 represents any number of networks that convey coded video data between the terminals 110, 120, including, for example, wireline and/or wireless communication networks. The communication network 130 may exchange data in circuit-switched or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and/or the Internet. For the purposes of the present discussion, the architecture and topology of the network 130 are immaterial to the operation of the present disclosure unless explained herein.
The subtractor 212 may receive an input motion compensation block from a source image and, depending on a prediction mode used, a predicted motion compensation block from a prediction unit 250. The subtractor 212 may subtract the predicted block from the input block and generate a block of pixel residuals. If no prediction is performed, the subtractor 212 simply may output the input block without modification. The transform unit 214 may convert the block it receives to an array of transform coefficients according to a spatial transform such as a discrete cosine transform (“DCT”) or a wavelet transform. The quantizer 216 may truncate transform coefficients of each block according to a quantization parameter (“QP”). The QP values used for truncation may be transmitted to a decoder in a channel. The entropy coding unit 218 may code the quantized coefficients according to an entropy coding algorithm, for example, a variable length coding algorithm or context-adaptive binary arithmetic coding. Additional metadata containing the message, flag, and/or other information discussed above may be added to or included in the coded data, which may be output by the system 200.
The system 200 also may include an inverse quantization unit 222, an inverse transform unit 224, an adder 226, a filter system 230, a buffer 240, and a prediction unit 250. The inverse quantization unit 222 may quantize coded video data according to the QP used by the quantizer 216. The inverse transform unit 224 may transform re-quantized coefficients to the pixel domain. The adder 226 may add pixel residuals output from the inverse transform unit 224 with predicted motion data from the prediction unit 250. The summed output from the adder 226 may be output to the filtering system 230.
The filtering system 230 may include a deblocking filter 234, a strength derivation unit 232, and a sample adaptive offset (SAO) filter 236. The filters in the filtering system 230 may be applied to reconstructed samples before they are written into a decoded picture buffer 240 in a decoder loop. The deblocking filter 236 may apply deblock filtering to recover video data output from the adder 226 at a strength provided by the strength derivation unit 232. The strength derivation unit 232 may derive a strength value using any of the techniques described herein.
The SAO filter 236 may be configured to perform at least one of the offset features described herein, and in some instances may perform different combinations of two or more of the offset features described herein. SAO filtering may be applied adaptively to all samples satisfying particular conditions defined herein. SAO may modify decoded samples by conditionally adding an offset value to each sample based on values in look-up tables transmitted by an encoder. For example, a classifier index specifying classification of each sample and offsets of the samples may be encoded by entropy coder 218 in a bitstream. In a decoding processor, the classifier index and offsets may be decoded by a corresponding decoder. The filtering system 230 also may include other types of filters, but these are not illustrated in
The buffer 240 may store recovered frame data as output by the filtering system 230. The recovered frame data may be stored for use as reference frames during coding of later-received blocks.
The prediction unit 250 may include a mode decision unit 252 and a motion estimator 254. The motion estimator 254 may estimate image motion between a source image being coded and reference frame(s) stored in the buffer 240. The mode decision unit 252 may assign a prediction mode to code the input block and select a block from the buffer 240 to serve as a prediction reference for the input block. For example, it may select a prediction mode to be used (for example, uni-predictive P-coding or bi-predictive B-coding), and generate motion vectors for use in such predictive coding. In this regard, prediction unit 250 may retrieve buffered block data of selected reference frames from the buffer 240.
A largest coding unit (“LCU”), also known as a coding tree unit (“CTU”), forms the core of the coding layer in HEVC. The LCU corresponds to the macroblock of other coding protocols. A CTU includes a luma coding tree block (“CTB”) and a chroma CTB. According to subclause 7.4.9.3 of the HEVC standard, sao_type_idx_luma specifies an offset type for the luma component of a CTB, and sao_type_idx_chroma specifies an offset type for the chroma component of the CTB. SAO parameters used for one LCU may also be used for a neighboring LCU, if appropriate. For example, merge_left=1 specifies that some syntax elements, including sao_type_idx_luma and sao_type_idx_chroma, are derived from the corresponding syntax elements of a LCU to the left of the LCU being currently coded. merge_left=0 specifies that those syntax elements are not derived from the left LCU. When merge_left is not present, it is inferred to be equal to 0. Similarly, merge_up=1 specifies that some syntax elements, including sao_type_idx_luma and sao_type_idx_chroma, are derived from the corresponding syntax elements of a LCU above the LCU being currently coded. merge_up=0 specifies that those syntax elements are not derived from the above LCU, and when merge_left is not present, it is inferred to be equal to 0.
The existing SAO signaling protocol defined by the HEVC standard does not specify how the merge flags (merge_left and merge_up) are signaled for an LCU whose SAO is turned off. By perceiving a need in the art for efficiently using the number of flags (i.e., bits) used for signaling SAO, the inventors have developed methods for signaling that SAO is off for a LCU.
Sometimes an LCU may be located at an edge, i.e., it does not have a left and/or above neighbor. For example, row 456 may correspond to LCU 404 in which the left LCU 402.2 is on and there is no above LCU (indicated by “EDGE” in table 440). For the case of LCU 404, three syntax elements, merge_left, sao_type_idx_luma and sao_type_idx chroma, may be used for signaling. Because merge_left is FALSE for LCU 404 and there is no data for merge_up, the sao_type_idx_luma and sao_type_idx_chroma flags are also signaled to indicate that SAO for LCU 404 is off. In another example, row 452 corresponds to LCU 414 in which there is no left LCU and above LCU 402.3 is on. For the case of LCU 414, three syntax elements, merge_up, sao_type_idx_luma and sao_type_idx chroma, may be used for signaling. Because merge_up is FALSE for LCU 414 and there is no data for merge_left, the sao_type_idx_luma and sao_type_idx_chroma flags are also signaled to indicate that SAO for LCU 414 is off. In a further example, row 454 may correspond to LCU 424, in which there is no left LCU and the above LCU 414 is off. For the case of LCU 424, one syntax element, merge_up may be used for signaling.
If, in box 602, the method 600 determines that the left LCU does not exist, then the method 600 may skip the merge_left (box 622). This may save signaling resources by saving the bits associated with signaling the merge_left. The method 600 may then proceed to determine whether the above LCU exists (box 624). If the above LCU does not exist, the method 600 may signal sao_type_idx_luma and/or sao_type_idx_chroma in box 634. For example, row 462 of table 440 corresponds to such a result. If, on the other hand, the method 600 determines that the above LCU exists in box 624, then the method 600 may determine in box 626 whether the above LCU has SAO turned on. If LCU is not on for the above LCU, the method 600 may signal merge_up is TRUE (box 628) and terminate. For example, row 454 of table 440 corresponds to such a result. Otherwise, if the method 600 determines that SAO for the above LCU is off, the method may signal merge_up is FALSE (box 632) and proceed to box 634 in which it signals sao_type_idx_luma and/or sao_type_idx_chroma. For example, row 452 of table 440 corresponds to such a result.
In another embodiment, SAO syntax may be signaled at the slice level such that one slice is used for the LCUs 402 with SAO on and another slice is used for the LCUs 404-432 with SAO off. For the LCUs having SAO turned off, the slice_sao_luma_flag and the slice_sao_chroma_flag may be set to 0.
According to another method of signaling SAO, additional flags may be used to signal whether SAO is signaled for left and above LCUs. This may further conserve resources by avoiding signaling of the merge_left and merge_up in some situations. In an embodiment, the sample adaptive offset syntax (found in subclause 7.3.8.3 of the HEVC specification) may be modified to include parameters saoInLeft and saoInUp as follows:
The parameters saoInLeft and saoInUp may specify whether SAO syntax exists in the left and above LCUs respectively. For example, saoInLeft=1 may indicate that SAO syntax is present for the left LCU, while saoInLeft=0 may indicate that there is no SAO syntax for the left LCU. Similarly, saoInUp=1 may indicate that SAO syntax is present for the above LCU, while saoInUp=0 may indicate that there is no SAO syntax for the above LCU. saoInLeft and saoInUp may be implicitly derived as follows:
Other data elements may appear as defined by ITU-T H.265, e.g., SaoTypeIdx may be an array specifying an offset type for a LCU at location (rx, ry).
In box 802, the method 800 may determine whether SAO is performed for an LCU to the left of the LCU being coded, e.g., saoInLeft may be true if SAO is performed for the left LCU. If saoInLeft is true, then the method 800 may signal merge_left in box 804. For example, rows 742, 744, 756 of the table 750 may correspond to this branch. Otherwise, if saoInLeft is not true, the method 800 may skip the merge_left and proceed directly to box 806. For example, rows 746-754, 758, and 762 of the table 750 may correspond to this branch. This may save signaling resources by saving the bits associated with signaling the merge_left. In box 806, the method 800 determines whether SAO is performed for an LCU above the LCU being coded, e.g., saoInUp may be true if SAO is performed for the above LCU. If saoInUp is true, the method 800 may signal merge_up in box 808. For example, rows 742, 746, 752 of the table 750 may correspond to this branch. Otherwise, if saoInUp is not true, the method 800 may skip the merge_up and proceed directly to box 812. For example, rows 744, 748, 754-762 of the table 750 may correspond to this branch.
To determine whether either or both sao_type_idx_luma and sao_type_idx_chroma are signaled, the method 800 may determine in box 812 whether SAO is enabled for a luma component and the method 800 may determine in box 816 whether SAO is enabled for a chroma component. For example, according to subclause 7.4.7.1 of the HEVC standard, slice_sao_luma_flag equals 1 specifies that SAO is enabled for the luma component in the current slice; slice_sao_luma_flag equals 0 specifies that SAO is disabled for the luma component in the current slice; and when slice_sao_luma_flag is not present, it is inferred to be 0. Likewise, slice_sao_chroma_flag equals 1 specifies that SAO is enabled for the chroma component in the current slice; slice_sao_chroma_flag equals 0 specifies that SAO is disabled for the chroma component in the current slice; and when slice_sao_chroma_flag is not present, it is inferred to be 0.
If it is determined in box 812 that the luma component is enabled for the current slice, e.g., slice_sao_luma==1, the method 800 may signal sao_type_idx_luma in box 814 and proceed to box 816. If it is determined in box 812 that the luma component is not enabled, the method 800 may proceed to box 816 without signaling sao_type_idx_luma. In box 816 the method 800 may evaluate whether SAO is enabled for the chroma component, e.g., slice_sao_chroma==0. If the chroma component uses SAO, then the method 800 may proceed to step 818 in which it signals sao_type_idx_chroma. If the chroma component does not use SAO, the method 800 may terminate without signaling sao_type_idx_chroma.
The method 900 may perform boxes 916 to 926 at a decoding terminal. In box 916, the method 900 may receive coded data of the LCU. The method 900 may then identify coding decisions of a predetermined type for the received LCU (box 918). If the method 900 determines in box 922 that the coding decision of the predetermined type exceeds a threshold, then the method 900 may skip SAO signaling while parsing the coded data of the LCU (box 924). Otherwise, the method 900 may recognize that SAO offsets are signaled in the bitstream and parse the coded data of the LCU accordingly (box 926).
According to this embodiment, an LCU's coding parameters may indicate whether SAO filtering is used for the entire LCU. The coding decisions considered by method 900 to determine whether SAO signaling is skipped may include: a number (including percentage or ratio) of coding units (“CUs”) within the LCU that are skipped, a number of coefficients or energy level of coefficients surviving entropy coding, a number of motion vectors for the CUs, and prediction modes for the CUs. For instance, a percentage of CUs within the LCU exceeding a threshold may indicate that a large number of CUs have been skipped and SAO signaling for the entire LCU may be skipped. An energy level of coefficients below a threshold may indicate that a reference frame has relatively few variations and SAO signaling may be skipped. A small number of motion vectors may also indicate that a particular portion of the bitstream may not benefit from SAO, and the entire LCU may skip SAO filtering. Whether SAO filtering is skipped for an entire LCU may be implicitly signaled, for example by using a flag at the sequence (SPS) or picture parameter set (PPS) level.
As used in the appended claims, the term “computer-readable medium” may include a single medium or multiple media, such as a centralized or distributed database, and/or associated caches and servers that store one or more sets of instructions. The term shall also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by a processor or that cause a computer system to perform any one or more of the embodiments disclosed herein.
The computer-readable medium may comprise a non-transitory computer-readable medium or media and/or comprise a transitory computer-readable medium or media. In a particular non-limiting, exemplary embodiment, the computer-readable medium may include a solid-state memory such as a memory card or other package that houses one or more non-volatile read-only memories. Further, the computer-readable medium may be a random access memory or other volatile re-writable memory. Additionally, the computer-readable medium may include a magneto-optical or optical medium, such as a disk or tapes or other storage device to capture carrier wave signals such as a signal communicated over a transmission medium. Accordingly, the disclosure is considered to include any computer-readable medium or other equivalents and successor media, in which data or instructions may be stored.
The present specification describes components and functions that may be implemented in particular embodiments which may operate in accordance with one or more particular standards and protocols. However, the principles of the present disclosure may find application with other standards and protocols as they are defined.
Operation of the disclosed embodiments has been described in the context of terminals that implement video compression, coding, and decoding. These systems can be embodied in electronic devices or integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors. Alternatively, they can be embodied in computer programs that execute on personal computers, notebook computers, tablets, smartphones or computer servers. Such computer programs typically are stored in physical storage media such as electronic-, magnetic- and/or optically-based storage devices, where they may be read to a processor, under control of an operating system and executed. And, of course, these components may be provided as hybrid systems that distribute functionality across dedicated hardware components and programmed general-purpose processors, as desired.
Several embodiments of the disclosure are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations of the disclosure are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the disclosure.
The present application claims priority to U.S. Patent Application Ser. No. 62/004,451, filed May 29, 2014, the disclosure of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62004451 | May 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14726365 | May 2015 | US |
| Child | 15682922 | US |
