Patent Application
20240357110
The present document relates to image coding technology, and more particularly, to an image coding method and apparatus based on a low-frequency non-separable transform in an image coding system.
Recently, the demand for high resolution, high quality image/video such as 4K or 8K Ultra High Definition (UHD) image/video is increasing in various fields. As the image/video resolution or quality becomes higher, relatively more amount of information or bits are transmitted than for conventional image/video data. Therefore, if image/video data are transmitted via a medium such as an existing wired/wireless broadband line or stored in a legacy storage medium, costs for transmission and storage are readily increased.
Moreover, interests and demand are growing for virtual reality (VR) and artificial reality (AR) contents, and immersive media such as hologram; and broadcasting of images/videos exhibiting image/video characteristics different from those of an actual image/video, such as game images/videos, are also growing.
Therefore, a highly efficient image/video compression technique is required to effectively compress and transmit, store, or play high-resolution, high-quality images/videos showing various characteristics as described above.
The technical problem of the present document is to provide a method and an apparatus for enhancing image coding efficiency.
Another technical problem of the present document is to provide an image coding method and apparatus to which LFNST is applied under various conditions.
Another technical problem of the present document is to provide an image coding method and apparatus that sets the LFNST kernel by considering the computational complexity of the sample.
Another technical problem of the present document is to provide an image coding method and apparatus using LFNST that can improve coding performance and minimize complexity.
According to an embodiment of the present document, an image decoding method performed by a decoding apparatus is provided. The method comprises deriving modified transform coefficients based on an inverse secondary transform on transform coefficients, wherein the deriving the modified transform coefficients comprises deriving a transform kernel to be applied for the inverse secondary transform, wherein based on both of horizontal and vertical lengths of the target block being greater than or equal to 4, and the horizontal or the vertical length being equal to 4, the transform kernel may be derived as a 16×16 matrix, and wherein based on both of the horizontal and the vertical lengths of the target block being greater than or equal to 8, and the horizontal or the vertical length being equal to 8, the transform kernel may be derived as a 48×16 matrix.
According to an embodiment of the present document, an image encoding method performed by an encoding apparatus is provided. The method comprises deriving modified transform coefficients based on a secondary transform on the transform coefficients, wherein the deriving the modified transform coefficients comprises deriving a transform kernel to be applied for the secondary transform, wherein based on both of horizontal and vertical lengths of the target block being greater than or equal to 4, and the horizontal or the vertical length being equal to 4, the transform kernel is derived as a 16×16 matrix, and wherein based on both of the horizontal and the vertical lengths of the target block being greater than or equal to 8, and the horizontal or the vertical length being equal to 8, the transform kernel is derived as a 16×48 matrix.
According to another embodiment of the present document, a digital storage medium storing image data including encoded image information and/or bitstream generated based on an image encoding method performed by an encoding apparatus, and a method for transmitting such image information and/or bitstream may be provided.
According to another embodiment of the present document, a digital storage medium storing image data including encoded image information and/or bitstream that causes the image decoding method to be performed by a decoding apparatus, and a method for transmitting such image information and/or bitstream may be provided.
According to an embodiment of the present document, overall video/image compression efficiency may be improved.
According to an embodiment of the present document, LFNST may be applied based on various conditions.
According to an embodiment of the present document, the LFNST set index may be efficiently derived based on the intra prediction mode.
According to the present document, the LFNST kernel can be set by considering the computational complexity of the sample.
According to the present document, an image coding method and apparatus on which LFNST that may improve coding performance and minimize complexity is applied may be provided.
The effects that can be achieved through specific examples of the present specification are not limited to the effects listed above. For example, there may be various technical effects that a person having ordinary skill in the related art can understand or derive from the present specification. Accordingly, the specific effects of the present specification are not limited to those explicitly described in the present specification, and may include various effects that can be understood or derived from the technical features of the present specification.
This disclosure may be modified in various forms, and specific embodiments thereof will be described and shown in the drawings. However, the embodiments are not intended for limiting this disclosure. The terms used in the following description are used to merely describe specific embodiments, but are not intended to limit this disclosure. An expression of a singular number includes an expression of the plural number, so long as it is clearly read differently. The terms such as “include” and “have” are intended to indicate that features, numbers, steps, operations, elements, components, or combinations thereof used in the following description exist and it should be thus understood that the possibility of existence or addition of one or more different features, numbers, steps, operations, elements, components, or combinations thereof is not excluded.
Meanwhile, each of the components in the drawings described in this disclosure are shown independently for the convenience of description regarding different characteristic functions, and do not mean that the components are implemented in separate hardware or separate software. For example, two or more of each configuration may be combined to form one configuration, or one configuration may be divided into a plurality of configurations. Embodiments in which each configuration is integrated and/or separated are also included in the scope of this disclosure without departing from the spirit of this disclosure.
Hereinafter, exemplary embodiments of this disclosure will be described in detail with reference to the accompanying drawings. Hereinafter, the same reference numerals are used for the same components in the drawings, and redundant description of the same components may be omitted.
The present document relates to a video/image coding. For example, the method/embodiment disclosed in the present document may relates to the Versatile Video Coding (VVC) standard (ITU-T Rec. H.266), the next-generation video/image coding standard after VVC, or other video coding-related standards (For example, the High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), essential video coding (EVC) standard, AVS2 standard, etc.).
The present document presents various embodiments of video/image coding, and unless otherwise specified, the embodiments may be performed in combination with each other. In this document, video may refer to a series of images over time.
Technical features described individually in a drawing in the present specification may be implemented individually or simultaneously.
Referring to
The source device may include a video source, an encoding apparatus, and a transmitter. The receiving device may include a receiver, a decoding apparatus, and a renderer. The encoding apparatus may be called a video/image encoding apparatus, and the decoding apparatus may be called a video/image decoding apparatus. The transmitter may be included in the encoding apparatus. The receiver may be included in the decoding apparatus. The renderer may include a display, and the display may be configured as a separate device or an external component.
The video source may acquire video/image through a process of capturing, synthesizing, or generating the video/image. The video source may include a video/image capture device and/or a video/image generating device. The video/image capture device may include, for example, one or more cameras, video/image archives including previously captured video/images, and the like. The video/image generating device may include, for example, computers, tablets and smartphones, and may (electronically) generate video/images. For example, a virtual video/image may be generated through a computer or the like. In this case, the video/image capturing process may be replaced by a process of generating related data.
The encoding apparatus may encode input video/image. The encoding apparatus may perform a series of procedures such as prediction, transform, and quantization for compression and coding efficiency. The encoded data (encoded video/image information) may be output in the form of a bitstream.
The transmitter may transmit the encoded image/image information or data output in the form of a bitstream to the receiver of the receiving device through a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage mediums such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, and the like. The transmitter may include an element for generating a media file through a predetermined file format and may include an element for transmission through a broadcast/communication network. The receiver may receive/extract the bitstream and transmit the received bitstream to the decoding apparatus.
The decoding apparatus may decode the video/image by performing a series of procedures such as dequantization, inverse transform, and prediction corresponding to the operation of the encoding apparatus.
The renderer may render the decoded video/image. The rendered video/image may be displayed through the display.
Referring to
The image partitioner 210 may partition an input image (or a picture or a frame) input to the encoding apparatus 200 into one or more processing unit. For example, the processing unit may be called a coding unit (CU). As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving a transform coefficient and/or a unit for deriving a residual signal from the transform coefficient.
The unit may be used interchangeably with terms such as block or area in some cases. In a general case, an M×N block may represent a set of samples or transform coefficients composed of M columns and N rows. A sample may generally represent a pixel or a value of a pixel.
The subtractor 231 may generate a residual signal (residual block, residual samples or residual sample array) by subtracting the prediction signal (predicted block, prediction samples, or prediction sample array) output from the predictor 220 from the input image signal (original block, original samples, or original sample array). The generated residual signal is transmitted to the transformer 232.
The predictor 220 may perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The predictor 220 may determine whether intra prediction or inter prediction is applied on a current block or CU basis. The predictor may generate various kinds of information related to prediction, such as prediction mode information, and transmit the generated information to the entropy encoder 240. The information on the prediction may be encoded in the entropy encoder 240 and output in the form of a bitstream.
The intra predictor 222 may predict the current block by referring to the samples (reference samples) in the current picture. In the intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra predictor 222 may determine the prediction mode being applied to the current block by using a prediction mode being applied to a neighboring block.
The inter predictor 221 may derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. Here, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, subblocks, or samples based on correlation of motion information between the neighboring block and the current block. The motion information may include a motion vector and a reference picture index.
The prediction signal generated through the inter predictor 221 and/or the intra predictor 222 may be used to generate a reconstructed signal or a residual signal.
The transformer 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include a discrete cosine transform (DCT), a discrete sine transform (DST), a graph-based transform (GBT), Karhunen Loeve Transform(KLT), or a conditionally non-linear transform (CNT), etc. The transformer 232 may perform primary transform and/or secondary transform.
The quantizer 233 may quantize the transform coefficients and transmit them to the entropy encoder 240 and the entropy encoder 240 may encode the quantized signal (information on the quantized transform coefficients) and output a bitstream. The information on the quantized transform coefficients may be referred to as residual information. The quantizer 233 may rearrange block type quantized transform coefficients into a one-dimensional vector form based on a coefficient scanning order and generate information on the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.
The entropy encoder 240 may perform various encoding methods such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), and the like. The entropy encoder 240 may encode information necessary for video/image reconstruction other than quantized transform coefficients (ex. values of syntax elements, etc.) together or separately. A transmitter (not shown) that transmits the signal output from the entropy encoder 240 and/or a storage (not shown) that stores the signal output from the entropy encoder 240 may be configured as internal/extremal elements of the encoding apparatus 200, or the transmitter may be included in the entropy encoder 240.
The quantized transform coefficients output from the quantizer 233 may be used to generate a prediction signal. For example, the residual signal (residual block or residual samples) may be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients through the dequantizer 234 and the inverse transformer 235.
The adder 250 adds the reconstructed residual signal to the prediction signal output from the predictor 220 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed samples, or reconstructed sample array). If there is no residual for the block to be processed, such as a case where the skip mode is applied, the predicted block may be used as the reconstructed block. The generated reconstructed signal may be used for intra prediction of a next block to be processed in the current picture and may be used for inter prediction of a next picture through filtering as described below.
The filter 260 may improve subjective/objective image quality by applying filtering to the reconstructed signal. For example, the filter 260 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture and store the modified reconstructed picture in the memory 270, specifically, a DPB of the memory 270. The various filtering methods may include, for example, deblocking filtering, a sample adaptive offset (SAO), an adaptive loop filter, a bilateral filter, and the like.
The modified reconstructed picture transmitted to the memory 270 may be used as the reference picture in the inter predictor 280. When the inter prediction is applied through the encoding apparatus, prediction mismatch between the encoding apparatus 200 and the decoding apparatus may be avoided and encoding efficiency may be improved.
The DPB of the memory 270 DPB may store the modified reconstructed picture for use as a reference picture in the inter predictor 221. The memory 270 may store the motion information of the block from which the motion information in the current picture is derived (or encoded) and/or the motion information of the blocks in the picture that have already been reconstructed. The memory 270 may store reconstructed samples of reconstructed blocks in the current picture and may transfer the reconstructed samples to the intra predictor 222.
Referring to
When a bitstream including video/image information is input, the decoding apparatus 300 may reconstruct an image corresponding to a process in which the video/image information is processed in the encoding apparatus of
The decoding apparatus 300 may receive a signal output from the encoding apparatus of
The dequantizer 321 may dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 321 may rearrange the quantized transform coefficients in the form of a two-dimensional block form. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding apparatus. The dequantizer 321 may perform dequantization on the quantized transform coefficients by using a quantization parameter (ex. quantization step size information) and obtain transform coefficients.
The inverse transformer 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array). The inverse transformer 322 may perform inverse primary transform and/or inverse secondary transform.
The predictor 330 may perform prediction on the current block and generate a predicted block including prediction samples for the current block.
The intra predictor 331 may predict the current block by referring to the samples in the current picture. The referred samples may be located in the neighborhood of the current block or may be located apart according to the prediction mode. In the intra prediction, prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra predictor 332 may determine the prediction mode being applied to the current block by using a prediction mode being applied to a neighboring block.
The inter predictor 332 may derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In the case of inter prediction, the neighboring block may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. For example, the inter predictor 331 may configure a motion information candidate list based on neighboring blocks and derive a motion vector of the current block and/or a reference picture index based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the information on the prediction may include information indicating a mode of inter prediction for the current block.
The adder 340 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the obtained residual signal to the prediction signal (predicted block, prediction sample array) output from the predictor 330. If there is no residual for the block to be processed, such as when the skip mode is applied, the predicted block may be used as the reconstructed block. The generated reconstructed signal may be used for intra prediction of a next block to be processed in the current picture, may be output through filtering as described below, or may be used for inter prediction of a next picture.
The filter 350 may improve subjective/objective image quality by applying filtering to the reconstructed signal. For example, the filter 350 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture. The various filtering methods may include, for example, deblocking filtering, a sample adaptive offset, an adaptive loop filter, a bilateral filter, and the like.
The (modified) reconstructed picture stored in the DPB of the memory 360 may be used as a reference picture in the inter predictor 331. The memory 360 may store the motion information of the block from which the motion information in the current picture is derived (or decoded) and/or the motion information of the blocks in the picture that have already been reconstructed.
In the present specification, the embodiments described in the predictor 330, the inverse quantizer 321, the inverse transformer 322, the filter 350, and the like of the decoding apparatus 300 may be applied in the same or corresponding manner, respectively, in the predictor 220, the inverse quantizer 234, the inverse transformer 235, the filter 260, and the like of the encoding apparatus 200.
In addition, the content streaming system to which the present document are applied may include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device, in a big picture.
The encoding server serves to compress the content input from the multimedia input devices such as a smartphone, a camera, and a camcorder into the digital data to generate a bitstream and transmit the bitstream to the streaming server. As another example, if the multimedia input devices such as a smartphone, a camera, and a camcorder directly generate the bitstream, the encoding server may be omitted. The bitstream to which the present document is applied may be generated by the encoding method or the bitstream generation method, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.
The streaming server serves to transmit the multimedia data to the user device based on the user request through the web server, and the web server serves as a medium which informs the user of what services are available. When the user requests the desired service to the web server, the web server delivers the user's request to the streaming server, and the streaming server transmits the multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server serves to control commands/responses between the devices within the content streaming system.
The streaming server may receive the contents from the media storage and/or the encoding server. For example, when receiving the contents from the encoding server, the streaming server may receive the contents in real time. In this case, to provide the smooth streaming service, the streaming server may store the bitstream for a predetermined time.
As an example of the user device, there may be a portable phone, a smartphone, a laptop computer, a digital broadcast terminal, a personal digital assistants (PDA), a portable multimedia player (PMP), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smart watch, a smart glass, a head mounted display (HMD)), a digital TV, a desktop computer, a digital signage, or the like. The respective servers within the content streaming system may be operated by a distribution server, and in this case, the data received by each server may be processed separately.
Meanwhile, intra prediction modes may include non-directional (or non-angular) intra prediction modes and directional (or angular) intra prediction modes.
Referring to
Referring to
The transformer may derive (primary) transform coefficients by performing a primary transform based on residual samples (residual sample array) in a residual block (S610). This primary transform may be referred to as a core transform. Herein, the primary transform may be based on multiple transform selection (MTS), and when a multiple transform is applied as the primary transform, it may be referred to as a multiple core transform.
The multiple core transform may represent a method of transforming additionally using discrete cosine transform (DCT) type 2 and discrete sine transform (DST) type 7. DCT type 8, and/or DST type 1. That is, the multiple core transform may represent a transform method of transforming a residual signal (or residual block) of a space domain into transform coefficients (or primary transform coefficients) of a frequency domain based on a plurality of transform kernels selected from among the DCT type 2, the DST type 7, the DCT type 8 and the DST type 1. Herein, the DCT type 2, the DST type 7, the DCT type 8, and the DST type 1 may be called a transform type, transform kernel or transform core. These DCT/DST transform types can be defined based on basis functions.
When the multiple core transform is performed, a vertical transform kernel and a horizontal transform kernel for a target block may be selected from among the transform kernels, a vertical transform may be performed on the target block based on the vertical transform kernel, and a horizontal transform may be performed on the target based on the horizontal transform kernel. Here, the horizontal transform may indicate a transform on horizontal components of the target block, and the vertical transform may indicate a transform on vertical components of the target block.
According to an example, if the primary transform is performed by applying the MTS, a mapping relationship for transform kernels may be set by setting specific basis functions to predetermined values and combining basis functions to be applied in the vertical transform or the horizontal transform. For example, when the horizontal transform kernel is expressed as trTypeHor and the vertical direction transform kernel is expressed as trTypeVer, a trTypeHor or trTypeVer value of 0 may be set to DCT2, a trTypeHor or trTypeVer value of 1 may be set to DST7, and a trTypeHor or trTypeVer value of 2 may be set to DCT8.
In this case, MTS index information may be encoded and signaled to the decoding apparatus to indicate any one of a plurality of transform kernel sets. For example, an MTS index of 0 may indicate that both trTypeHor and trTypeVer values are 0, an MTS index of 1 may indicate that both trTypeHor and trTypeVer values are 1, an MTS index of 2 may indicate that the trTypeHor value is 2 and the trTypeVer value. Is 1, an MTS index of 3 may indicate that the trTypeHor value is 1 and the trTypeVer value is 2, and an MTS index of 4 may indicate that both both trTypeHor and trTypeVer values are 2.
In one example, transform kernel sets according to MTS index information are illustrated in the following table.
The transformer may derive modified (secondary) transform coefficients by performing the secondary transform based on the (primary) transform coefficients (S620). The primary transform is a transform from a spatial domain to a frequency domain, and the secondary transform refers to transforming into a more compressive expression by using a correlation existing between (primary) transform coefficients. The secondary transform may include a non-separable transform. In this case, the secondary transform may be referred to as a non-separable secondary transform (NSST), or a mode-dependent non-separable secondary transform (MDNSST).
The non-separable secondary transform may represent a transform which generates modified transform coefficients (or secondary transform coefficients) for a residual signal by secondary-transforming, based on a non-separable transform matrix, (primary) transform coefficients derived through the primary transform. At this time, the vertical transform and the horizontal transform may not be applied separately (or horizontal and vertical transforms may not be applied independently) to the (primary) transform coefficients, but the transforms may be applied at once based on the non-separable transform matrix. For example, in case of the non-separable secondary transform, two-dimensional signals (transform coefficients) are re-arranged to a one-dimensional signal through a certain determined direction (e.g., row-first direction or column-first direction), and then modified transform coefficients (or secondary transform coefficients) are derived based on the matrix operation between this one-dimensional vector and the non-separable transform matrix.
For example, according to a row-first order, M×N blocks are disposed in a line in an order of a first row, a second row, . . . , and an Nth row. According to a column-first order, M×N blocks are disposed in a line in an order of a first column, a second column, . . . , and an Nth column. That is, for the non-separable secondaryc transform, the transform coefficients derived through the primary transform may be arranged in a one-dimensional vector according to the row-first order direction and then, the matrix operation may be performed to the transform coefficients, or arranged in a one-dimensional vector according to the column-first order direction, and then, the matrix operation may be performed to the transform coefficients.
The non-separable secondary transform may be applied to a top-left region of a block configured with (primary) transform coefficients (hereinafter, may be referred to as a transform coefficient block or a transform block). For example, if the width (W) and the height (H) of the transform coefficient block are all equal to or greater than 8, an 8×8 non-separable secondary transform may be applied to a top-left 8×8 region of the transform coefficient block. Further, if the width (W) and the height (H) of the transform coefficient block are all equal to or greater than 4, and the width (W) or the height (H) of the transform coefficient block is less than 8, then a 4×4 non-separable secondary transform may be applied to a top-left min(8, W)×min(8,H) region of the transform coefficient block. However, the embodiment is not limited to this, and for example, even if only the condition that the width (W) or height (H) of the transform coefficient block is equal to or greater than 4 is satisfied, the 4×4 non-separable secondary transform may be applied to the top-left min(8, W)×min(8,H) region of the transform coefficient block. In summary, the non-separable secondary transform may be applied to a 4×4 or 8×8 region at the top-left of the transform block according to the size of the transform block. According to an example, the transform for the top-left 4×4 region may be named a 4×4 transform, and the transform for the top-left 8×8 region may be referred to as an 8×8 transform.
Here, to select a transform kernel, two non-separable secondary transform kernels per transform set for a non-separable secondary transform may be configured for both the 8×8 transform and the 4×4 transform, and there may be four transform sets. That is, four transform sets may be configured for the 8×8 transform, and four transform sets may be configured for the 4×4 transform. In this case, each of the four transform sets for the 8×8 transform may include two 8×8 transform kernels, and each of the four transform sets for the 4×4 transform may include two 4×4 transform kernels.
However, as the size of the transform, that is, the size of a region to which the transform is applied, may be, for example, a size other than 8×8 or 4×4, the number of sets may be n, and the number of transform kernels in each set may be k.
The transform set may be referred to as an NSST set or an LFNST set. A specific set among the transform sets may be selected, for example, based on the intra prediction mode of the current block (CU or subblock). A low-frequency non-separable transform (LFNST) may be an example of a reduced non-separable transform, which will be described later, and may represent a non-separable transform for a low frequency component.
On the other hand, when it is determined that a specific set is used for the non-separable transform, one of k transform kernels in the specific set may be selected through a non-separable secondary transform index. An encoding apparatus may derive a non-separable secondary transform index indicating a specific transform kernel based on a rate-distortion (RD) check and may signal the non-separable secondary transform index to a decoding apparatus. The decoding apparatus may select one of the k transform kernels in the specific set based on the non-separable secondary transform index. For example, lfnst index value 0 may refer to a first non-separable secondary transform kernel, lfnst index value 1 may refer to a second non-separable secondary transform kernel, and lfnst index value 2 may refer to a third non-separable secondary transform kernel. Alternatively, lfnst index value 0 may indicate that the first non-separable secondary transform is not applied to the target block, and lfnst index values 1 to 3 may indicate the three transform kernels.
The transformer may perform the non-separable secondary transform based on the selected transform kernels, and may obtain modified (secondary) transform coefficients. As described above, the modified transform coefficients may be derived as transform coefficients quantized through the quantizer, and may be encoded and signaled to the decoding apparatus and transferred to the dequantizer/inverse transformer in the encoding apparatus.
Meanwhile, as described above, if the secondary transform is omitted, (primary) transform coefficients, which are an output of the primary (separable) transform, may be derived as transform coefficients quantized through the quantizer as described above, and may be encoded and signaled to the decoding apparatus and transferred to the dequantizer/inverse transformer in the encoding apparatus.
The inverse transformer may perform a series of procedures in the inverse order to that in which they have been performed in the above-described transformer. The inverse transformer may receive (dequantized) transformer coefficients, and derive (primary) transform coefficients by performing a secondary (inverse) transform (S630), and may obtain a residual block (residual samples) by performing a primary (inverse) transform on the (primary) transform coefficients (S640). In this connection, the primary transform coefficients may be called modified transform coefficients from the viewpoint of the inverse transformer. As described above, the encoding apparatus and the decoding apparatus may generate the reconstructed block based on the residual block and the predicted block, and may generate the reconstructed picture based on the reconstructed block.
The inverse transformer may derive the modified transform coefficients by applying a transform kernel matrix to transformed (inverse quantized) transform coefficients arranged in a specific order, for example, in a diagonal scan order (specifically, a diagonal scan order starting from the top left corner of the transform block and proceeding in the bottom right direction). The modified transform coefficients may be arranged in two dimensions in the top left region of the transform block according to the direction in which the transform coefficients are scanned for the secondary transform in the transformer, that is, the row-first direction or the column-first direction. When the transformer performs the 4×4 transform, the inverse transformer may arrange the modified transform coefficients in two dimensions in the 4×4 region of the transform block, and when the transformer performs the 8×8 transform, the inverse transformer may arrange the modified transform coefficients in two dimensions in the 8×8 region of the transform block.
Meanwhile, the secondary inverse transform may be NSST, reduced secondary transform (RST) or LFNST, and whether to apply the secondary inverse transform may be determined based on a secondary transform flag parsed from a bitstream. As another example, whether to apply the secondary inverse transform may be determined based on transform coefficients of the residual block.
This secondary inverse transform (i.e. transform kernel, transform matrix or transform kernel matrix) may be determined based on the LFNST (NSST or RST) transform set specified according to the intra prediction mode. Also, as an embodiment, the secondary transform determination method may be determined depending on the primary transform determination method. Depending on the intra prediction mode, various combinations of primary transform and secondary transform may be determined. Also, for example, a region to which a secondary inverse transform is applied may be determined based on the size of the current block.
Meanwhile, as described above, if the secondary (inverse) transform is omitted, (dequantized) transform coefficients may be received, the primary (separable) inverse transform may be performed, and the residual block (residual samples) may be obtained. As described above, the encoding apparatus and the decoding apparatus may generate the reconstructed block based on the residual block and the predicted block, and may generate the reconstructed picture based on the reconstructed block.
Meanwhile, in the present disclosure, a reduced secondary transform (RST) in which the size of a transform matrix (kernel) is reduced may be applied in the concept of NSST in order to reduce the amount of computation and memory required for the non-separable secondary transform. In addition, since the RST is mainly performed in a low-frequency region including non-zero coefficients in a transform block, it may be referred to as a low-frequency non-separable transform (LFNST). The transform index may be named LFNST index.
In this disclosure, the LFNST may mean a transform performed on residual samples of a target block based on a transform matrix having a reduced size. When the reduced transform is performed, the amount of computation required for transform may be reduced due to the reduction in the size of the transform matrix. That is, the LFNST can be used to solve the computational complexity issue that occurs when transforming or non-separable transforming a large block.
Meanwhile, when the secondary inverse transform is performed based on LFNST, the inverse transformer 235 of the encoding apparatus 200 and the inverse transformer 322 of the decoding apparatus 300 may include an inverse reduced secondary transformer which derives modified transform coefficients based on the inverse RST of the transform coefficients, and an inverse primary transformer which derives residual samples for the target block based on the inverse primary transform of the modified transform coefficients. The inverse primary transform refers to the inverse transform of the primary transform applied to the residual. In the present disclosure, deriving a transform coefficient based on a transform may refer to deriving a transform coefficient by applying the transform.
In the present disclosure, a “target block” may refer to a current block to be coded, a residual block, or a transform block.
In the RST according to an example, an N-dimensional vector may be mapped to an R-dimensional vector located in another space, so that the reduced transform matrix may be determined, where R is less than N. N may mean the square of the length of a side of a block to which the transform is applied, or the total number of transform coefficients corresponding to a block to which the transform is applied, and the reduced factor may mean an R/N value. The reduced factor may be referred to as a reduced factor, reduction factor, reduced factor, simple factor or other various terms. Meanwhile, R may be referred to as a reduced coefficient, but according to circumstances, the reduced factor may mean R. Further, according to circumstances, the reduced factor may mean the N/R value.
The size of the reduced transform matrix according to an example may be R×N less than N×N, the size of a conventional transform matrix, and may be defined as in Equation 1 below.
The matrix T in the Reduced Transform block shown in (a) of
In an example, if the size of the block to which the transform is applied is 8×8 and R=16 (i.e., R/N=16/64=1/4), then the RST according to (a) of
In the present disclosure, a matrix operation may be understood as an operation of multiplying a column vector by a matrix, disposed on the left of the column vector, to obtain a column vector.
In Equation 2, r1 to r64 may represent residual samples for the target block and may be specifically transform coefficients generated by applying a primary transform. As a result of the calculation of Equation 2 transform coefficients c1 for the current block may be derived, and a process of deriving c1 may be as in Equation 3.
As a result of the calculation of Equation 3, transform coefficients c1 to ca for the target block may be derived. That is, when R=16, transform coefficients c1 to c16 for the target block may be derived. If, instead of RST, a regular transform is applied and a transform matrix of 64×64 (N×N) size is multiplied to residual samples of 64×1 (N×1) size, then only 16 (R) transform coefficients are derived for the target block because RST was applied, although 64 (N) transform coefficients are derived for the target block. Since the total number of transform coefficients for the target block is reduced from N to R, the amount of data transmitted by the encoding apparatus 200 to the decoding apparatus 300 decreases, so efficiency of transmission between the encoding apparatus 200 and the decoding apparatus 300 can be improved.
When considered from the viewpoint of the size of the transform matrix, the size of the regular transform matrix is 64×64 (N×N), but the size of the reduced transform matrix is reduced to 16×64 (R×N), so memory usage in a case of performing the LFNST can be reduced by an R/N ratio when compared with a case of performing the regular transform. In addition, when compared to the number of multiplication calculations N×N in a case of using the regular transform matrix, the use of the reduced transform matrix can reduce the number of multiplication calculations by the R/N ratio (R×N).
In an example, the transformer 232 of the encoding apparatus 200 may derive transform coefficients for the target block by performing the primary transform and the RST-based secondary transform on residual samples for the target block. These transform coefficients may be transferred to the inverse transformer of the decoding apparatus 300, and the inverse transformer 322 of the decoding apparatus 300 may derive the modified transform coefficients based on the inverse reduced secondary transform (RST) for the transform coefficients, and may derive residual samples for the target block based on the inverse primary transform for the modified transform coefficients.
The size of the inverse RST matrix TN×R according to an example is N×R less than the size N×N of the regular inverse transform matrix, and is in a transpose relationship with the reduced transform matrix TR×N shown in Equation 1.
The matrix Tt in the Reduced Inv. Transform block shown in (b) of
More specifically, when the inverse RST is applied as the secondary inverse transform, the modified transform coefficients for the target block may be derived when the inverse RST matrix TR×NT is multiplied to the transform coefficients for the target block.
Meanwhile, according to an example, the inverse RST may be applied as the inverse primary transform, and in this case, the residual samples for the target block may be derived when the inverse RST matrix TR×NT is multiplied to the transform coefficients for the target block.
In an example, if the size of the block to which the inverse transform is applied is 8×8 and R=16 (i.e., R/N=16/64=1/4), then the RST according to (b) of
In Equation 4, c1 to c16 may represent transform coefficients of the target block, that is, transform coefficients derived through residual coding. As a result of the operation of Equation 4, ri representing modified transform coefficients of the target block or residual samples of the target block may be derived, and the derivation process of r may be the same as Equation 5.
As a result of the operation of Equation 5, r1 to rN indicating modified transform coefficients of the target block or residual samples of the target block may be derived. Since N is 64 in Equation 4, 64 modified transform coefficients can be derived through Equation 5.
Considering the size of the inverse transform matrix, the size of the normal inverse transform matrix is 64×64 (N×N), but the size of the reduced inverse transform matrix is reduced to 64×16 (N×R), and compared to performing the normal inverse transform, the memory usage can be reduced by R/N ratio when performing the inverse RST. In addition, compared to the number of multiplication operations N×N when using the normal inverse transform matrix, the number of multiplication operations can be reduced by an R/N ratio (N×R) when a reduced inverse transform matrix is used. A reduced inverse transform matrix or inverse transform matrix may also be named a reduced transform matrix or a transform matrix if it is not confusing whether it is a transform or an inverse transform.
According to an embodiment of the present disclosure, for a transform in an encoding process, only 48 pieces of data may be selected and a maximum 16×48 transform kernel matrix may be applied thereto, rather than applying a 16×64 transform kernel matrix to 64 pieces of data forming an 8×8 region. Here, “maximum” means that m has a maximum value of 16 in an m×48 transform kernel matrix for generating m coefficients.
That is, when an RST is performed by applying an m×48 transform kernel matrix (m≤16) to an 8×8 region, 48 pieces of data are input and m coefficients are generated. When m is 16, 48 pieces of data are input and 16 coefficients are generated. That is, assuming that 48 pieces of data form a 48×1 vector, a 16×48 matrix and a 48×1 vector are sequentially multiplied, thereby generating a 16×1 vector. In this embodiment, the column vectors of Equation 2 are r1 to r48, the size of the transform matrix is 16×48, and 16 modified transform coefficients (c1 to c16) are derived through matrix operation.
At this time, a 48×1 vector can be configured by appropriately arranging 48 pieces of data constituting an 8×8 region. For example, a 48×1 vector may be constructed based on 48 pieces of data constituting an region excluding the 4×4 region at the bottom right of the 8×8 region. At this time, when matrix operation is performed by applying the maximum 16×48 transform kernel matrix, 16 modified transform coefficients are generated. The 16 modified transform coefficients can be arranged in the top left 4×4 region according to the scanning order, and the top right 4×4 region and the bottom left 4×4 region can be filled with zeros.
For the inverse transform of the decoding process, a transposed matrix of the transform kernel matrix described above may be used. That is, when inverse RST or inverse LFNST is performed as an inverse transform process performed by the decoding apparatus, the input coefficient data to which the inverse RST is applied is composed of a 1-dimensional vector according to a predetermined arrangement order (diagonal scanning order), and the modified coefficient vector obtained by multiplying the 1-dimensional vector by the corresponding inverse RST matrix from the left side may be arranged in a 2-dimensional block according to a predetermined arrangement order.
If RST or LFNST is performed in the 8×8 region and inverse RST or inverse LFNST is performed, the size of the transform matrix of Equation 4 is 48×16, the column vector includes c1 to c16, and 48 modified transform coefficients (r1 to r48) are derived through the matrix operation.
In summary, in the transform process performed by the encoding apparatus, when RST or LFNST is applied to the 8×8 region, the matrix operation of the 48 transform coefficients of the top left, top right, and bottom left regions of the 8×8 region excluding the bottom right region of the 8×8 region among the transform coefficients of the 8×8 region and the 16×48 transform kernel matrix is performed. For the matrix operation, the 48 transform coefficients are input as a one-dimensional array. When this matrix operation is performed, 16 modified transform coefficients are derived, and the 16 modified transform coefficients may be arranged in the top left region of the 8×8 region.
Conversely, in the inverse transform process performed by the encoding apparatus or the decoding apparatus, when the inverse RST or LFNST is applied to the 8×8 region, 16 transform coefficients at the top left of the 8×8 region among the transform coefficients of the 8×8 region may be input in the form of a 1-dimensional array according to a scanning order and subjected to matrix operation with a 48×16 transform kernel matrix. That is, the matrix operation in this case can be expressed as (48×16 matrix)*(16×1 transform coefficient vector)=(48×1 modified transform coefficient vector). Here, since the n×1 vector can be interpreted in the same sense as an n×1 matrix, it can also be expressed as an n×1 column vector. Also, * means matrix multiplication operation. When this matrix operation is performed, the 48 modified transform coefficients can be derived, and the 48 modified transform coefficients can be arranged in the top left, top right, and bottom left regions of the 8×8 region except for the bottom right region.
The following figures are made to explain a specific example of the present specification. Since the names of specific devices or specific signals/messages/fields described in the figures are provided as examples, the technical features of this specification are not limited to the specific names used in the figures below.
In following, various embodiments that extend the LFNST described above are discussed.
The forward LFNST receives as input transform coefficients to which the primary transform is applied. In this case, instead of receiving all transform coefficients as inputs, transform coefficients belonging to a specific region predefined in the transform block may be received as inputs.
When a non-separable transform matrix is derived based on N samples belonging to the ROI, an N×N transform matrix is derived when the corresponding transform matrix is a square matrix. Here, when the R described with reference to
The ROIs in (a) and (b) of
Meanwhile, for a specific forward LFNST transform matrix (that is, assuming that the position of column vectors is fixed), the input samples constituting the ROI may be read in the order shown in
As another example of the ROI, the top left quadrangular region of the transform block may be set as the ROI. That is, in the M×N transform block, the top left m×n (m≤M, n≤N) region can be set as the ROI, from the forward LFNST point of view, the number of input samples (transform coefficients after the first transform) is m×n. In a more specific embodiment, both m and n may be 8, and the dimension of the forward LFNST matrix may be R×64 (R is equal to or less than 64, examples of R values are 16, 32, 48, 64, etc.). A method of selecting R rows from an mn×mn square matrix (e.g., a 64×64 matrix) may be the same as the method of generating an R×96 matrix from a 96×96 described above.
Meanwhile, the ROI may not be composed of only 4×4 sub-blocks as shown in
If the ROI is non-square rather than square (i.e., m≠n for the top left region m×n), symmetry between intra prediction modes that are symmetric for one M×N transform block (For example, modes 24 and 44, as two symmetrical mode around mode 34) cannot be used.
For example, as shown in (b) of
As another example, ROIs with somewhat irregular shapes can also be applied to LFNST. According to one example, the ROI may be composed of lines with different lengths each other, or the ROI may be composed of segments of non-contiguous samples.
Meanwhile, in the VVC standard, different LFNST kernels are configured to be applied according to the transform block size. That is, for a 4×4 transform block or a 4×N/N×4 (N≥8) transform block (a transform block whose horizontal and vertical lengths are both greater than or equal to 4 and the horizontal or vertical length is 4), an LFNST kernel having a 16×16 matrix form applicable to the top left 4×4 region is applied (which can be named LFNST_4×4). Also, for transform blocks whose horizontal and vertical lengths are both equal to or greater than 8, the ROI consists of the top left 4×4 subblock, the 4×4 subblock adjacent to the right of the top left 4×4 subblock, and the 4×4 subblock adjacent to the bottom of the top left 4×4 subblock, and an LFNST kernel with a 16×48 matrix form based on forward LFNST is applied to ROI (can be named LFNST_8×8).
LFNST_4×4 and LFNST_8×8 each consist of 4 sets, each set consists of 2 transform kernels, which set of kernels to apply is determined by the intra prediction mode. For the determined set, which of the two kernels to apply and whether to apply LFNST may be specified through signaling of the LFNST index. If the LFNST index value is 0, LFNST is not applied, if it is 1, the first kernel may be applied, and if it is 2, the second kernel may be applied.
As described above, the LFNST structure in the VVC standard has been simplified and described, but there may be also some exceptions. For example, for a 4×4 transform block and an 8×8 transform block, an 8×16 matrix and an 8×48 matrix sampled from the corresponding matrix are applied as forward LFNST, rather than a 16×16 matrix and a 16×48 matrix, respectively, when the MIP prediction mode is applied, the intra prediction mode may be regarded as a planner mode and the LFNST set may be determined.
Since the LFNST_4×4 and LFNST_8×8 are each composed of 4 LFNST sets, a bundle of LFNST sets named LFNST_4×4 or LFNST_8×8 may be represented by an LFNST set list for convenience of description below.
Meanwhile, in this disclosure, LFNST_8×8 may indicate an LFNST set list applied to a transform block having a horizontal length or a vertical length of 8 with both horizontal and vertical lengths greater than or equal to 8, additionally, an LFNST set list applied to a transform block having both a horizontal length and a vertical length greater than or equal to 16 may be named LFNST_16×16.
Additional embodiments of matrix dimensions and ROIs that LFNST_4×4, LFNST_8×8, and LFNST_16×16 may have are as follows. In the following embodiment, the transform matrix is based on when forward transform is applied.
1. LFNST_4×4 can have a 16×16 matrix, and the ROI can be the top left 4×4 area.
2. LFNST_8×8 can have an R×48 matrix or an S×64 matrix, and 16, 32, and 48 are possible as R values and 16, 32, 48, and 64 are possible as S values. The ROI for the R×48 matrix may be (a) of
3. LFNST_16×16 can have R×96 matrix or S×64 matrix or T×48 matrix, 16, 32, 48, 64, 80, 96 are possible as R values, 16, 32, 48, 64 are available as S values, T values of 16, 32, and 48 are possible. The ROI for the Rx % matrix may be (b) in
As an architecture for LFNST_4×4, LFNST_8×8, and LFNST_16×16, any combination of matrix dimensions and ROI suggested in Nos. 1, 2, and 3 above is possible. For example, in the case of LFNST_4×4, the ROI of the top left 4×4 area is applied to a 16×16 matrix, in the case of LFNST_8×8, the ROI of the top left 8×8 area is applied to a 32×64 matrix, in the case of LFNST_16×16, the ROI shown in (b) of
In addition, if any one pair of LFNST_4×4, LFNST_8×8, and LFNST_16×16 has the same matrix dimension, it can share the LFNST set and LFNST kernel for that pair. For example, if the matrix dimension of LFNST_8×8 is 32×64 and the matrix dimension of LFNST_16×16 is 32×64, the same LFNST set list can be assigned to LFNST_8×8 and LFNST_16×16, and the same ROI can be set (for example, the ROI can be set to the top left 8×8 area).
As another example, when the matrix dimension of LFNST_8×8 is 32×48 and the matrix dimension of LFNST_16×16 is 32×48, the same LFNST set list can be allocated to LFNST_8×8 and LFNST_16×16, and the same ROI can be set (for example, the ROI can be set as shown in (a) of
On the other hand, when inverse LFNST is applied, when an input vector is constructed with R transform coefficients as input and the left side of the input vector is multiplied by an N×R matrix, N output samples (output transform coefficients) are generated. Here, the N×R matrix becomes a transposed matrix of the R×N matrix in the forward LFNST, and N output samples may be arranged in ROIs of
In the VVC standard, the scan order for transform coefficients is hierarchically configured. There is a scan order for CGs and there is an internal scan order for each CG.
If scanning from the bottom left to the top right once in CG units is one scan line, the first scan line consists of 1 CG and the second and third scan lines consist of two and three CGs, respectively, according to the same method, the Nth scan line is also composed of a plurality of CGs.
The ROIs shown in (a) of
As described above, when the number of output transform coefficients in the forward LFNST criterion is R and the number of input samples is N. R may be set less than or equal to N. In particular, as shown in
For example, if a transform coefficient is parsed in a region other than a region in which the LFNST transform coefficient may exist, signaling of the LFNST index may be omitted and it may be inferred that the LFNST is not applied. Here, if an area where LFNST transform coefficients can exist is configured in units of 4×4 subblocks and residual coding is performed in units of corresponding 4×4 subblocks, it can be performed more simply to check whether a transform coefficient exists in an area other than the area where the LFNST transform coefficient can exist.
According to another embodiment, the CG may have a shape other than a 4×4 sub-block, and in this case (e.g. m×n block, m≠n), the R value may be set to a multiple of m×n. In addition, CGs in which forward LFNST output transform coefficients may exist may be composed of the first k CGs arranged according to the scanning order of the CGs.
Basically, the output coefficients of the forward LFNST can be arranged according to the transform coefficient scanning order. Since row vectors of the forward LFNST kernel are usually arranged from top to bottom in order of importance, assuming that the transform coefficients constituting the output vector are arranged in order from top to bottom (here, the output vector is assumed to be a column vector), coefficients can be arranged sequentially, starting with more significant coefficients. It is usually assumed that the order of scanning the transform coefficients is to scan from the most important coefficients, as the distance from the DC position is increased by scanning from the DC position indicated by the top left position, transform coefficients of less importance are arranged and mainly have a value of 0 or close to 0. Therefore, it may be advantageous in terms of coding performance to sequentially arrange the output transform coefficients of the forward LFNST according to the scan order starting from the DC position. Also, in many cases, the residual coding part is designed to increase coding efficiency when transform coefficients having 0 or values close to 0 frequently appear as the distance from the DC position increases.
Meanwhile, the output transform coefficients of the forward LFNST do not necessarily have to be arranged according to one fixed scan order. That is, according to another embodiment, the output transform coefficients of the LFNST may be sorted according to an order other than the scan order.
If it is statistically determined that a scan order other than the scan order in the VVC standard is suitable for the corresponding LFNST output coefficient, in the case where it is known in advance whether or not to apply the LFNST before performing the residual coding, a scan order specific to the LFNST may be applied instead of a previously determined scan order. In addition, when the optimal scan order varies depending on the coding context such as the intra prediction mode, according to an example, a different scan order may be applied to forward LFNST output transform coefficients for each intra prediction mode (or group of intra prediction modes).
According to one example, unlike the LFNST set list, LFNST set, and LFNST kernel configuration per set in VVC (here, the LFNST kernel configuration per set indicates by how many candidate kernels a specific LFNST set is configured, and etc.), the LFNST set list can be applied more subdivided based on the size of the transform block.
For example, different LFNST set lists may be applied for every possible transform block shape (i.e., every possible M×N block), the corresponding set list may be expressed as, for example, LFNST_MxN. Alternatively, the corresponding LFNST set list may be applied to each group by grouping transform block shapes. In the case of the VVC standard, it can be seen that two types of LFNST set lists, namely LFNST_4×4 and LFNST_8×8, are applied by dividing into two groups according to the shape of the transform block. Examples of other groupings are as follows.
1. A separate group is set for cases where both the horizontal and vertical lengths of the transform block are equal to or greater than 16, and the LFNST set list applied to the group can be allocated. Here, the LFNST set list may be named LFNST_16×16. When combined with the grouping of the VVC standard, it can be divided into three groups as (Group 1) 4×4, 4×N/N×4 (N ≥8) transform block, (Group 2) 8×8, 8×N/N×8 (N≥16) transform block, (Group 3) transform block with both width and height greater than or equal to 16, each group and/or the LFNST set list applied to the group may be named LFNST_4×4, LFNST_8×8, or LFNST_16×16.
2. In the grouping in item 1 above, Group 1 can be further divided into 4×4 transform blocks and 4×N/N×4 (N≠8), it can be divided into Group 1A and Group 1B. Group 2 can also be divided into 8×8 transform blocks and 8×N/N×8 (N≥16) transform blocks, it can be classified as Group 2A and Group 2B. Also, Group 3 can be divided into Group 3A and Group 3B through a specific criterion. For example, 16×16 and 16×N/N×16 (N≥16) transform blocks may be set as Group 3A, and the remaining cases may be classified as Group 3B.
In addition, Group 1, Group 2, and Group 3 may or may not be divided into detailed groups as described above. For example, if only Group 1 and Group 3 are divided into detailed groups, all groups may be configured as Group 1A, Group 1B, Group 2, Group 3A, and Group 3B. Naturally, if Group 1, Group 2, and Group 3 are all divided, the groups can be classified as Group 1A, Group 1B, Group 2A, Group 2B, Group 3A, and Group 3B.
In addition to the above two embodiments, grouping can be applied according to various criteria based on the size of the transform block, a corresponding LFNST set list may be assigned to each group. This LFNST set list may be configured differently for each group.
For example, the number of kernels per LFNST set constituting the LFNST set list can be set differently (e.g. For Group 1, the number of LFNST kernels per set is 3, and for Group 2, the number of LFNST kernels per set is 2, that is, for Group 1, the number of LFNST kernels constituting the set is set as 3, for Group 2, the number of LFNST kernels constituting the set is set as 2), in more detail, the number of kernels constituting the set may be set differently for each LFNST set constituting one LFNST set list.
Alternatively, the number of LFNST sets included in each LFNST set list may be set differently, for example, Group 1 can consist of 18 LFNST sets and Group 2 can consist of 10 LFNST sets. Naturally, the dimension of the kernel matrix may be set differently according to the LFNST set list. Taking the VVC standard as an example, LFNST_4×4 consists of a 16×16 matrix and LFNST_8×8 consists of a 16×48 matrix.
More diversely, the dimension of the kernel matrix may be set differently for each LFNST set constituting the LFNST set list. A specific example of the detailed configuration of the LFNST set list is as follows.
1. Group 1 (LFNST_4×4) consists of 18 LFNST sets, and each LFNST set consists of 3 kernels, and the dimension of the corresponding kernel matrix may be 16×16. Group 2 (LFNST_8×8) consists of 18 LFNST sets, and each LFNST set consists of 3 kernels, and the dimension of the corresponding kernel matrix may be 16×48. Group 3 (LFNST_16×16) consists of 18 LFNST sets, and each LFNST set consists of 3 kernels, and the dimension of the corresponding kernel matrix may be 32×6.
2. In the configuration of item 1 above, all LFNST sets can be configured with 2 kernels instead of 3 kernels.
3. In the configuration of item 1 above, all LFNST set lists can be configured with a different number of sets instead of 18 sets. For example, the LFNST set list may be configured with 16, 15, 10, 6, or 4 transform sets.
4. In the configuration of item 1 above, the dimensions of the kernel matrices constituting LFNST_8×8 may be set to 32×48 to 48×48.
5. In the configuration of item 1 above, the dimensions of the kernel matrices constituting LFNST_16×16 may be set to one of 16×96, 48×96, 64×96, 80×96, and 96×96. Here, 96 may represent the number of input samples (input transform coefficients) constituting the ROI in terms of the forward LFNST, and the ROI may be configured as shown in (b) of
6. With the item 1 being the base, item 2, item 3, item 4, and item 5 above can be freely combined. For example, by applying the item 3, the number of LFNST sets is set to 15, and by applying the item 4, the dimensions of the kernel matrices constituting LFNST_8×8 may be set to 32×48.
According to one embodiment, the LFNST set list, LFNST set, and LFNST kernel may be applied based on the coding tool and the mode to be set.
For example, different LFNST set lists, LFNST sets, and LFNST kernels may be applied according to the range of quantization parameter (QP) values.
For example, the LFNST set list applied to the low QP range and the LFNST set list applied to the high QP range may be used separately. Here, the low QP range may indicate a case that is equal to or less than a predefined threshold QP value, and the high QP range may indicate a case that exceeds the predefined QP value. For threshold QP values, 27, 28, 29, 30, 31, etc. can be used.
As another example, all possible QP values can be divided into N sets. Here, the N sets may not include overlapping values. That is, when two different sets are selected among N sets, the intersection between the two sets may be an empty set. At this time, when the union of N sets is a set consisting of all possible QP values, a different LFNST set list, LFNST set, and LFNST kernel can be applied to each of the N sets.
Additionally, when there are M available LFNST set lists, a mapping relationship can be formed between the N sets and the M LFNST set lists. That is, the LFNST set list mapped to each of the N sets may be one of the M LFNST set lists. Naturally, the LFNST set lists mapped to N sets may overlap each other.
For example, if LFNST_4×4 and LFNST_8×8 are present as LFNST set lists, or if LFNST_4×4, LFNST_8×8, and LFNST_16×16 are present, and when M LFNST set lists are present as described above, M LFNST set lists for each of LFNST_4×4, LFNST_8×8, and LFNST_16×16 may be present. More specifically, for LFNST_4×4, a LFNST set list called LFNST_4×4_i (i=1, 2, . . . , M) may be present, and for LFNST_8×8, a LFNST set list called LFNST_8×8_i (i=1, 2, . . . , M) may be present, for LFNST_16×16, a LFNST set list called LFNST_16×16_i (i=1, 2 . . . . , M) may be present.
If the value of M is 2 and No. 1 LFNST set list is mapped for the low QP range and No. 2 LFNST set list is mapped for the high QP range, then for the low QP range, LFNST_4×4_1, LFNST_8×8_1, or LFNST_16×16_1 may be mapped depending on the transform block size, and for the high QP range, LFNST_4×4_2, LFNST_8×8_2, or LFNST_16×16_2 can be mapped depending on the transform block size.
When a tuple of LFNST_4×4, LFNST_8×8, and LFNST_16×16 is represented as (LFNST_4×4, LFNST_8×8, LFNST_16×16), or a pair of LFNST_4×4, LFNST_8×8 is represented as (LFNST_4×4, LFNST_8×8), the tuple or pair in the i-th LFNST set list may be represented as (LFNST_4×4_i, LFNST_8×8_i, LFNST_16×16_i) or (LFNST_4×4_i, LFNST_8×8_i). Regarding to this, when LFNST_4×4, LFNST_8×8, and LFNST_16×16 are present, or when LFNST_4×4 and LFNST_8×8 are present, ‘being mapped to the i-th LFNST set list’ may mean ‘being mapped to (LFNST_4×4_i, LFNST_8×8_i, LFNST_16×16_i) or (LFNST_4×4_i, LFNST_8×8_i)’.
Therefore, mapping to one of M LFNST set lists according to the QP value may mean that an LFNST set list mapped to each QP value are present and the corresponding LFNST set list is applied. For example, if the corresponding LFNST set list is the j-th LFNST set list, (LFNST_4×4_j, LFNST_8×8_j, LFNST_16×16_j) may be applied.
In addition, for example, when M applicable LFNST set lists are present as described above, rather than applying the LFNST set list for the corresponding condition based on whether or not a specific condition (range of QP values) is satisfied, it can be configured to specify the LFNST set list through a high-level syntax element (hereinafter also referred to as HLS element).
The corresponding HLS element may be located in Sequence Parameter Set (SPS), Picture Parameter Set (PPS), and Picture Header (PH), and/or Slice Header (SH), etc, which are syntax tables in which high-level syntax elements are collected. Regarding to this, the corresponding HLS element may have a value from 0 to M−1 to indicate one of M available LFNST set lists.
For example, the corresponding HLS element may be related to LFNST set list index information. As an example, LFNST set list index information may be located in SPS, PPS, PH, SH, etc. Relatedly, the LFNST set list index information may have a value from 0 to M−1 to specify one of M availabe LFNST set lists.
For example, if LFNST_4×4, LFNST_8×8, and LFNST_16×16 or LFNST_4×4 and LFNST_8×8 are present, and the i-th LFNST set list is specified by LFNST set list index information, (LFNST_4×4_i, LFNST_8×8_i, LFNST_16×16_i) or (LFNST_4×4_i, LFNST_8×8_i) may be applied. Additionally, as an example, if signaling for LFNST set list index information is omitted, the value of LFNST set list index information may be inferred as a specific value, and the LFNST set list specified by the inferred value may become the default LFNST set list. As an example, if the default LFNST set list is the k-th LFNST set list, the default LFNST set list may be (LFNST_4×4_k, LFNST_8×8_k, LFNST_16×16_k) or (LFNST_4×4_k, LFNST_8×8_k).
According to an embodiment, different LFNST set lists may be applied for each transform block size. As a more specific example, instead of applying only LFNST_4×4 and LFNST_8×8 as in the VVC standard, different LFNST set lists may be applied to each of cases of 4×4, 4×N/N×4 (N≥8), 8×8, 8×N/N×8 (N≥16), 16×16, 16×N/N×16 (N≥32), 32×32 or more (both horizontal and vertical lengths are 32 or more). Here, the LFNST set list for each block size set may be expressed as LFNST_4×4, LFNST_4×N_N×4, LFNST_8×8, LFNST_8×N_N×8, LFNST_16×16, LFNST_16×N_N×16, LFNST_32×32.
For example, when the ith LFNST set list among the M LFNST set lists for block size sets is applied according to the index value specified by the coding mode, condition or the LFNST set list index information, the ith LFNST set list may be expressed as a tuple (LFNST_4×4_i, LFNST_4×N_N×4_i, LFNST_8×8_i, LFNST_8×N_N×8_i, LFNST_16×16_i, LFNST_16×N_N×16_i, LFNST_32×32_i).
According to the VVC standard, LFNST_4×4 and LFNST_8×8, each indicating an LFNST set list, may each consist of four LFNST sets, and the four LFNST sets may be distinguished by index values of 0, 1, 2, and 3. That is, the LFNST sets may be classified as the 0th LFNST set, the 1st LFNST set, the 2nd LFNST set, and the 3rd LFNST set, and the LFNST set index for each LFNST set may have a value of 0, 1, 2 or 3.
In addition, the VVC standard may support a Wide Angle Intra Prediction (WAIP) mode, where the WAIP mode may correspond to modes −14 to −1 and modes 67 to 80 among intra prediction modes. In the VVC standard, an existing first LFNST set may be mapped and used for WAIP mode without allocating a separate LFNST set for WAIP mode. That is, as shown in Table 2 below, the first LFNST set is mapped to intra-prediction mode values from −14 to 80, respectively.
In Table 2, Intra pred, mode may represent the intra prediction mode for the current block, and the case of that the value of the Intra pred. mode is one of −14 to −1 and 67 to 80 may represent the intra prediction mode for the current block being the WAIP mode.
In this regard, according to an embodiment proposed in this disclosure, unlike the VVC standard, a larger number of LFNST sets may be introduced, and separate LFNST sets may be allocated for WAIP mode. If separate LFNST sets are not allocated to the WAIP mode, the first LFNST set may be allocated as shown in Table 2 above.
According to one embodiment, when modes other than the WAIP mode are named Normal Angle Intra Prediction (NAIP) mode (ie, modes from 0 to 66), if the number of LFNST sets allocated to the NAIP mode is N, and the number of LFNST sets separately allocated only to the WAIP mode is M, the composition of the LFNST set may be represented by (N, M). Examples for (N, M) may be as follows.
Here, LFNST set index values may be assumed to be allocated from 0 to N−1 for N LFNST sets of NAIP mode, and LFNST set index values may be assumed to be allocated from N to N+M−1 for M LFNST sets of WAIP mode. In this case, examples of a mapping table between the intra prediction mode and the LFNST set for the first to sixth embodiments above may be shown in Tables 3 to 8, respectively.
In Tables 3 to 8 above, Intra pred. mode may represent the intra prediction mode for the current block, and the case of that the value of the Intra pred. mode is one of −14 to −1 and 67 to 80 may represent the intra prediction mode for the current block being the WAIP mode.
Hereinafter, the specific LFNST kernel applied to LFNST_4×4, LFNST_8×8, that is, the LFNST matrix, is described. According to one example, the LFNST matrix below can be applied when the ROI for LFNST_8×8 is 4×4 subblocks at from the top left to third location, in scan order, of target transform block ((a) of
Tables 9 to 18 below show an example of kernel coefficient data for LFNST_4×4 that can be applied to 4×N or N×4 blocks (N≥4). In g_lfnst4×4[36][3][16][16] array of Tables 9 to 18, [36] represents that the number of LFNST sets is 36, [3] represents that the number of LNFST kernel candidates per LFNST set is 3, [16][16] represents that it is a 16×16 matrix based on forward LFNST (the corresponding array definitions in Tables 9 to 18 are described according to C/C++ grammar). For example, 16 may represent the horizontal (x-axis) length of the matrix, and 48 may represent the vertical (y-axis)length of the matrix. If Tables 9 to 18 are used for LFNST where the number of LFNST sets is 35, the array can be represented as g_lfnst4×4[35][3][16]1[16].
The ROI to which the LFNST kernel of Tables 9 to 18 can be applied may be the top left 4×4 region. One LFNST kernel consists of 16 transform basis vectors (row vectors), and one vector has a length of 16 ([16][16]). In the case of forward secondary transform performed in the encoding apparatus, the row basis vector of the LFNST kernel (16×16 dimensional matrix) in Tables 9 to 18 and transform coefficient may be multiplied during matrix operation. For the inverse secondary transform performed in the decoding apparatus, the row basis vector of a transposed kernel (16×16 dimensional matrix) of the LFNST kernel below and the transform coefficient may be multiplied.
Tables 9 to 18 may represent a portion of the 36 LFNST sets. As described above, LFNST set can be selected according to the intra prediction mode and mapped according to Table 3 or Table 4. According to Table 3, 35 LFNST sets are used, and according to Table 4, 36 LFNST sets are used. Tables 9 to 18 may be kernels corresponding to specific set numbers among 35 or 36 sets.
Tables 9 and 10 may illustrate the three LFNST kernels applied when the LFNST set index in Table 3 or Table 4 is 0 (when the intra prediction mode is planner mode), and Tables 11 and 12 may illustrate the three LFNST kernels applied when the LFNST set index in Table 3 or Table 4 is 1 (when the intra prediction mode is DC mode), and Tables 13 and 14 may illustrate the three LFNST kernels applied when the LFNST set index of Table 3 or Table 4 is 2 (when the intra prediction mode represents bottom left direction), Table 15 and Table 16 may illustrate the three LFNST kernels applied when the LFNST set index of Table 3 or Table 4 is 18 (when the intra prediction mode represents horizontal direction or vertical direction), and Table 17 and Table 18 may illustrate the three LFNST kernels applied when the LFNST set index of Table 3 or Table 4 is 34 (when the intra prediction mode represents top left direction).
Tables 19 to 33 below show an example of kernel coefficient data for LFNST_8×8 that can be applied to 8×N or N×8 blocks (N≥8). In the g_lfnst8×8[36][3][16][48] array of Tables 19 to 33, [36] indicates that the number of LFNST sets is 36, [3] indicates that the number of LNFST kernel candidates per LFNST set is 3, [16][48] indicates that it is a 16×48 matrix based on forward LFNST (the corresponding array definitions in Tables 19 to 33 are described according to C/C++ grammar). If Tables 19 to 33 are used for LFNST where the number of LFNST sets is 35, the array can be represented as g_lfnst8×8[35][3][16][48].
The ROI to which the LFNST kernel in Tables 19 to 33 can be applied may be an region excluding the bottom right 4×4 region from the top left 8×8 region. One LFNST kernel consists of 16 transform basis vectors (row vectors), and one vector has a length of 48 ([16][48]). In the case of forward secondary transform performed in the encoding apparatus, the row basis vector of the LFNST kernel (16×48 dimension matrix) in Tables 19 to 33 and transform coefficient may be multiplied during matrix operation. For the inverse secondary transform performed in the decoding apparatus, the row basis vector of the transposed kernel (48×16 dimension matrix) of the LFNST kernel below and the transformation coefficient may be multiplied.
Tables 19 to 33 may represent a portion of the 36 LFNST sets. As described above, the LFNST set can be selected based on the intra prediction mode and mapped according to Table 3 or Table 4. According to Table 3, 35 LFNST sets are used, and according to Table 4, 36 LFNST sets are used. Tables 19 to 33 may be kernels corresponding to specific set numbers among 35 or 36 sets.
Tables 19 to 21 illustrate three LFNST kernels applied when the LFNST set index in Table 3 or Table 4 is 0 (when the intra prediction mode is planner mode), Tables 22 to 24 show three LFNST kernels applied when the LFNST set index in Table 3 or Table 4 is 1 (when the intra prediction mode is DC mode), Tables 25 to 27 show three LFNST kernels applied when the LFNST set index in Table 3 or Table 4 is 2 (when the intra prediction mode indicates the bottom left direction), Tables 28 to 30 show three LFNST kernels applied when the LFNST set index in Table 3 or Table 4 is 18 (when the intra prediction mode indicates the horizontal or vertical direction), and Tables 31 to 33 may represent three LFNST kernels applied when the LFNST set index of Table 3 or Table 4 is 34 (when the intra prediction mode indicates the upper left direction). One LFNST kernel is represented by one table. For example, Table 19 is the first LFNST kernel applied when LFNST set index is 0, Table 20 may be the second LFNST kernel applied when the LFNST set index is 0, and Table 21 may be the third LFNST kernel applied when the LFNST set index is 0.
The following figures are made to explain a specific example of the present disclosure. Since the names of specific apparatus or names of specific signals/messages/fields described in the figures are provided as examples, the technical features of the present disclosure are not limited to the specific names used in the figures below.
Each step disclosed in
The decoding apparatus 300 according to one embodiment may receive residual information for the target block from the bitstream (S1310). The target block may be a coding block or a transform block that is subject to coding or transform.
The decoding apparatus can obtain information on quantized transform coefficients from residual information and receive various information for image decoding. More specifically, the decoding apparatus 300 may decode the information on the quantized transform coefficients for the target block from the bitstream, derive quantized transform coefficients for the target block based on information on the quantized transform coefficients for the target block.
In addition, information on LFNST applied to the target block may also be received, and information on LFNST may be included in a Sequence Parameter Set (SPS) or slice header. This information may include at least one of information on whether LFNST is applied, information on the minimum transform size to which LFNST is applied, information on the maximum transform size to which LFNST is applied, and information on a transform index indicating one of the transform kernels included in the transform set.
The decoding apparatus 300 according to one embodiment may derive transform coefficients for the target block based on residual information (S1320). The decoding apparatus may derive transform coefficients by performing inverse quantization on the quantized transform coefficients for the target block.
The derived transform coefficients may be arranged in a reverse diagonal scan order in units of 4×4 subblocks, and transform coefficients in the 4×4 subblocks may also be arranged according to a reverse diagonal scan order. That is, the transform coefficients subjected to inverse quantization may be arranged according to an inverse scan order applied in a video codec such as VVC or HEVC.
The decoding apparatus 300 according to an embodiment may derive a transform kernel to be applied for the inverse secondary transform (S1330), based on the derived transform kernel, inverse secondary transform can be performed on the transform coefficients to derive modified transform coefficients (S1340).
In this case, based on both of the horizontal and vertical lengths of the target block being greater than or equal to 4, and the horizontal or vertical length being equal to 4, the transform kernel is derived as a 16×16 matrix, and based on both of the horizontal and vertical lengths of the target block being greater than or equal to 8, and the horizontal or vertical length being equal to 8, the transform kernel can be derived as a 48×16 matrix.
In addition, according to one example, based on both of the horizontal and vertical lengths of the target block being equal to or greater than 8, and the horizontal or vertical length being equal to 8, the transform kernel is derived as a 64×32 matrix, and based on both of the horizontal and vertical lengths of the target block being equal to 8, a 64×16 matrix sampled from a 64×32 matrix can be applied for the inverse secondary transform of the target block.
According to this document, the inverse secondary transform may include LFNST, that is, a non-separable secondary transform reflecting RST, the inverse secondary transform may be performed based on the LFNST kernel, and the LFNST kernel may be a non-square matrix of which the number of the columns is less than the number of the rows.
According to the preset document, unlike the existing VVC standard, LFNST can be applied to the upper left 16×16 region of a transform block larger than or equal to 16×16. In this document, a P×Q block being greater than or equal to an M×N block means that P and Q are greater than or equal to M and N, respectively. As an example, LFNST can be applied to the region for 96 samples consisting of 6 4×4 sub-blocks in the top left 16×16. In other words, modified transform coefficients of which number being greater than a portion of transform coefficients included in the top left 16×16 region of target block, that is, transform coefficient based on the input array may be derived. According to one example, the decoding apparaus may derive L (48<L≤96) modified transform coefficients based on R transform coefficients of the upper left region of the target block, and the derived L (48<L≤96) modified transform coefficients may be arranged in a predetermined output region. R is smaller than L. Meanwhile, in this document, an M×N block being larger than a K×L block may represent that M and N are larger than or equal to K and L, respectively, and M is larger than K or N is larger than L.
R, the number of input transform coefficients configuring the input array, and L, the number of output transform coefficients arranged in the output region, may change depending on the dimension of the transform kernel. According to one example, R can be 16, 32, 48, 80, etc., and L can be 64 or %.
According to one example, the input array may be arranged in units of 4×4 sub-blocks, which can be arranged in a forward diagonal scanning order from the DC position of the target block, and may be arranged according to a forward diagonal scanning order within the 4×4 sub-block. Accordingly, R, the number of transform coefficients configuring the input array, may be set to a multiple of 16, which is the number of transform coefficients in a 4×4 sub-block.
Since the output region refers to the region of the input transform coefficients input to perform secondary transform in the encoding apparatus, it may refer to the region in which the output transform coefficients are arranged when performing inverse secondary transform in the decoding apparatus. The output region may correspond to the ROI described with reference to the above-described figures.
Meanwhile, the step of deriving the modified transform coefficient may include deriving a transform kernel for transform, and the transform kernel may be derived based on a transform set derived based on the intra prediction mode applied to the target block.
A 16×16 matrix may be configured based on Tables 9 to 18, and a 48×16 dimensional matrix may be configured based on Tables 19 to 33.
Meanwhile, the size of the inverse second transform may be set based on the size of the target block, and based on the size of the inverse secondary transform, at least one of the number of transform sets, the number of transform kernels configuring the transform set, and the dimension of the transform kernel may be derived.
Based on both of the horizontal and vertical lengths of the target block being equal to or greater than 4, and the horizontal or vertical length being 4, the size of the inverse secondary transform may be set to a first value. For example, the first value may be set to 2. The size of the inverse secondary transform being equal to 2, which means that LFNST is applied to the upper left 4×4 region of the target block, may correspond to the LFNST_4×4 described above.
Alternatively, based on both of the horizontal and vertical lengths of the target block being equal to or greater than 8, and the horizontal or vertical length being 8, the size of the inverse secondary transform may be set to a second value. For example, the second value may be set to 3. The size of the inverse secondary transform being equal to 3, which means that LFNST is applied to the upper left 8×8 region of the target block, may correspond to the LFNST_8×8 described above.
Alternatively, based on both of the horizontal and vertical lengths of the target block being equal to or greater than 16, the size of the inverse secondary transform may be set to a third value. For example, the third value may be set to 4. The size of the inverse secondary transform being equal to 4, which means that LFNST is applied to the upper left 16×16 region of the target block, may correspond to the LFNST_16×16 described above.
Based on the size of the inverse secondary transform, that is, grouping according to the size to which LFNST is applied, at least one of the number of transform sets applied to the target block, the number of transform kernels configuring the transform set, and the dimension of the transform kernel may be derived. In other words, the number of transform sets, the number of transform kernels configuring the transform set, and the dimensions of the transform kernel, etc. can be set and configured in various ways according to the size of the inverse secondary transform or the size of the target block.
For example, based on the size of the inverse secondary transform being equal to 2, the dimension of the transform kernel can be set to 16×16. Additionally, based on the size of the inverse secondary transform being equal to 3, the dimension of the transform kernel can be set to 48×R or 64×S, where R may be set to one of 16, 32, or 48, and S may be set to one of 16, 32, 48, or 64. Additionally, based on the size of the inverse secondary transform being equal to 4, the dimension of the transform kernel may be set to one of 96×R, 64×S, or 48×T, where R may be set to one of 16, 32, 48, 64, 80, or 96, and S may be set to one of 16, 32, 48, or 64, and T may be set to one of 16, 32, or 48.
Or, according to one example, based on the size of the inverse secondary transform being equal to 1, that is, if both the horizontal and vertical lengths of the target block are equal to or greater than 4, and the horizontal or vertical length is equal to 4, the dimension of the transform kernel to which LFNST4×4 is applied may be 16×16.
Alternatively, according to one example, based on both of the horizontal and vertical lengths of the target block being equal to or greater than 8, and the horizontal or vertical length being equal to 8, that is, LFNST_8×8 being applied, the transform kernel can be set to a 48×16 matrix.
The decoding apparatus 300 according to an embodiment may derive residual samples for the target block based on the inverse primary transform on the modified transform coefficients (S1350).
The decoding apparatus 300 may perform an inverse primary transform on the modified transform coefficients for the target block, and in this case, for the inverse primarv transform, a reduced inverse transform may be applied or a normal separable transform may be used.
According to one embodiment, LFNST may also be applied to a target block to which DST-7, DCT-8, or Karhunen Loeve Transform (KLT), etc. is as inverse primary transform.
Alternatively, according to one example, LFNST may also be applied to a target block to which a transform skip is applied for horizontal transform or vertical transform.
Based on the transform kernel (DCT-2, DST-7, DCT-8, etc.) applied to the inverse primary transform or a combination of transform kernels, various combination designs about the number of transform sets, the number of transform kernels configuring the transform set, and the dimension of the transform kernel, etc. are possible.
The decoding apparatus 300 according to an embodiment may generate a reconstructed picture based on residual samples for the target block and prediction samples for the target block (S1360).
According to an example in the present document, LFNST can be applied to a target block in which prediction samples are derived based on not only intra prediction but also inter prediction. Transform sets, transform kernels, etc. can be designed in various ways based on motion information, features of motion vectors, etc.
The following figures are made to explain a specific example of the present specification. Since the names of specific devices or specific signals/messages/fields described in the figures are provided as examples, the technical features of this specification are not limited to the specific names used in the figures below.
Each step disclosed in
The encoding apparatus 200 according to one embodiment may derive prediction samples based on the prediction mode applied to the target block (S1410).
According to an example in the preset document, LFNST can be applied to a target block in which prediction samples are derived based on not only intra prediction but also inter prediction. Transform sets, transform kernels, etc., which will be described later, can be designed in various ways based on motion information, features of motion vectors, etc.
The encoding apparatus 200 according to an embodiment may derive residual samples for the target block based on the prediction samples (S1420).
The encoding apparatus 200 according to an embodiment may derive transform coefficients for the target block based on a primary transform on the residual samples (S1430).
Primary transform may be performed through a plurality of transform kernels, and in this case, a transform kernel may be selected based on the intra prediction mode.
As for the primary transform, a reduced inverse transformation may be applied, or a normal separable transform may be used.
DCT-2, DST-7. DCT-8, or KLT (Karhunen Loeve Transform) may be applied as the primary transform, and according to an embodiment of this document, LFNST can also be applied to target blocks to which DST-7, DCT-8 or KLT etc., other than DCT-2, is applied as the primary transform.
Alternatively, according to one example, LFNST may also be applied to a target block to which a transform skip has been applied to horizontal transform or vertical transform.
Based on transform kernel (DCT-2, DST-7, DCT-8, etc.) applied to the primary transform or combinations of the transform kernels, various combination designs for the number of transform sets applied to LFNST, the number of transform kernels configuring the transform set, and the dimension of transform kernel, etc. are possible.
The encoding apparatus 200 according to one embodiment derives a transform kernel to be applied to the secondary transform (S1440), and by performing the secondary transform on the transform coefficients based on the derived transform kernel, the modified transform coefficients for the target block may be derived (S1450).
In this case, based on both of the horizontal and vertical lengths of the target block being greater than or equal to 4, and the horizontal or vertical length being equal to 4, the transform kernel can be derived as a 16×16 matrix, and based on both of the horizontal and vertical lengths of the target block being greater than or equal to 8, and the horizontal or vertical length being equal to 8, the transform kernel can be derived as a 16×48 matrix.
Or, according to one example, based on both of the horizontal and vertical lengths of the target block being greater than or equal to 8, and the horizontal or vertical length being equal to 8, the transform kernel may be derived as a 32×64 matrix, and based on both of the horizontal and vertical lengths of the target block being equal to 8, a 16×64 matrix sampled from a 32×64 matrix can be applied to the secondary transform of the target block.
According to this document, the secondary transform may include LFNST, that is, a non-separable secondary transform reflecting RST, and the secondary transform may be performed based on the LFNST kernel, and the LFNST kernel may be a non-square matrix of which the number of rows is less than the number of columns.
According to this disclosure, unlike the existing VVC standard, LFNST can be applied to the top left 16×16 region of a transform block equal to or larger than 16×16. In this document, a P×Q block is greater than or equal to an M×N block means that each of P and Q are greater than or equal to each of M and N, respectively. For example, LFNST may be applied to a region of 96 samples composed of 6 4×4 sub-blocks in the top left 16×16. That is, fewer modified transform coefficients than the transform coefficients may be derived based on some transform coefficients belonging to the top left 16×16 region of the target block. According to an example, the encoding apparatus is based on L (48<L≤96) transform coefficients of the top left region of the target block based on the size of the target block being M×N (M≥16, N≥16). The derived R modified transform coefficients may be derived as an output array according to a predetermined scanning order. R is smaller than L.
An input region, which means a region of input transform coefficients subject to secondary transform in the encoding apparatus, may correspond to an ROI described with reference to the output region described in the decoding method and the figures described above. Therefore, redundant description of the ROI is omitted.
The number L of input transform coefficients arranged in the input region and the number R of modified transform coefficients derived through the matrix operation may be changed according to the dimension of the transform kernel. According to one example, R can be 16, 32, 48, 80, etc., and L can be 64 or 96.
As described above, deriving the modified transform coefficients may include deriving a transform kernel for transform, and the transform kernel may be derived based on a transform set derived based on the intra prediction mode applied to the target block.
A 16×16 matrix may be configured based on Tables 9 to 18, and a 16×48 dimensional matrix may be configured based on Tables 19 to 33.
Meanwhile, the size of the secondary transform can be set based on the size of the target block, and based on the size of the secondary transform, at least one of the number of transform sets, the number of transform kernels configuring the transform set, and the dimension of the transform kernel may be derived.
Based on both of the horizontal and vertical lengths of the current block being greater than or equal to 4, and the horizontal or vertical length is 4, the size of the secondary transform may be set to a first value. For example, the first value may be set to 2. The size of the secondary transform being equal to 2, which means that LFNST is applied to the upper left 4×4 region of the target block, may correspond to the LFNST_4×4 described above.
Alternatively, based on both of the horizontal and vertical lengths of the target block being greater than or equal to 8, and the horizontal or vertical length is 8, the size of the secondary transform may be set to a second value. For example, the second value may be set to 3. The size of the secondary transform being equal to 3, which means that LFNST is applied to the upper left 8×8 region of the target block, may correspond to the LFNST_8×8 described above.
Alternatively, based on both of the horizontal and vertical lengths of the target block being greater than or equal to 16, the size of the secondary transform may be set to a third value. For example, the third value may be set to 4. The size of the secondary transform being equal to 4, which means that LFNST is applied to the upper left 16×16 region of the target block, may correspond to the LFNST_16×16 described above.
Based on the size of the secondary transform, that is, grouping according to the size to which LFNST is applied, at least one of the number of transform sets applied to the target block, the number of transform kernels configuring the transform set, and the dimension of the transform kernel can be derived. In other words, the number of transform sets, the number of transform kernels configuring the transform set, and the dimensions of the transform kernel, etc. can be set and configured in various ways according to the size of the secondary transform or the size of the target block.
For example, based on the size of the secondary transform being 2, the dimension of the transform kernel can be set to 16×16. Additionally, based on the size of the secondary transform being 3, the dimension of the transform kernel can be set to R×48 or S×64, where R may be set as one of 16, 32, or 48, and S may be set as one of 16, 32, 48, or 64. Additionally, based on the size of the secondary transform being 4, the dimension of the transform kernel can be set to one of R×96, S×64, or T×48, where R may be set as one of 16, 32, 48, 64, 80, and 96. S may be set as one of 16, 32, 48, or 64, and T may be set as one of 16, 32, or 48.
Or, according to one example, based on the size of the secondary transform being 1, that is, if both the horizontal and vertical lengths of the target block are greater than or equal to 4, and the horizontal or vertical length is 4, the dimension of the transform kernel applied to LFNST4×4 may be 16×16.
Or, according to one example, based on both of the horizontal and vertical lengths of the target block being greater than or equal to 8, and the horizontal or vertical length being equal to 8, that is, LFNST_8×8 being applied, the transform kernel may be set to a 16×48 matrix.
The encoding apparatus 200 according to an embodiment may encode image information including residual information for the target block (S1460).
The encoding apparatus may perform quantization based on the modified transform coefficients to derive quantized transform coefficients, and generate and encode residual information related to the quantized transform coefficients. The residual information may include the above-described transform related information/syntax elements. The encoding apparatus may encode image/video information including residual information and output it in a bitstream format.
Additionally, the encoding apparatus can also encode information on the LFNST applied to the target block, the information on the LFNST may be included in a Sequence Parameter Set (SPS) or a slice header (SH). This information includes at least one of information on whether LFNST is applied, information on the minimum transform size to which LFNST is applied, information on the maximum transform size to which LFNST is applied, and a transform index indicating one of the transform kernels included in the transform set.
In this document, at least one of quantization/dequantization and/or transform/inverse transform may be omitted. If the quantization/dequantization is omitted, the quantized transform coefficient may be referred to as a transform coefficient. If the transform/inverse transform is omitted, the transform coefficients may be referred to as coefficients or residual samples, or may still be referred to as transform coefficients for unity of expression.
Further, in the present document, the quantized transform coefficient and the transform coefficient may be referred to as a transform coefficient and a scaled transform coefficient, respectively. In this case, the residual information may include information about the transform coefficient(s), and the information about the transform coefficient(s) may be signaled through a residual coding syntax. The transform coefficients may be derived based on the residual information (or the information about the transform coefficient(s)), and the scaled transform coefficients may be derived through the inverse transform (scaling) for the transform coefficients. The residual samples may be derived based on the inverse transform (transform) for the scaled transform coefficients. This may be likewise applied to/expressed in other parts of the present disclosure.
In the above-described embodiment, the methods are described based on the flowchart having a series of steps or blocks, but this embodiment is not limited to the order of the above steps or blocks and some steps may occur simultaneously or in a different order from other steps as described above. Further, those skilled in the art will understand that the steps shown in the above flowchart are not exclusive, that further steps may be included, or that one or more steps in the flowchart may be deleted without affecting the scope of the embodiments of the present disclosure.
The method according to the embodiments of the present disclosure described above may be implemented in software. The encoding apparatus and/or decoding apparatus according to the present disclosure may be included in a device that performs image processing, for example, a TV, a computer, a smartphone, a set-top box, or a display device.
When the embodiments of the present disclosure are implemented in software, the above-described method may be implemented by modules (processes, functions, and so on) that perform the functions described above. Such modules may be stored in memory and executed by a processor. The memory may be internal or external to the processor, and the memory may be coupled to the processor using various well-known means. The processor may include an application-specific integrated circuit (ASIC), other chipsets, a logic circuit and/or a data processing device. The memory may include a ROM (read-only memory), a RAM (random access memory), a flash memory, a memory card, a storage medium, and/or other storage device. That is, the embodiments described in the present disclosure may be implemented and performed on a processor, a microprocessor, a controller, or a chip. For example, the functional units shown in each drawing may be implemented and executed on a computer, a processor, a microprocessor, a controller, or a chip.
In addition, the decoding apparatus and the encoding apparatus to which the present disclosure are applied may be comprised in multimedia communication devices such as a multimedia broadcasting transmitting and receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video chatting device, (3D) video devices, video telephony video devices, and medical video devices, and the like, which may be included in, for example, a storage medium, a camcorder, a video on demand (VoD) service provision device, an OTT video (Over the top video), an Internet streamlining service providing device, a 3D video device, a video call device, medical video device, and may be used to process video signals or data signals. For example, the OTT video (over the top video) device may include a game console, a Blu-ray player, an Internet access TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR).
Further, the processing method to which the present disclosure are applied may be produced in the form of a computer-executed program, and may be stored in a computer-readable recording medium. The multimedia data having the data structure according to the present disclosure may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium may be, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium includes media implemented in the form of a carrier wave (for example, transmission over the Internet). In addition, the bit stream generated by the encoding method may be stored in a computer-readable recording medium or transmitted over a wired or wireless communication network. Additionally, the embodiments of the present disclosure can be implemented as a computer program product using program code, and the program code can be executed on a computer by the embodiments of the present disclosure. The program code may be stored on a computer readable carrier.
Claims described in the present specification may be combined in various methods. For example, the technical features of method claims of the present specification may be combined and implemented as a device, and the technical features of device claims of the present specification may be combined and implemented as a method. Further, the technical features of the method claims of the present specification and the technical features of the device claims thereof may be combined and implemented as a device, and the technical features of the method claims of the present specification and the technical features of the device claims thereof may be combined and implemented as a method.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/KR2022/008521 | 6/16/2022 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63211519 | Jun 2021 | US |
