Patent Grant
12120342
An embodiment of the disclosure relates to a method and apparatus for processing a video signal based on inter prediction and, more particularly, to a method and apparatus for performing inter prediction using a history-based motion vector prediction.
Compression encoding means a series of signal processing techniques for transmitting digitized information through a communication line or techniques for storing information in a form suitable for a storage medium. The medium including a picture, an image, audio, etc. may be a target for compression encoding, and particularly, a technique for performing compression encoding on a picture is referred to as video image compression.
Next-generation video contents are supposed to have the characteristics of high spatial resolution, a high frame rate and high dimensionality of scene representation. In order to process such contents, a drastic increase in the memory storage, memory access rate and processing power will result.
Accordingly, it is required to design a coding tool for processing next-generation video contents efficiently.
An objective of the embodiment of the disclosure is to proposes constraints on redundancy check for enhancing prediction efficiency, when adding an HMVP candidate to a merge candidate list (or AMVP candidate list).
Another objective of the embodiment of the disclosure is to propose a method of defining a size of a HMVP table in order to efficiently apply an HVMV candidate to the merge candidate list (or AMVP candidate list).
Technical objects to be achieved in an embodiment of the disclosure are not limited to the aforementioned technical objects, and other technical objects not described above may be evidently understood by a person having ordinary skill in the art to which the disclosure pertains from the following description.
In an aspect of an embodiment of the disclosure, a method of processing a video signal based on history-based motion vector prediction may comprise: configuring a merge candidate list based on a spatial neighboring block and a temporal neighboring block to a current block; adding a history-based merge candidate of the current block to the merge candidate list; obtaining a merge index indicating a merge candidate used for an inter prediction of the current block within the merge candidate list; generating a prediction sample of the current block based on motion information of the merge candidate indicated by the merge index; and updating a history-based merge candidate list based on the motion information, wherein the history-based merge candidate is added to the merge candidate list when it has motion information different with a predefined merge candidate of merge candidates included in the merge candidate list.
In an embodiment, a size of the history-based merge candidate list may be determined based on a maximum number of merge candidates within the merge candidate list.
In an embodiment, a size of the history-based merge candidate list may be determined as a value being 1 subtracted from a maximum number of merge candidates within the merge candidate list.
In an embodiment, a size of the history-based merge candidate list may be defined as 5.
In an embodiment, the history-based merge candidate may be added to the merge candidate list when the history-based merge candidate includes motion information different from motion information of a specific number of predefined merge candidates among merge candidates included in the merge candidate list.
In an embodiment, the history-based merge candidate may be added to the merge candidate list when the history-based merge candidate includes motion information different from motion information of specific spatial merge candidates included in the merge candidate list.
In an embodiment, the history-based merge candidate may be derived from a predetermined number of candidates within the history-based merge candidate list.
In another aspect of an embodiment, an apparatus of processing a video signal based on inter prediction may comprise: a memory configured to storing the video signals; and a processor combined to the memory, wherein the processor is configured for: configuring a merge candidate list based on a spatial neighboring block and a temporal neighboring block to a current block; adding a history-based merge candidate of the current block to the merge candidate list; obtaining a merge index indicating a merge candidate used for an inter prediction of the current block within the merge candidate list; generating a prediction sample of the current block based on motion information of the merge candidate indicated by the merge index; and updating a history-based merge candidate list based on the motion information, wherein the history-based merge candidate is added to the merge candidate list when it has motion information different with a predefined merge candidate of merge candidates included in the merge candidate list.
According to an embodiment of the disclosure, complexity according to redundancy check can be improved and compression performance can be enhanced by limiting redundancy check for addition to a merge candidate list (or AMVP candidate list).
According to an embodiment of the disclosure, the memory burden associated with storing a HMVP table can be reduced by defining the size of the HMVP table,
Effects which may be obtained in the disclosure are not limited to the aforementioned effects, and other technical effects not described above may be evidently understood by a person having ordinary skill in the art to which the disclosure pertains from the following description.afd
The accompany drawings, which are included as part of the detailed description in order to help understanding of the disclosure, provide embodiments of the disclosure and describe the technical characteristics of the disclosure along with the detailed description.
Hereinafter, preferred embodiments of the disclosure will be described by reference to the accompanying drawings. The description that will be described below with the accompanying drawings is to describe exemplary embodiments of the disclosure, and is not intended to describe the only embodiment in which the disclosure may be implemented. The description below includes particular details in order to provide perfect understanding of the disclosure. However, it is understood that the disclosure may be embodied without the particular details to those skilled in the art. In some cases, in order to prevent the technical concept of the disclosure from being unclear, structures or devices which are publicly known may be omitted, or may be depicted as a block diagram centering on the core functions of the structures or the devices.
In some cases, in order to prevent the technical concept of the disclosure from being unclear, structures or devices which are publicly known may be omitted, or may be depicted as a block diagram centering on the core functions of the structures or the devices.
Further, although general terms widely used currently are selected as the terms in the disclosure as much as possible, a term that is arbitrarily selected by the applicant is used in a specific case. Since the meaning of the term will be clearly described in the corresponding part of the description in such a case, it is understood that the disclosure will not be simply interpreted by the terms only used in the description of the disclosure, but the meaning of the terms should be figured out.
Specific terminologies used in the description below may be provided to help the understanding of the disclosure. Furthermore, the specific terminology may be modified into other forms within the scope of the technical concept of the disclosure. For example, a signal, data, a sample, a picture, a slice, a tile, a frame, a block, etc may be properly replaced and interpreted in each coding process.
Hereinafter, in this specification, a “processing unit” means a unit in which an encoding/decoding processing process, such as prediction, a transform and/or quantization, is performed. A processing unit may be construed as having a meaning including a unit for a luma component and a unit for a chroma component. For example, a processing unit may correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU) or a transform unit (TU).
Furthermore, a processing unit may be construed as being a unit for a luma component or a unit for a chroma component. For example, the processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PB) or a transform block (TB) for a luma component. Alternatively, a processing unit may correspond to a coding tree block (CTB), a coding block (CB), a prediction block (PB) or a transform block (TB) for a chroma component. Furthermore, the disclosure is not limited thereto, and a processing unit may be construed as a meaning including a unit for a luma component and a unit for a chroma component.
Furthermore, a processing unit is not essentially limited to a square block and may be constructed in a polygon form having three or more vertices.
Furthermore, hereinafter, in this specification, a pixel, a picture element, a coefficient (a transform coefficient or a transform coefficient after a first order transformation) etc. are generally called a sample. Furthermore, to use a sample may mean to use a pixel value, a picture element value, a transform coefficient or the like.
The video coding system may include a source device 10 and a receive device 20. The source device 10 may transmit encoded video/image information or data to the receive device 20 in a file or streaming format through a storage medium or a network.
The source device 10 may include a video source 11, an encoding apparatus 12, and an transmitter 13. The receive device 20 may include a receiver 21, a decoding apparatus 22 and a renderer 23. The source device may be referred to as a video/image encoding apparatus and the receive device may be referred to as a video/image decoding apparatus. The transmitter 13 may be included in the encoding apparatus 12. The receiver 21 may be included in the decoding apparatus 22. The renderer may include a display and the display may be configured as a separate device or an external component.
The video source may acquire video/image data through a capture, synthesis, or generation process of video/image. The video source may include a video/image capturing device and/or a video/image generating device. The video/image capturing device may include, for example, one or more cameras, a video/image archive including previously captured video/images, and the like. The video/image generating device may include, for example, a computer, a tablet, and a smartphone, and may electronically generate video/image data. For example, virtual video/image data may be generated through a computer or the like, and in this case, a video/image capturing process may be replaced by a process of generating related data.
The encoding apparatus 12 may encode an input video/image. The encoding apparatus 12 may perform a series of procedures such as prediction, transforma, and quantization for compression and coding efficiency. The encoded data (encoded video/video information) may be output in a form of a bitstream.
The transmitter 13 may transmit the encoded video/video information or data output in the form of a bitstream to the receiver of the receive device through a digital storage medium or a network in a file or streaming format. The digital storage media may include various storage media such as universal serial bus USB, secure digital SD, compact disk CD, digital video disk DVD, bluray, hard disk drive HDD, and solid state drive SSD. The transmitter 13 may include an element for generating a media file through a predetermined file format, and may include an element for transmission through a broadcast/communication network. The receiver 21 may extract the bitstream and transmit it to the decoding apparatus 22.
The decoding apparatus 22 may decode video/image data by performing a series of procedures such as dequantization, inverse transform, and prediction corresponding to the operations of the encoding apparatus 12.
The renderer 23 may render the decoded video/image. The rendered video/image may be displayed through the display.
Referring to
The image divider 110 may divide an input image (or picture or frame), input to the encoding apparatus 100, into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or the largest coding unit (LCU) based on a quadtree binary-tree (QTBT) structure. For example, one coding unit may be split into a plurality of coding units of a deeper depth based on a quadtree structure and/or a binary-tree structure. In this case, for example, the quadtree structure may be first applied, and the binary-tree structure may be then applied. Alternatively the binary-tree structure may be first applied. A coding procedure according to the disclosure may be performed based on the final coding unit that is no longer split. In this case, the largest coding unit may be directly used as the final coding unit based on coding efficiency according to an image characteristic or a coding unit may be recursively split into coding units of a deeper depth, if necessary. Accordingly, a coding unit having an optimal size may be used as the final coding unit. In this case, the coding procedure may include a procedure, such as a prediction, transform or reconstruction to be described later. For another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, each of the prediction unit and the transform unit may be divided or partitioned from each final coding unit. The prediction unit may be a unit for sample prediction, and the transform unit may be a unit from which a transform coefficient is derived and/or a unit in which a residual signal is derived from a transform coefficient.
A unit may be interchangeably used with a block or an area according to circumstances. In a common case, an M×N block may indicate a set of samples configured with M columns and N rows or a set of transform coefficients. In general, a sample may indicate a pixel or a value of a pixel, and may indicate only a pixel/pixel value of a luma component or only a pixel/pixel value of a chroma component. In a sample, one picture (or image) may be used as a term corresponding to a pixel or pel.
The encoding apparatus 100 may generate a residual signal (residual block or residual sample array) by subtracting a prediction signal (predicted block or prediction sample array), output by the inter predictor 180 or the intra predictor 185, from an input image signal (original block or original sample array). The generated residual signal is transmitted to the transformer 120. In this case, as illustrated, a unit in which the prediction signal (prediction block or prediction sample array) is subtracted from the input image signal (original block or original sample array) within the encoding apparatus 100 may be called the subtractor 115. The predictor may perform prediction on a processing target block (hereinafter referred to as a current block), and may generate a predicted block including prediction samples for the current block. The predictor may determine whether an intra prediction is applied or inter prediction is applied in a current block or a CU unit. The predictor may generate various pieces of information on a prediction, such as prediction mode information as will be described later in the description of each prediction mode, and may transmit the information to the entropy encoder 190. The information on prediction may be encoded in the entropy encoder 190 and may be output in a bitstream form.
The intra predictor 185 may predict a current block with reference to samples within a current picture. The referred samples may be located to neighbor the current block or may be spaced from the current block depending on a prediction mode. In an intra prediction, prediction modes may include a plurality of non-angular modes and a plurality of angular modes. The non-angular mode may include a DC mode and a planar mode, for example. The angular mode may include 33 angular prediction modes or 65 angular prediction modes, for example, depending on a fine degree of a prediction direction. In this case, angular prediction modes that are more or less than the 33 angular prediction modes or 65 angular prediction modes may be used depending on a configuration, for example. The intra predictor 185 may determine a prediction mode applied to a current block using the prediction mode applied to a neighboring block.
The inter predictor 180 may derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in an inter prediction mode, motion information may be predicted as a block, a sub-block or a sample unit based on the correlation of motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction) information. In the case of inter prediction, a neighboring block may include a spatial neighboring block within a current picture and a temporal neighboring block within a reference picture. A reference picture including a reference block and a reference picture including a temporal neighboring block may be the same or different. The temporal neighboring block may be referred to as a name called a co-located reference block or a co-located CU (colCU). A reference picture including a temporal neighboring block may be referred to as a co-located picture (colPic). For example, the inter predictor 180 may construct a motion information candidate list based on neighboring blocks, and may generate information indicating that which candidate is used to derive a motion vector and/or reference picture index of a current block. An inter prediction may be performed based on various prediction modes. For example, in the case of a skip mode and a merge mode, the inter predictor 180 may use motion information of a neighboring block as motion information of a current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of a motion information prediction (MVP) mode, a motion vector of a neighboring block may be used as a motion vector predictor. A motion vector of a current block may be indicated by signaling a motion vector difference.
A prediction signal generated through the inter predictor 180 or the intra predictor 185 may be used to generate a reconstructed signal or a residual signal.
The transformer 120 may generate transform coefficients by applying a transform scheme to a residual signal. For example, the transform scheme may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loève transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). In this case, the GBT means a transform obtained from a graph if relation information between pixels is represented as the graph. The CNT means a transform obtained based on a prediction signal generated u sing all of previously reconstructed pixels. Furthermore, a transform process may be applied to pixel blocks having the same size of a square form or may be applied to blocks having variable sizes not a square form.
The quantizer 130 may quantize transform coefficients and transmit them to the entropy encoder 190. The entropy encoder 190 may encode a quantized signal (information on quantized transform coefficients) and output it in a bitstream form. The information on quantized transform coefficients may be called residual information. The quantizer 130 may re-arrange the quantized transform coefficients of a block form in one-dimensional vector form based on a coefficient scan sequence, and may generate information on the quantized transform coefficients based on the quantized transform coefficients of the one-dimensional vector form. The entropy encoder 190 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoder 190 may encode information (e.g., values of syntax elements) necessary for video/image reconstruction in addition to the quantized transform coefficients together or separately. The encoded information (e.g., encoded video/image information) may be transmitted or stored in a network abstraction layer (NAL) unit unit in the form of a bitstream. The bitstream may be transmitted over a network or may be stored in a digital storage medium. In this case, the network may include a broadcast network and/or a communication network. The digital storage medium may include various storage media, such as a USB, an SD, a CD, a DVD, Blueray, an HDD, and an SSD. A transmitter (not illustrated) that transmits a signal output by the entropy encoder 190 and/or a storage (not illustrated) for storing the signal may be configured as an internal/external element of the encoding apparatus 100, or the transmitter may be an element of the entropy encoder 190.
Quantized transform coefficients output by the quantizer 130 may be used to generate a prediction signal. For example, a residual signal may be reconstructed by applying de-quantization and an inverse transform to the quantized transform coefficients through the dequantizer 140 and the inverse transformer 150 within a loop. The adder 155 may add the reconstructed residual signal to a prediction signal output by the inter predictor 180 or the intra predictor 185, so a reconstructed signal (reconstructed picture, reconstructed block or reconstructed sample array) may be generated. A predicted block may be used as a reconstructed block if there is no residual for a processing target block as in the case where a skip mode has been applied. The adder 155 may be called a reconstructor or a reconstruction block generator. The generated reconstructed signal may be used for the intra prediction of a next processing target block within a current picture, and may be used for the inter prediction of a next picture through filtering as will be described later.
The filter 160 can improve subjective/objective picture quality by applying filtering to a reconstructed signal. For example, the filter 160 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture. The modified reconstructed picture may be stored in the DPB 170. The various filtering methods may include deblocking filtering, a sample adaptive offset, an adaptive loop filter, and a bilateral filter, for example. The filter 160 may generate various pieces of information for filtering as will be described later in the description of each filtering method, and may transmit them to the entropy encoder 190. The filtering information may be encoded by the entropy encoder 190 and output in a bitstream form.
The modified reconstructed picture transmitted to the DPB 170 may be used as a reference picture in the inter predictor 180. The encoding apparatus can avoid a prediction mismatch in the encoding apparatus 100 and a decoding apparatus and improve encoding efficiency if inter prediction is applied.
The DPB 170 may store a modified reconstructed picture in order to use the modified reconstructed picture as a reference picture in the inter predictor 180.
Referring to
When a bitstream including video/image information is input, the decoding apparatus 200 may reconstruct an image in accordance with a process of processing video/image information in the encoding apparatus of
The decoding apparatus 200 may receive a signal, output by the encoding apparatus of
The dequantizer 220 may de-quantize quantized transform coefficients and output transform coefficients. The dequantizer 220 may re-arrange the quantized transform coefficients in a two-dimensional block form. In this case, the re-arrangement may be performed based on a coefficient scan sequence performed in the encoding apparatus. The dequantizer 220 may perform de-quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information), and may obtain transform coefficients.
The inverse transformer 230 may output a residual signal (residual block or residual sample array) by applying inverse-transform to transform coefficients.
The predictor may perform a prediction on a current block, and may generate a predicted block including prediction samples for the current block. The predictor may determine whether an intra prediction is applied or inter prediction is applied to the current block based on information on a prediction, which is output by the entropy decoder 210, and may determine a detailed intra/inter prediction mode.
The intra predictor 265 may predict a current block with reference to samples within a current picture. The referred samples may be located to neighbor a current block or may be spaced apart from a current block depending on a prediction mode. In an intra prediction, prediction modes may include a plurality of non-angular modes and a plurality of angular modes. The intra predictor 265 may determine a prediction mode applied to a current block using a prediction mode applied to a neighboring block.
The inter predictor 260 may derive a predicted block for a current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in an inter prediction mode, motion information may be predicted as a block, a sub-block or a sample unit based on the correlation of motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction) information. In the case of inter prediction, a neighboring block may include a spatial neighboring block within a current picture and a temporal neighboring block within a reference picture. For example, the inter predictor 260 may configure a motion information candidate list based on neighboring blocks, and may derive a motion vector and/or reference picture index of a current block based on received candidate selection information. An inter prediction may be performed based on various prediction modes. Information on the prediction may include information indicating a mode of inter prediction for a current block.
The adder 235 may generate a reconstructed signal (reconstructed picture, reconstructed block or reconstructed sample array) by adding an obtained residual signal to a prediction signal (predicted block or prediction sample array) output by the inter predictor 260 or the intra predictor 265. A predicted block may be used as a reconstructed block if there is no residual for a processing target block as in the case where a skip mode has been applied.
The adder 235 may be called a reconstructor or a reconstruction block generator. The generated reconstructed signal may be used for the intra prediction of a next processing target block within a current picture, and may be used for the inter prediction of a next picture through filtering as will be described later.
The filter 240 can improve subjective/objective picture quality by applying filtering to a reconstructed signal. For example, the filter 240 may generate a modified reconstructed picture by applying various filtering methods to a reconstructed picture, and may transmit the modified reconstructed picture to the DPB 250. The various filtering methods may include deblocking filtering, a sample adaptive offset SAO, an adaptive loop filter ALF, and a bilateral filter, for example.
A reconstructed picture transmitted (modified) in the DPB 250 may be used as a reference picture in the inter predictor 260.
In the disclosure, the embodiments described in the filter 160, inter predictor 180 and intra predictor 185 of the encoding apparatus 100 may be applied to the filter 240, inter predictor 260 and intra predictor 265 of the decoding apparatus 200, respectively, identically or in a correspondence manner.
The content streaming system to which the disclosure is applied may largely include an encoding server 410, a streaming server 420, a web server 430, a media storage 440, a user device 450, and a multimedia input device 460.
The encoding server 410 may compress the content input from multimedia input devices such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmit it to the streaming server 420. As another example, when the multimedia input devices 460 such as the smartphone, camera, and camcorder directly generate a bitstream, the encoding server 410 may be omitted.
The bitstream may be generated by an encoding method or a bitstream generation method to which the disclosure is applied, and the streaming server 420 may temporarily store the bitstream in the process of transmitting or receiving the bitstream.
The streaming server 420 transmits multimedia data to the user device 450 based on a user request through the web server 430, and the web server 430 serves as an intermediary to inform the user of what service is present. When a user requests a desired service through the web server 430, the web server 430 delivers it to the streaming server 420, and the streaming server 420 transmits multimedia data to the user. At this time, the content streaming system may include a separate control server, in which case the control server serves to control commands/responses between devices in the content streaming system.
The streaming server 420 may receive content from the media storage 440 and/or the encoding server 410. For example, the streaming server 420 may receive content in real time from the encoding server 410. In this case, in order to provide a smooth streaming service, the streaming server 420 may store the bitstream for a predetermined time.
For example, the user device 450 may include a mobile phone, a smart phone, a laptop computer, a terminal for digital broadcasting, a personal digital assistant PDA, a portable multimedia player PMP, a navigation terminal, a slate PC, a tablet PC, an ultrabook, a wearable device (for example, a smart watch, a smart glass, a head mounted display HMD, a digital TV, a desktop computer, and digital signage.
Each server in the content streaming system may operate as a distributed server, and in this case, data received from each server may be processed in a distributed manner.
The video signal processing apparatus 500 according to the embodiment of the disclosure may include a memory 520 for storing a video signal, and a processor 510 for processing the video signal while being combined with the memory 520.
The processor 510 according to an embodiment of the disclosure may be configured with at least one processing circuit for processing the video signal, and may process the video signal by executing instructions for encoding or decoding the video signal. That is, the processor 510 may encode the original video signal or decode the encoded video signal by executing the encoding or decoding methods described below.
In video coding, one block may be split based on QT. Also, one sub-block split by QT may be further split recursively using QT. A leaf block that is no longer QT split may be split by at least one of BT, TT, or AT. BT may have two types of splitting: horizontal BT (2N×N, 2N×N) and vertical BT (N×2N, N×2N). TT may have two types of splitting: horizontal TT (2N×1/2N, 2N×N, 2N×1/2N) and vertical TT (1/2N×2N, N×2N, 1/2N×2N). AT may have four types of splitting: horizontal-up AT (2N×1/2N, 2N×3/2N), horizontal-down AT (2N×3/2N, 2N×1/2N), vertical-left AT (1/2N×2N, 3/2N×2N), vertical-right AT (3/2N×2N, 1/2N×2N). Each BT, TT, AT may be further split recursively using BT, TT, AT.
Meanwhile, BT, TT and AT partitionings may be combined. For example, a sub-block split by BT may be split by TT or AT. In addition, the sub-block split by TT may be split by BT or AT. The sub-block split by AT may be split by BT or TT. For example, after horizontal BT splitting, each sub-block may be split into vertical BT, or after vertical BT splitting, each sub-block may be split into horizontal BT. The two types of splitting methods have different splitting order, but final split shapes are same.
Furthermore, when a block is split, the order in which the block are searched may be variously defined. In general, the search operation is performed from left to right and from top to bottom. Searching a block may mean an order of determining whether to further split each split sub-block, or an encoding order of respective sub-blocks when the block is no longer split, or a search order when sub-blocks refer to information of other neighboring blocks.
The encoding apparatus 100 performs inter prediction on a current block (S710). The encoding apparatus 100 may derive an inter prediction mode and motion information of a current block, and may generate the prediction samples of the current block. In this case, the inter prediction mode determination, motion information derivation and prediction sample generation procedure may be performed at the same time, and any one procedure may be performed prior to another procedure. For example, the inter predictor 180 of the encoding apparatus 100 may include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183. The prediction mode determination unit 181 may determine a prediction mode for a current block. The motion information derivation unit 182 may derive motion information of the current block. The prediction sample derivation unit 183 may derive prediction samples of the current block. For example, the inter predictor 180 of the encoding apparatus 100 may search a given area (search area) of reference pictures for a block similar to a current block through motion estimation, and may derive a reference block having a minimum difference or a difference of a given reference or less with respect to the current block. The inter predictor 180 may derive a reference picture index indicating a reference picture in which a reference block is located based on the reference block, and may derive a motion vector based on a location difference between the reference block and the current block. The encoding apparatus 100 may determine a mode applied to the current block among various prediction modes. The encoding apparatus may compare RD costs for the various prediction modes, and may determine an optimal prediction mode for the current block.
For example, if a skip mode or merge mode is applied to the current block, the encoding apparatus 100 may configure a merge candidate list to be described later, and may derive a reference block having a minimum difference or a difference of a given reference or less with respect to the current block among reference blocks indicated by merge candidates included in a merge candidate list. In this case, a merge candidate associated with the derived reference block may be selected. Merge index information indicating the selected merge candidate may be generated and signaled to the decoding apparatus 200. Motion information of the current block may be derived using motion information of the selected merge candidate.
For another example, if an (A)MVP mode is applied to the current block, the encoding apparatus may configure an (A)MVP candidate list to be described later, and may use a motion vector of a motion vector predictor (mvp) candidate, selected among mvp candidates included in the (A)MVP candidate list, as the mvp of the current block. In this case, for example, a motion vector indicating the reference block derived by the motion estimation may be used as the motion vector of the current block. An mvp candidate including a motion vector having the smallest difference with respect to the motion vector of the current block, among the mvp candidates, may become the selected mvp candidate. A motion vector difference (MVD), that is, a difference obtained by subtracting the mvp from the motion vector of the current block, may be derived. In this case, information on the MVD may be signaled to the decoding apparatus 200. Furthermore, if an (A)MVP mode is applied, a value of the reference picture index may be configured as reference picture index information and may be separately signaled to the decoding apparatus.
The encoding apparatus 100 may derive residual samples based on the prediction samples (S720). The encoding apparatus 100 may derive the residual samples through a comparison between the original samples of the current block and the prediction samples.
The encoding apparatus 100 encodes image information including prediction information and residual information (S730). The encoding apparatus may output the encoded image information in a bitstream form. The prediction information may include information on prediction mode information (e.g., skip flag, merge flag or mode index) and motion information as information related to the prediction procedure. The information related to motion information may include candidate selection information (e.g., merge index, mvp flag or mvp index), that is, information for deriving a motion vector. Furthermore, the information related to motion information may include information on the MVD and/or reference picture index information. Furthermore, the information related to motion information may include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. The residual information is information on the residual samples. The residual information may include information on quantized transform coefficients for the residual samples.
The output bitstream may be stored in a (digital) storage medium and transmitted to the decoding apparatus or may be transmitted to the decoding apparatus over a network.
Meanwhile, as described above, the encoding apparatus may generate a reconstructed picture (including reconstructed samples and reconstructed block) based on the reference samples and the residual samples. This is for deriving, in the encoding apparatus 100, the same prediction results as those performed in the decoding apparatus 200. Accordingly, coding efficiency can be improved. Accordingly, the encoding apparatus 100 may store the reconstructed picture (or reconstructed samples and reconstructed block) in the memory, and may use the reconstructed picture as a reference picture for inter prediction. As described above, an in-loop filtering procedure may be further applied to the reconstructed picture.
The decoding apparatus 200 may perform an operation corresponding to an operation performed in the encoding apparatus 100. The decoding apparatus 200 may perform prediction on a current block based on received prediction information, and may derive prediction samples.
Specifically, the decoding apparatus 200 may determine a prediction mode for the current block based on received prediction information (S910). The decoding apparatus 200 may determine which inter prediction mode is applied to the current block based on prediction mode information within the prediction information.
For example, the decoding apparatus 200 may determine whether the merge mode or (A)MVP mode is applied to the current block based on the merge flag. Alternatively, the decoding apparatus 200 may select one of various inter prediction mode candidates based on the mode index. The inter prediction mode candidates may include a skip mode, a merge mode and/or an (A)MVP mode or may include various inter prediction modes to be described later.
The decoding apparatus 200 derives motion information of the current block based on the determined inter prediction mode (S920). For example, if a skip mode or merge mode is applied to the current block, the decoding apparatus 200 may configure a merge candidate list to be described later and select one of merge candidates included in the merge candidate list. The selection of the merge candidate may be performed based on the merge index. Motion information of the current block may be derived from the motion information of the selected merge candidate. The motion information of the selected merge candidate may be used the motion information of the current block.
For another example, if an (A)MVP mode is applied to the current block, the decoding apparatus 200 may configure an (A)MVP candidate list to be described later, and may use a motion vector of a motion vector predictor (mvp) candidate, selected among mvp candidates included in the (A)MVP candidate list, as the mvp of the current block. The selection may be performed based on the selection information (mvp flag or mvp index). In this case, the decoding apparatus 200 may derive the MVD of the current block based on information on the MVD. The decoding apparatus may derive the motion vector of the current block based on the mvp of the current block and the MVD. Furthermore, the decoding apparatus may derive the reference picture index of the current block based on the reference picture index information. A picture indicated by the reference picture index within a reference picture list regarding the current block may be derived as a reference picture referred for the inter prediction of the current block.
Meanwhile, as will be described later, motion information of the current block may be derived without a candidate list configuration. In this case, motion information of the current block may be derived according to a procedure disclosed in a prediction mode to be described later. In this case, a candidate list configuration, such as that described above, may be omitted.
The decoding apparatus 200 may generate prediction samples for the current block based on the motion information of the current block (S930). In this case, the decoding apparatus 200 may derive a reference picture based on the reference picture index of the current block, and may derive the prediction samples of the current block indicated on the reference picture by the motion vector of the current block. In this case, as will be described later, a prediction sample filtering procedure may be further performed on some of or all the prediction samples of the current block according to circumstances.
For example, the inter predictor 260 of the decoding apparatus 200 may include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263. The decoding apparatus 200 may determine a prediction mode of the current block based on prediction mode information received from the prediction mode determination unit 261, may derive motion information (motion vector and/or the reference picture index) of the current block based on information related to motion information received from the motion information derivation unit 262. The prediction sample derivation unit 263 may derive the prediction samples of the current block.
The decoding apparatus 200 generates residual samples for the current block based on the received residual information (S940). The decoding apparatus 200 may generate reconstructed samples for the current block based on the prediction samples and the residual samples, and may generate a reconstructed picture based on the reconstructed samples (S950). Thereafter, as described above, an in-loop filtering procedure may be further applied to the reconstructed picture.
As described above, the inter prediction procedure may include an inter prediction mode determination step, a motion information derivation step according to a determined prediction mode, and a prediction execution (prediction sample generation) step based on derived motion information.
Determination of Inter Prediction Mode
Various inter prediction modes may be used for the prediction of a current block within a picture. For example, various modes, such as a merge mode, a skip mode, an MVP mode, and an affine mode, may be used. A decoder side motion vector refinement (DMVR) mode, an adaptive motion vector resolution (AMVR) mode, etc. may be further used as additional modes. The affine mode may be referred to as an affine motion prediction mode. The MVP mode may be referred to as an advanced motion vector prediction (AMVP) mode.
Prediction mode information indicating an inter prediction mode of a current block may be signaled from an encoding apparatus to a decoding apparatus. The prediction mode information may be included in a bitstream and received by the decoding apparatus. The prediction mode information may include index information indicating one of multiple candidate modes. Alternatively, an inter prediction mode may be indicated through the hierarchical signaling of flag information. In this case, the prediction mode information may include one or more flags. For example, a flag may be further signaled in order to indicate whether a skip mode is applied by signaling a skip flag, to indicate whether a merge mode is applied by signaling a merge flag if a skip mode is not applied, and to indicate that an MVP mode is applied if a merge mode is not applied or for an additional identification. The affine mode may be signaled as an independent mode or may be signaled as a mode dependent on a merge mode or MVP mode. For example, the affine mode may be configured as one of a merge candidate list or MVP candidate list, as will be described later.
Derivation of Motion Information According to Inter Prediction Mode
The encoding apparatus 100 or the decoding apparatus 200 may perform inter prediction using motion information of a current block. The encoding apparatus 100 may derive optimal motion information for a current block according to a motion estimation procedure. For example, the encoding apparatus 100 may search a reference block having a similar correlation using the original block within the original picture for a current block in a fraction pixel unit within a determined search range within a reference picture. Accordingly, the encoding apparatus may derive motion information. The similarity of a block may be derived based on a difference between phase-based sample values. For example, the similarity of a block may be calculated based on a SAD (Sum of Absolute Difference) between a current block (or the template of the current block) and a reference block (or the template of the reference block). In this case, motion information may be derived based on a reference block having the smallest SAD within a search area. The derived motion information may be signaled to the decoding apparatus using several methods based on an inter prediction mode.
Merge Mode and Skip Mode
If a merge mode is applied, motion information of a current prediction block is not directly transmitted, and motion information of the current prediction block is derived using motion information of a neighboring prediction block. Accordingly, the encoding apparatus 100 may indicate the motion information of the current prediction block by transmitting flag information to notify that a merge mode has been used and a merge index to notify which neighboring prediction block has been used.
The encoding apparatus 100 should search a merge candidate block used to derive motion information of a current prediction block in order to perform a merge mode. For example, a maximum of up to 5 merge candidate blocks may be used, but the disclosure is not limited thereto. Furthermore, a maximum number of merge candidate blocks may be transmitted in a slice header, and the disclosure is not limited thereto. After searching merge candidate blocks, the encoding apparatus 100 may generate a merge candidate list, and may select a merge candidate block having the smallest cost, among the merge candidate blocks, as the final merge candidate block.
An embodiment of the disclosure provides various embodiments of merge candidate blocks constructing a merge candidate list.
The merge candidate list may use 5 merge candidate blocks, for example. For example, 4 spatial merge candidates and 1 temporal merge candidate may be used.
Referring to
A coding apparatus (the encoding apparatus 100 or the decoding apparatus 200) searches spatial neighboring blocks of a current block and inserts derived spatial merge candidates into a merge candidate list (S1210). For example, the spatial neighboring blocks may include the bottom left corner neighboring block, left neighboring block, top right corner neighboring block, top neighboring block, and top left corner neighboring block of the current block. In this case, this is an example, and additional neighboring blocks, such as a right neighboring block, a bottom neighboring block, and a bottom right neighboring block, in addition to the spatial neighboring blocks may be further used as the spatial neighboring blocks. The coding apparatus may detect available blocks by searching the spatial neighboring blocks based on priority, and may derive motion information of the detected blocks as the spatial merge candidates. For example, the encoding apparatus 100 or the decoding apparatus 200 may search the 5 blocks illustrated in
The coding apparatus searches a temporal neighboring block of the current block and inserts a derived temporal merge candidate into the merge candidate list (S1220). The temporal neighboring block may be located on a reference picture, that is, a picture different from a current picture in which the current block is located. A reference picture in which the temporal neighboring block is located may be called a co-located picture or a col-picture. The temporal neighboring block may be searched in the sequence of the bottom right corner neighboring block and bottom right center block of a co-located block for the current block on the col-picture. Meanwhile, if motion data compression is applied, specific motion information may be stored in the col-picture as representative motion information for each given storage unit. In this case, it is not necessary to store motion information for all blocks within the given storage unit, and thus a motion data compression effect can be obtained. In this case, the given storage unit may be predetermined as a 16×16 sample unit or an 8×8 sample unit, for example, or size information for the given storage unit may be signaled from the encoding apparatus 100 to the decoding apparatus 200. If the motion data compression is applied, motion information of the temporal neighboring block may be substituted with representative motion information of the given storage unit in which the temporal neighboring block is located. That is, in this case, in an implementation aspect, after an arithmetic right shift is performed by a given value based on the coordinates (top left sample position) of the temporal neighboring block not a prediction block in which the coordinates of the temporal neighboring block are located, the temporal merge candidate may be derived based on motion information of a prediction block that covers the arithmetic left-shifted location. For example, if the given storage unit is a 2n×2n sample unit, assuming that the coordinates of the temporal neighboring block are (xTnb, yTnb), motion information of a prediction block located in ((xTnb>>n)<<n), (yTnb>>n)<<n)), that is, a modified location, may be used for the temporal merge candidate. Specifically, for example, if the given storage unit is a 16×16 sample unit, assuming that the coordinates of the temporal neighboring block are (xTnb, yTnb), motion information of a prediction block located in ((xTnb>>4)<<4), (yTnb>>4)<<4)), that is, a modified location, may be used for the temporal merge candidate. Alternatively, for example, if the given storage unit is an 8×8 sample unit, assuming that the coordinates of the temporal neighboring block are (xTnb, yTnb), motion information of a prediction block located in ((xTnb>>3)<<3), (yTnb>>3)<<3)), that is, a modified location, may be used for the temporal merge candidate.
The coding apparatus may check whether the current number of merge candidates is smaller than a maximum number of merge candidates (S1230). The maximum number of merge candidates may be pre-defined or may be signaled from the encoding apparatus 100 to the decoding apparatus 200. For example, the encoding apparatus 100 may generate information on the maximum number of merge candidates, may encode the information, and may transmit the information to the decoding apparatus 200 in a bitstream form. If the maximum number of merge candidates is filled, a candidate addition process may not be performed.
If, as a result of the check, the current number of merge candidates is smaller than the maximum number of merge candidates, the coding apparatus inserts an added merge candidate into the merge candidate list (S1240). The added merge candidate may include an ATMVP (Adaptive Temporal Motion Vector Prediction), a combined bi-predictive merge candidate (if the slice type of a current slice is a B type) and/or a zero vector merge candidate, for example.
If a motion vector prediction (MVP) mode is applied, a motion vector predictor (mvp) candidate list may be generated based on a motion vector of a reconstructed spatial neighboring block (e.g., the neighboring block described in
Referring to
After constructing the spatial candidate block list, the coding apparatus compares the number of spatial candidates included in the prediction candidate list with a preset reference number (eg, 2) (S1320). If the number of the spatial candidates included in the prediction candidate list is greater than or equal to the reference number (eg, 2), the coding apparatus may end the construction of the prediction candidate list.
But if the number of spatial candidate lists included in the prediction candidate list is less than the reference number (eg, 2), the coding apparatus searches for a temporal candidate block and inserts it into the prediction candidate list (S1330), and when the temporal candidate block is unavailable, adds a zero motion vector to the prediction candidate list (S1340).
Generation of Prediction Sample
A predicted block for a current block may be derived based on the motion information derived according to a prediction mode. The predicted block may include prediction samples (prediction sample array) of the current block. When the motion vector of the current block indicates a fractional sample unit, an interpolation procedure may be performed, and through this prediction samples of the current block may be derived based on the reference samples in a fractional sample unit in a reference picture. When affine inter prediction is applied to the current block, prediction samples may be generated based on a motion vector in a sample/subblock unit. When bi-direction prediction is applied, final prediction samples may be derived through weighted (according to the phase) sums of prediction samples derived based on first direction prediction (eg, L0 prediction) and prediction samples derived based on second direction prediction. Reconstruction samples and reconstruction pictures may be generated based on the derived prediction samples, and as described above, a procedure such as in-loop filtering may be performed afterwards.
Affine Motion Prediction
In a conventional image compression technology (e.g., high efficiency video coding (HEVC)), one motion vector is used to represent a motion of an encoding block. Although an optimal motion of a block unit may be represented using a method using one motion vector for each block, it may not be an actual optimal motion of each picture element. Accordingly, if an optimal motion vector is determined in a picture element unit, encoding efficiency may be increased. Accordingly, an embodiment of the disclosure describes a motion prediction method of encoding or decoding a video signal using a multi-motion model. In particular, a motion vector may be represented in each picture element unit of a block or a sub-block unit using the motion vector at 2 to 4 control points. A prediction scheme using the motion vector of a plurality of control points may be referred to as affine motion prediction or affine prediction.
An affine motion model according to an embodiment of the disclosure may represent 4 motion models, such as those illustrated in
As shown in
In Equation 1, v_0(v_0={v_0x, v_0y}) indicates a motion vector CPMV0 at the first control point of the top left location of a current block 1500. v_1(v_1={v_1x, v_1 y}) indicates a motion vector CPMV1 at the second control point of the top right location of the current block 1500. Furthermore, w indicates the width of the current block 1500. v(v={v_x, v_y}) indicates a motion vector at a {x,y} location. A motion vector of a sub-block (or picture element) unit may be derived using Equation 1. In one embodiment, motion vector accuracy may be rounded as 1/16 accuracy.
Referring to
If an affine motion vector field is determined in a picture element unit, a motion vector may be obtained based on each picture element value. In the case of a block unit, the motion vector of a corresponding block may be obtained based on a center picture element value of the block. In this document, it is assumed that an affine motion vector field (MVF) is determined in a 4*4 block unit as in
Affine motion prediction may include an affine merge mode (or AF_MERGE) and an affine inter mode (or AF_INTER). The AF_INTER mode may include an AF_4_INTER mode using a 4 parameter-based motion model and an AF_6_INTER mode using a 6 parameter-based motion model.
Affine Merge Mode
AF_MERGE determines a control point motion vector (CPMV) based on the affine motion model of a neighboring block coded as affine motion prediction. A neighboring block affine-coded in a search sequence may be used for AF_MERGE. When one or more neighboring blocks are coded as affine motion prediction, a current block may be coded as AF_MERGE.
That is, if an affine merge mode is applied, the CPMVs of a current block may be derived using the CPMVs of a neighboring block. In this case, the CPMVs of the neighboring block may be used as the CPMVs of the current block without any change, and the CPMVs of the neighboring block may be modified based on the size of the neighboring block and the size of the current block and may be used as the CPMVs of the current block.
In the affine merge (AF_MERGE) mode, the encoder may perform encoding as in the following process.
Step-1: Scan the neighboring blocks A to E 1710, 1720, 1730, 1740, and 1750 of a current encoding block 1700 in sequence of the alphabet, and determine a block, first encoded according to an affine prediction mode based on the scanning sequence, as the candidate block of affine merge (AF_MERGE)
Step-2: Determine an affine motion model using the control point motion vector (CPMV) of the determined candidate block
Step-3: Determine the control point motion vector (CPMV) of the current block 1700 based on the affine motion model of the candidate block and determine the MVF of the current block 1700
For example, as in
Referring to
In an embodiment, if the number of candidates (hereinafter referred to as affine candidates) encoded according to an affine mode (or an affine prediction) available in the neighboring blocks (i.e., A, B, C, D, E) is 0, the affine merge mode of a current block may be skipped. If the number of available affine candidates is one (e.g., A), the motion model of the corresponding candidate may be used to derive the control point motion vectors (CPMV_0 and CPMV_1) of a current block. In this case, an index indicating the corresponding candidate may not be required (or coded). If the number of available affine candidates is two or more, two candidates on the scanning sequence may be configured as a candidate list for AF_MERGE. In this case, candidate selection information, such as an index indicating a candidate selected within the candidate list, may be signaled. The selection information may be a flag or index information, and may be referred to as AF_MERGE_flag or AF_merge_idx.
In an embodiment of the disclosure, a motion compensation for a current block may be performed based on the size of a sub-block. In this case, the sub-block size of an affine block (i.e., current block) is derived. If each of the width and height of a sub-block are larger than 4 luma samples, a motion vector for each sub-block may be derived, and a DCT-IF-based motion compensation ( 1/16 pel for luma and 1/32 for chroma) may be performed on the sub-block. If not, an enhanced bi-linear interpolation filter based motion compensation may be performed on all affine blocks.
In an embodiment of the disclosure, if a merge/skip flag is true and both the width and height of a CU is greater than or equal to 8, in a CU level, an affine flag is signaled through a bitstream indicating whether an affine merge mode is used. When a CU is coded as AF_MERGE, a merge candidate index having a maximum value of ‘5’ is signaled to designate that a motion information candidate is used for the CU in an affine merge candidate list.
Referring to
1) Insertion of Model-Based Affine Candidate
A model-based affine candidate means that a candidate is derived from a valid neighboring reconstructed block coded according to an affine mode. As illustrated in
If the neighboring bottom left block A is coded in a 6-parameter affine mode, the motion vectors (v_4, v_5, v_6) of the top left corner, top right corner, and bottom left corner of a CU including the block A are obtained. The motion vectors (v_0, v_1, v_2) of the top left corner of the current block are calculated based on the motion vectors (v_4, v_5, and v_6) according to the 6-parameter affine model.
If the neighboring bottom left block A is coded in a 4-parameter affine mode, the a motion vectors (v_4, v_5) of the top left corner and top right corner of the CU including the block A are obtained. The motion vectors (v_0, v_1) of the top left corner on the current block are calculated based on the motion vectors (v_4, v_5) according to the 4-parameter affine model.
2) Insertion of Control Point-Based Affine Candidates
Referring to
Motion information of control points are first derived from designated spatial neighboring blocks and temporal neighboring block illustrated in
The coordinates of CP_1, CP_2, CP_3 and CP_4 are (0, 0), (W, 0), (H, 0) and (W, H), respectively. In this case, W and H are the width and height of the current block
Motion information of each control point is obtained based on the following priority.
With respect to CP_1, checking priority is A→B→C, and A is used if A is available. If not and if B is available, B is used. If both A and B are not available, C is used. If all the three candidates are not available, motion information of CP_1 cannot be obtained.
With respect to CP_2, checking priority is E→D.
With respect to CP_3, checking priority is G→F.
With respect to CP_4, H is used.
Second, combinations of control points are used to configure a motion model.
Motion vectors of two control points are necessary to calculate transform parameters in a 4-parameter affine model. The two control points may be selected from one of the following 6 combinations {CP_1, CP_4}, {CP_2, CP_3}, {CP_1, CP_2}, {CP_2, CP_4}, {CP_1, CP_3}, and {CP_3, CP_4}. For example, to use the CP_1 and CP_2 control points in constructing the 4-parameter affine motion model is marked as “affine (CP_1, CP_2).”
Motion vectors of three control points are necessary to calculate transform parameters in a 6-parameter affine model. The three control points may be selected from one of the following 4 combinations {CP_1, CP_2, CP_4}, {CP_1, CP_2, CP_3}, {CP_2, CP_3, CP_4}, and {CP_1, CP_3, CP_4}. For example, to use the CP_1, CP_2 and CPv3 control points in constructing the 6-parameter affine motion model is marked as “affine (CP_1, CP_2, CP_3).”
Furthermore, in an embodiment of the disclosure, if an affine merge candidate is present in an affine merge mode, this may be always considered as a 6-parameter affine mode.
Affine Inter Mode
Referring to
Step-1: Determine a two-CPMVP pair candidate
Step-1.1: Determine a maximum of 12 CPMVP candidate combinations (refer to Equation 2)
{(v0,v1,v2)|v0={vA,vB,vC},v1={vD,vE},v2{=vF,vG}} [Equation 2]
In Equation 2, v_0 indicates a motion vector CPMV0 at the top left control point 2210 of a current block 2200. v_1 indicates a motion vector CPMV1 at the top right control point 2211 of the current block 2200. v_2 indicates a motion vector CPMV2 at a control point 2212 on the bottom left side of the current block 2200. v_A indicates the motion vector of a neighboring block A 2220 neighboring the top left of the top left control point 2210 of the current block 2200. v_B indicates the motion vector of a neighboring block B 2222 neighboring the top of the top left control point 2210 of the current block 2200. v_C indicates the motion vector of a neighboring block C 2224 neighboring the left of the top left control point 2210 of the current block 2200. v_D is the motion vector of a neighboring block D 2226 neighboring the top of the top right control point 2211 of the current block 2200. v_E indicates the motion vector of a neighboring block E 2228 neighboring the top right of the top right control point 2211 of the current block 2200. v_F indicates the motion vector of a neighboring block F 2230 neighboring the left of the bottom left control point 2212 of the current block 2200. v_G indicates the motion vector of a neighboring block G 2232 neighboring the left of the bottom left control point 2212 of the current block 2200.
Step-1.2: Sort CPMVP candidate combinations based on a value having a small difference value (DV) and use top two candidates (refer to Equation 3 below)
DV=|(v1X−v0X)*h−(v2y−v0y)*w|+|(v1y−v0y)*h+(v2x−v0x)*w| [Equation 3]
v_0x indicates the x-axis element of the motion vector (V0 or CPMV0) at the top left control point 2210 of the current block 2200. v_1x indicates the x-axis element of the motion vector (V1 or CPMV1) at the top right control point 2211 of the current block 2200. v_2x indicates the x-axis element of the motion vector (V_2 or CPMV_2) at the bottom left control point 2212 of the current block 2200. v_0y indicates the y-axis element of the motion vector (V_0 or CPMV_0) at the top left control point 2210 of the current block 2200. v_1y indicates the y-axis element of the motion vector (V_1 or CPMV_1) at the top right control point 2211 of the current block 2200. v_2y indicates the y-axis element of the motion vector (V_2 or CPMV_2) at the bottom left control point 2212 of the current block 2200. w indicates the width of the current block 2200. h indicates the height of the current block 2200.
Step-2: Use an AMVP candidate list when a control point motion vector predictor (CPMVP) pair candidate is smaller than 2
Step-3: Determine the control point motion vector predictor (CPMVP) of each of two candidates and optimally selects a candidate having a smaller value by comparing RD costs and a CPMV
Step-4: Transmit an index corresponding to the optimal candidate and a control point motion vector difference (CPMVD)
In an embodiment of the disclosure, a process of constructing a CPMVP candidate in AF_INTER is provided. Identically with the AMVP, the number of candidates is 2, and an index indicating the location of a candidate list is signaled.
A process of constructing a CPMVP candidate list is as follows.
1) Whether neighboring blocks are coded as affine motion prediction is checked by scanning the neighboring blocks. If the scanned blocks are coded as affine prediction, the motion vector pair of a current block is derived from the affine motion model of the scanned neighboring block until the number of candidates becomes 2.
2) If the number of candidates is smaller than 2, a candidate configuration process is performed. Furthermore, in an embodiment of the disclosure, a 4-parameter (2-control point) affine inter mode is used to predict the motion model of zoom-in/out and rotate and content. As illustrated in
The motion vector field (MVF) of a block is described by Equation 1 described above.
In a conventional technology, an advanced motion vector prediction (AMVP) mode is necessary to scan a motion vector prediction (MVP) index and motion vector differences (MVDs). When the AMVP mode is applied to the disclosure, an affine flag (affine_flag) is signaled to indicate whether affine prediction is used. If the affine prediction is applied, the syntaxes of inter_dir, ref_idx, mvp_index, and two MVDs (mvd_x and mvd_y) are signaled. An affine MVP pair candidate list including two affine MVP pairs is generated. The signaled mvp_index is used to select one of the two affine MVP pairs. The affine MVP pair is generated by two types of affine MVP candidates. One is a spatial inherited affine candidate, and the other is a corner-derived affine candidate. If neighboring CUs are coded in an affine mode, spatial inherited affine candidates may be generated. The affine motion model of a neighboring affine-coded block is used to generate the motion vectors of a 2-control point MVP pair. The MVs of the 2-control point MVP pair of a spatial inherited affine candidate are derived using the following equations.
V0x=VB0x+(VB2_x−VB0x)*(posCurCU_Y−posRefCU_Y)/RefCU_height+(VB1x−VB0x)*(posCurCU_X−posRefCU_X)/RefCU_width [Equation 4]
V0y=VB0y+(VB2_y−VB0y)*(posCurCU_Y−posRefCU_Y)/RefCU_height+(VB1y−VB0y)*(posCurCU_X−posRefCU_X)/RefCU_width [Equation 5]
If V_B0, V_B1, and V_B2 may be substituted with the top left MV, top right MV, and bottom left MV of a given reference/neighboring CU, (posCurCU_X, posCurCU_Y) are the location of the top left sample of a current CU for the top left sample of a frame. (posRefCU_X, posRefCU_Y) is the location of the top left sample of a reference/neighboring CU for the top left sample of a frame.
V1x=VB0x+(VB1x−VB0x)*CU_width/RefCU_width [Equation 6]
V1y=VB0y+(VB1y−VB0y)*CU_width/RefCU_width [Equation 7]
Referring to
In an embodiment of the disclosure, two candidate sets, including two (three) candidates {mv_0, mv_1} ({mv_0, mv_1, mv_2) are used to predict two (three) control points of an affine motion model. Given motion vector differences (mvd_0, mvd_1, mvd_2) and control points are calculated using the following equations.
mv0=
mv1=
mv2=
In the affine candidate list, an affine motion is extended from spatial neighboring blocks (extrapolated affine candidates), and the affine candidate list is appended by a combination of motion vectors from the spatial neighboring blocks (virtual affine candidates). Candidate sets are set as follows:
1. A maximum of two different affine MV predictor sets are derived from an affine motion of neighboring blocks. Neighboring blocks A0, A1, B0, B1, and B2 are checked as illustrated in
2.
If mv_0 and mv_1 are given, mv_2 may be derived by Equation 9 below.
In Equation 9, a current block size is W×H.
If only mv_0 and mv_2 are given, mv_1 may be derived by Equation 10 below.
In an embodiment of the disclosure, affine inter prediction may be performed according to the following sequence.
Input: affine motion parameters, reference picture samples
Output: the prediction block of a CU
Processor
Derive the sub-block size of an affine block
Furthermore, in an embodiment of the disclosure, if a merge/skip flag is false and the width and height of a CU are larger than or equal to 8, an affine flag is signaled in order to indicate whether an affine inter mode will be used in a CU level. If a CU is coded in an affine inter mode, a model flag is signaled in order to indicate whether a 4-parameter or 6-parameter affine model is applied to the CU. If the model flag is true, an AF_6_INTER mode (6-parameter affine model) is applied and MVDs are parsed. If not, an AF_4_INTER mode (4-parameter affine model) is applied, and two MVDs are parsed.
In the AF_4_INTER mode, similarly to the affine merge mode, motion vector pairs extrapolated from neighboring blocks coded by the affine mode are generated and preferentially inserted into a candidate list.
Thereafter, if the size of the candidate list is smaller than 4, candidates having a motion vector pair {(v_0,v_1)|v0={v_A,v_B,v_c},v_1={v_D, v_E}} are generated using neighboring blocks. As illustrated in
If the number of candidate lists is smaller than 4, the list is padded by a motion vector pair by duplicating AMVP candidates.
In the AF_6_INTER mode, similar to the affine merge mode, motion vector triples (affine motion vector triples) extrapolated from neighboring blocks coded in the affine merge mode are generated and preferentially inserted into a candidate list.
Thereafter, when the size of the candidate list is smaller than 4, candidates including motion vector triples {(v_0, v_1, v_2)| v0={v_A, v_B, v_c}, v1={v_D, v_E}, v2={v_G, v_H}} are generated using neighboring blocks. As illustrated in
When the number of candidate lists is smaller than 4, a list may be padded by motion vector triples configured by duplicating respective AMVP candidates.
After the CPMV of a current CU is derived, the MVF of the current CU is generated according to Equation 11 for a 4-parameter affine model and is generated according to Equation 12 for a 6-parameter affine model, based on the number of affine parameters.
In this case, a sub-block size M×N is derived in Equation 13, and MvPre is motion vector portion accuracy ( 1/16).
After derived by Equation 12, M and N need to be down adjusted, if necessary, in order to make them the divisor of w and h. When M or N is smaller than 8, a WIF is applied. If not, an affine motion compensation based on a sub-block is applied.
Referring to
The SHVC up-sampling interpolation filters having the same filter length and normalization factor as HEVC motion compensation interpolation filters may be used as motion compensation interpolation filters for additional fractional pel positions. Chroma component motion vector accuracy is a 1/32 sample. The additional interpolation filters of 1/32 pel portion locations are derived using averages of the filters of two neighboring 1/16 pel portion locations.
The AF_MERGE mode may be selected on the encoder side using the same method of selecting a common merge mode. A candidate list is preferentially generated, and a minimum RD-cost is selected in the candidates for a comparison with the RD-costs of other inter modes. A result of the comparison is a determination of whether AF_MERGE is applied or not.
For the AF_4_INTER mode, the check of an RD cost is used to determine whether a motion vector pair candidate is selected as the control point motion vector prediction (CPMVP) of a current CU. After the CPMVP of a current affine CU is determined, affine motion estimation is applied, and a control point motion vector (CPMV) is obtained. Accordingly, a difference between the CPMV and the CPMVP is determined.
On the encoder side, the AF_6_INTER mode is identified only when the AF_MERGE or AF_4_INTER mode is determined as an optimal mode in a previous mode selection stage.
In an embodiment of the disclosure, the affine inter (affine AMVP) mode may be performed as follows:
1) AFFINE_MERGE_IMPROVE: instead of searching the first neighboring block in an affine mode, an improvement is to search a neighboring block having a maximum coding unit size as an affine merge candidate.
2) AFFINE_AMVL_IMPROVE: neighboring blocks in the affine mode is added to an affine AMVP candidate list similarly to a common AMVP procedure.
A detailed process of generating the affine AMVP candidate list is as follows.
First, whether a neighboring block at the bottom left uses an affine motion model and has the same reference index as a current reference index is identified. If the neighboring block is not present, a left neighboring block is identified using the same method. If the neighboring block is not present, whether a neighboring block at the bottom left uses an affine motion model and has a different reference index is identified. If the neighboring block is present, a scaled affine motion vector is added to a reference picture list. If the neighboring block is not present, a left neighboring block is identified using the same method.
Second, a top right neighboring block, a top neighboring block, and a top left neighboring block are identified using the same method.
After the processes, if two candidates are retrieved, the process of generating an affine AMVP candidate list is terminated. If two candidates are not retrieved, an original operation within JEM software is performed to generate affine AMVP candidate lists.
3) AFFINE_SIX_PARAM: in addition to the 4-parameter affine motion model, a 6-parameter affine motion model is added as an additional model.
The 6-parameter affine motion model is derived through Equation 14.
Three motion vectors at the top left location MV_0, the top right location MV_1, and the bottom left location MV_2 are necessary to determine a model because 6-parameters are present in the motion model. Three motion vectors may be determined using a method similar to the method of two motion vectors in the 4-parameter affine motion model. The affine model merge is always set as a 6-parameter affine motion model.
4) AFFINE_CLIP_REMOVE: motion vector constraints for all affine motion vectors are removed. Motion compensation processes are made to control the motion vector constraints themselves.
Affine Motion Model
As described above, various affine motion models may be used or considered in affine inter prediction. For example, an affine motion model may represent four motions as in
In an embodiment of the disclosure, adaptive four and six parameter motion models are used. In AF_INTER, a 6-parameter motion model is proposed in addition to the 4-parameter motion model present in JEM. The 6-parameter affine motion model is described like Equation 15.
x′=a*x+b*y+c
y′=d*x+e*y+f [Equation 15]
In this case, coefficients a, b, c, d e, and f are affine motion parameters. (x,y) and (x′,y′) are coordinates at pixel locations before and after the transform of an affine motion model. In video coding, in order to use the affine motion model, if CPMV0, CPMV1, and CPMV2 are MVs for CP0 (top left), CP1 (top right), and CP2 (bottom left), Equation 16 may be described as follows.
In this case, CPMV_0={v_0x, v_0y}, CPMV_1={v_1x, v_1y}, CPMV_2={v_2x, v_2y}, and w and h are the width and height of each coding block. Equation 16 is the motion vector field (MVF) of the block.
A flag is parsed in a CU level in order to indicate whether a 4-parameter or 6-parameter affine motion model is used when neighboring blocks are coded as affine prediction. If a neighboring block coded as affine prediction is not present, the flag is omitted and the 4-parameter model is used for affine prediction. In other words, the 6-parameter model is considered in the condition in which one or more neighboring blocks are coded in an affine motion model. In relation to the number of CPMVDs, each of two or three CPMVDs is signaled with respect to the 4-parameter and 6-parameter affine motion models.
Furthermore, in an embodiment of the disclosure, pattern-matched motion vector refinement may be used. In the pattern-matched motion vector derivation (PMMVD, hereinafter abbreviated as a PMVD in the JEM encoder description) of JEM, the decoder needs to evaluate some motion vectors (MVs) in order to determine a start MV candidate for CU-level search. In sub-CU-level search, some MV candidates are added in addition to an optimal CU-level MV. The decoder needs to evaluate such MV candidates in order to search for an optimal MV. This requires a great memory band. In the proposed pattern-matched motion vector refinement (PMVR), the concepts of template matching and bilateral matching in PMVD are adopted in JEM. When a skip mode or merge mode is selected to indicate whether the PMVR is available, one PMVR_flag is signaled. In order to meaningfully reduce the memory bandwidth requirement compared to the PMVD, an MV candidate list is generated. If the PMVR is applied, a start MV candidate index is explicitly signaled.
A candidate list is generated using the merge candidate list generation process, but sub-CU merge candidates, for example, affine candidates and ATMVP candidates are excluded. For bilateral matching, only a uni-prediction MV candidate is included. A bi-prediction MV candidate is divided into two unit-prediction MV candidates. Furthermore, similar MV candidates (having MV differences smaller than a predefined threshold) are also removed. For CU-level search, diamond search MV refinement is performed starting from a signaled MV candidate.
Sub-CU-level search is available only in the bilateral matching merge mode. A search window for the sub-CU-level search for all sub-CUs is the same as a search window for CU-level search. Accordingly, an additional bandwidth is not necessary in the sub-CU-level search.
In order to refine an MVP in a mode, template matching is also used. In the AMVP mode, two MVPs are generated using an HEVC MVP generation process, and one MVP index is signaled to select one of the two MVPs. The selected MVP is further refined using template matching in PMVR. If adaptive motion vector resolution (AMVR) is applied, the MVP is rounded with corresponding accuracy prior to the template matching refinement. Such a refinement process is named pattern-matched motion vector predictor refinement (PMVPR). In the remainder of this document, PMVR includes template matching PMVR, bi-direction matching PMVR, and PMVPR unless specially defined otherwise.
In order to reduce a memory bandwidth requirement, PMVR is unavailable for 4×4, 4×8, and 8×4 CUs. In order to reduce the amount of a required additional memory bandwidth, the search range of {template matching, bi-direction matching} for a CU area equal to 64 may be reduced to {±2, ±4}. The search range of {template matching, bi-direction matching} for a CU area larger than 64 may be reduced to {±6, ±8}. Compared to the worst case in HEVC, a required memory bandwidth was reduced from 45.9x in PMVD of JEM-7.0 to 3.1x in PMVR using all the methods described in the PMVR section of this document.
General History-Based Motion Vector Prediction (HMVP)
In general, an image compression technology uses exploiting for spatial and temporal redundancy as two major schemes. For example, both high efficiency video coding (HEVC) and VVC use two motion compression schemes based on inter coding. One is a merge motion, and the other is advanced motion vector prediction (AMVP). In order to improve the two prediction modes, various modifications are being discussed. The various modifications includes increasing the number of candidates to search for more spatially extended candidates and the check of temporal candidates at non-traditional locations. The two schemes include primarily constructing a list using available candidates, minimizing a rate distortion (RD) cost, and signaling a selected candidate in a bitstream.
In particular, in a recent image compression technology, HMVP in which motion information of a previously coded block is stored and the stored motion information is used for the motion prediction of a subsequently coded block is discussed. Such HMVP may be added to a merge list (or merge candidate list) or an AMVP list (or AMVP candidate list).
The decoder maintains a look-up table (LUT) operating in a first in first out (FIFO)) system (or method) for HMVP. In the disclosure, the LUT is not limited to its name, and may be referred to as a table, an HMVP table, an HMVP candidate table, a buffer, an HMVP buffer, an HMVP candidate buffer, an HMVP list or an HMVP candidate list. Specifically, when a non-affine prediction unit (PU) (or coding unit (CU)) is decoded, corresponding motion information is stored in an LUT. The decoder performs decoding on a next PU. In this case, the stored motion information may include motion vectors, reference index information and mode information in x (horizontal) and y (vertical) directions.
The decoder maintains an LUT in which motion information of a progressively decoded non-affine candidate is stored. The size of the LUT may be limited to predefined S candidates. In one embodiment, the LUT may be reset at the start of a slice, the start of a CTU row or the start of a CTU.
HMVP may be applied both the merge mode and the AMVP mode. A merge list may have B candidates, and an AMVP list may have two candidates. In a conventional image compression technology, a merge list is configured with the following candidates: i) a spatial candidate, ii) a temporal candidate, iii) a bi-direction prediction (Bi-Pred) candidate, iv) a zero motion candidate. A method of additionally considering advanced motion vector prediction (ATMVP) as a candidate is recently discussed. For example, an ATMVP candidate may be inserted into a merge list prior to a temporal candidate. Candidates of the merge list are added to the merge list until they reach a maximum merge list size. A duplicate candidate may not be added to the merge list. Two candidates may be inserted into an AMVP list. For example, the first candidate of the two candidates may be selected from available spatial candidates, and the second candidate may be selected from temporal candidates. If the list is not filled, a zero motion vector candidate may be added.
HMVP is applied based on an FIFO order in which candidates exit from an LUT identically with their input order.
In one embodiment, when HMVP is applied to a merge list configuration, an HMVP candidate may be inserted (or added) to the third location of a list as follows:
In one embodiment, when HMVP is applied to an AMVP list configuration, the HMVP may be inserted into the third location after a temporal candidate as follows:
Referring to
The decoder checks whether the current PU is a block coded in a non-affine mode (S2702). In order to facilitate the use of HMVP candidates, if the current PU is a block coded in the affine mode, the decoder does not store motion information of the current PU in a table.
If the current PU is a block coded in the non-affine mode, the decoder stores (or updates) motion information of the current PU in a table (S2703).
In an embodiment of the disclosure, an HMVP table may be updated using two methods, that is, i) unconstrained FIFO ii) constraint FIFO methods. In the former, redundant motion information may be present, but a pruning process is not applied. This contributes to reducing the complexity of an overall process. On the other hand, in the latter the pruning process is applied and no redundant motion information is present. This is described with reference to the following figure.
Referring to
If the table is not fully filled with a maximum number of pre-defined candidates at an index L−1 (i.e., end), new candidates are added without a removed candidate. In contrast, if the table has already been fully filled, that is, if a maximum number of the table is satisfied, a candidate located at the front end, that is, the oldest in the table, is removed and a new candidate is added.
Referring to
With respect to HMVP candidates, in many cases, the latest history MVs may overlap motion information of a spatial candidate (or spatial neighboring candidate). Accordingly, the present embodiment proposes a method of setting the addition order of candidates differently from an HMVP LUT index order when an HMVP candidate is added to an AMVP or merge list.
According to an embodiment of the disclosure, a candidate list can be efficiently configured by adaptively adjusting an HMVP candidate. Accordingly, the number of signaling bins used for binarization can be reduced, and coding efficiency can be improved. That is, an HMVP candidate added to a merge list or an AMVP list may be not limited by an index in the HMVP list. As an embodiment, the following Table 1 illustrates a method of changing the order of adding an HMVP candidate to the AMVP or merge list.
1]
Referring to Table 1, as described above, there is a good possibility that the most recently inserted HMVP candidate may have the same motion information as a spatial candidate. Accordingly, the addition order of an HMVP candidate may be pre-defined regardless of an HMVP index by considering the possibility.
Furthermore, in one embodiment, the encoder or decoder may add an HMVP candidate to a merge list or AMVP list from an HMVP candidate starting from an n-th candidate in the list. Table 2 below illustrates a changed order to add a candidate to an AMVP or merge list.
Referring to Table 2, an HMVP candidate may be added to a merge list or AMVP list from the second index.
In one embodiment, information on the addition order of an HMVP candidate within a table (LUT) may be signaled from the encoder to the decoder. For example, such order information may be transmitted through a high level syntax (HLS). The high level syntax may be a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit and/or another proper syntax data header, for example.
Table 3 illustrates a high level syntax structure to which a method proposed in the disclosure may be applied.
Referring to Table 3, set_HMVP_order_flag equal to 1 indicates that set_HMVP_order_flag is present in a slice header within non-IDR pictures in CVS. set_HMVP_order_flag equal to 0 indicates that set_HMVP_order_flag is not present in slice headers and adaptive HMVP is not used in VCS.
Table 4 illustrates a slice segment header syntax structure to which a method proposed in the disclosure may be applied.
Referring to Table 4, slice_HMVP_idx means an index for the sequence of used candidates. For example, slice_HMVP_idx equal to 0 may represent a basic HMVP sequence, such as 0, 1, 2, 3. Likewise, an index value of 1 may be used to represent an HMVP sequence of 3, 2, 1, 0.
In an embodiment of the disclosure, in addition to the HMVP LUT, a method of using a long term list for motion prediction is proposed. Accordingly, the number of maintained HMVP candidates can be increased. In an embodiment, 2-HMVP tables may be considered. In this case, one may be used to store common HMVP candidates, and the other may be used as a long term list in which candidates that need to be further maintained is stored.
The following illustrates methods of resetting and constructing a long term list (or long-term HMVP list).
Referring to
The use of the long-term LUT may be signaled by using a new syntax element. In an embodiment, the syntax element may be signaled through a high level syntax. For example, the syntax element may be present in a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit and/or another proper syntax data header.
In an embodiment of the disclosure, there is proposed a method in which flexibility for decoding is considered in adding an HMVP candidate to an HMVP LUT. The encoder/decoder may consider a decision criterion for one or more characteristics of a PU (or CU) when adding an HMVP candidate to the HMVP LUT.
In an embodiment, the encoder/decoder may consider the following contents in adding an HMVP candidate to a table. The encoder/decoder may add a candidate to a table by considering characteristics, such as a mode (e.g., merge mode, affine mode or AMVP mode) of a PU and/or the size of a block, separately or in combination. In one embodiment, in addition to the characteristics, other additional characteristics may be considered. For example, a merge type (e.g., spatial candidate or temporal candidate) in which an HMVP LUT update is considered, whether it is a sub-PU, etc. may be considered as a criterion for selecting a candidate. The selection criterion may be determined to reduce redundancy with a previous history (or previous HMVP). For example, if a PU is coded in a merge mode and a merge type is a spatial merge, the decoder may not update an HMVP LUT with motion information of the corresponding PU.
Referring to
The encoder/decoder evaluates whether to update an LUT with the motion information of the candidate based on a pre-defined decision criterion (S3102). As described above, the decision criterion may include characteristics related to one or more of a mode (e.g., merge mode, affine mode or AMVP mode) of the candidate, the block size of the candidate and/or the merge type of the candidate.
The encoder/decoder updates the LUT based on the decision criterion (S4303). That is, if the candidate satisfies the pre-defined decision criterion, the encoder/decoder may add the motion information of the candidate to the LUT.
In an embodiment of the disclosure, there are proposed constraints on a redundancy check for adding an HMVP candidate to a merge list (or AMVP list). The constraints on a redundancy check may be defined or implemented in various manners.
In one embodiment, the encoder/decoder may limit the number of pruning checks for a first specific number of candidates in a merge list. As an embodiment, the encoder/decoder may limit the number of pruning checks for candidates from the first candidate of a merge list to a specific candidate. For example, the encoder/decoder may perform a pruning process on candidates from the first candidate of a merge list to a specific candidate. And, an HMVP candidate, that is, a target of a pruning check may be limited to a pre-defined number.
Furthermore, in one embodiment, the encoder/decoder may limit a pruning check by performing the pruning check on a specific type of a merge candidate within a merge list. For example, the encoder/decoder may perform a pruning check on only a spatial candidate of a merge list in adding an HMVP candidate. Alternatively, for example, the encoder/decoder may perform a pruning check on only some of the spatial candidates of a merge list in adding an HMVP candidate. Some of the spatial candidates may be pre-defined. For example, the pre-defined spatial candidates may be at least one of a left neighboring spatial candidate and/or a top neighboring spatial candidate. Or, for example, in adding an HMVP candidate, the encoder/decoder may perform a pruning check only on a part of the spatial candidates of the merge list, and the part of the spatial candidates may be predefined as left and upper sides. An embodiment of the disclosure is not limited to the above examples, and other types may be combined and limited as a target of a pruning check.
Referring to
Alternatively, the encoder/decoder may check the redundancy of motion information between the first M candidates within an HMVP LUT and a currently decoded PU in adding the motion information of the decoded processing block (e.g. PU) to the HMVP LUT.
Referring to
In one embodiment, as in Table 4 and Table 5, information related to a pruning check may be signaled through a high level syntax. In this case, a syntax element transmitted from the encoder to the decoder may be signaled through a higher level syntax for indicating the number of pruning checks. The high level syntax may be included in a sequence parameter set, a picture parameter set, a slice header, a coding tree unit, a coding unit and/or other proper syntax data header, for example.
In an embodiment of the disclosure, there is proposed an efficient method of selecting an HMVP candidate. When a history motion vector candidate (i.e., HMVP candidate) is inserted into a merge list (or AMVP list), a pruning check may be performed so that the HMVP candidate does not overlap the existing merge list. At this time, in order to perform total redundancy checks between the merge list of size M and the history LUT of size N, the checks of (M−1)×N times are necessary.
Accordingly, in an embodiment of the disclosure, the number of HMVP candidates may be dependent on the merge candidate. For example, the number of HMVP candidates may be dependent on the number of spatial candidates present in a merge list. Alternatively, for example, the number of HMVP candidates may be dependent on the number of spatial candidates and temporal candidates present in a merge list.
If there is another merge candidate present in a merge list, the number of HMVP candidates on which a pruning check will be performed based on a specific criterion (or rule) based on the number of merge candidates and/or the number of HVVP of a merge list can be reduced. Accordingly, the number of redundancy checks in the worst case can be reduced.
For example, in the case of a merge list having a size (or length) of 6, if the merge list is fully filled, the merge list may include a maximum of 5 spatial or other merge candidates. In order to insert an HMVP candidate into 6 HMVP lists, 30 redundancy checks may be necessary in the worst case.
In one embodiment, an example related to constraints on the number of HMVPs to be the targets of the pruning checks is illustrated in Equation 17 and Table 5.
if (existing_candidates>=3) number_hist_to_check=7−existing_candidates [Equation 17]
Referring to Table 5, the number of redundancy checks for an HMVP addition in the worst case can be reduced 12 times not 30 times by limiting the number of HMVPs, that is, a target of a pruning check, to 2.
In an embodiment of the disclosure, there is proposed a method of constructing a merge list using history-based spatial temporal motion vector prediction (H-STMVP). H-STMVP indicates a candidate derived as an average of two history-based spatial MVP and TMVP. The two spatial HMVPs may be obtained from an HMVP buffer. The TMVP may be obtained from a current merge list. In this case, the spatial candidate may be a candidate obtained from the last 2-coded MVs in a decoding sequence prior to a current block.
For example, the last coded MV (referred to as MV_L in the disclosure), an MV, that is, the last to the second (referred to as MV_(L−1) in the disclosure), and MV_TMVP may be used to generate an H-STMVP candidate to be inserted into a merge list.
If all the three candidates can be used, MVs added to the merge list may be calculated by Equation 18 below.
{MVL+MVL-1+MVTMVP}*43/128 [Equation 18]
In one embodiment, if only two of the three candidates are available, only two the candidates may be averaged to generate H-STMVP. Likewise, if only one candidate is available, the one candidate may be used. If an available candidate is not present, H-STMVP may not be used for a merge list configuration.
In an embodiment of the disclosure, there is proposed a method of obtaining the motion vector of an H-STMVP candidate using another method other than Equation 18.
For example, instead of averaging three or more candidates at once, to first average spatial candidates and then average two candidates again using a result of the average may be computatively simpler. An example thereof is illustrated in the following equations.[(MVL+MVL-1)>>1+MVTMVP]>>1
[Equation 19]
Alternatively, an average value may be obtained as follows.[(MVL+MVTMVP)>>1+MVL-1]>>1
[Equation 20]
[(MVL-1+MVTMVP)>>1+MVL]>>1
[Equation 21]
[2MVL+MVL-1+MVTMVP]>>2
[Equation 22]
The encoder/decoder may first average two candidates as in Equations 19 to 21, and then may average result values using the third candidate. Alternatively, the encoder/decoder may assign higher importance/weight to a candidate, that is, MV_L, by applying a twice shift operation as in Equation 22. An average value can be derived without a division operation by only a shift operation using Equations 19 to 22.
In an embodiment of the disclosure, there is proposed a method using a given number (n) of spatial candidates instead of two history-based spatial candidates in deriving H-STMVP. The n candidates do not need to be an essentially consecutive decoding sequence. The n candidates may be selected randomly or according to some rule.
Accordingly, Equation 18 may be represented using a more general method like Equation 23 below.
{MV1+MV2 . . . +MVn+MVTMVP}*1/(n+1) [Equation 23]
In another embodiment, assuming that 5 spatial candidates are used, the influence of spatial candidates increase to generate an H-STMVP candidate can be minimized and a spatial candidate and temporal candidate can be properly incorporated by improving weight applied to a temporal candidate.
Accordingly, to this end, after spatial candidates are averaged together using Equation 24, the above object can be achieved by averaging MV_TMVP using a result of the average.
[{MV1+MV2++MVn}*1/n+MVTMVP]*½ [Equation 24]
In an embodiment of the disclosure, there is proposed a method of adding weight (or weighting factor) to a motion vector candidate used to derive H-STMVP. In this case, the weight may be empirically determined or may be determined by considering a temporal distance up to a fixed reference frame or may be determined by considering a location in a history table. For example, a new candidate may have heavier weight than a previous candidate.
That is, in the present embodiment, Equation 18 may be represent like Equation 25 below.
In this case, weight may have the same value or an unequally distributed value.
In an embodiment of the disclosure, there is proposed a method of scaling a motion vector used to derive an H-STMVP candidate as a single reference picture.
Referring to
The average of Equations 18 to 25 may be made an unequal result value because a frame having a close reference frame may essentially have a greater influence on the motion vector of H-STMVP. Accordingly, there is proposed a method of scaling all motion vectors into a single reference frame for an equal comparison and incorporation.
In this case, the encoder may determine which single frame performed as part of RD optimization is most suitable for being used as a reference frame. In an embodiment, a selected reference frame may be signaled in a slice header similar to a TMVP array index present in a slice header. For example, a reference frame to be used may be generated using a fixed rule. Alternatively, for example, a list may be scaled as the first available reference frame from L0 or a list may be scaled based on a current picture sequence count.
In one embodiment, in order to achieve the above object, the encoder may transmit, to the decoder, information for a single fixed picture using a high level syntax (HLS) that may be part of a sequence parameter set, a picture parameter set, a slice header, a coding tree unit and/or another data header. For example, a high level syntax structure, such as Table 6 and/or Table 7 below, may be defined.
Referring to Table 6, set_HSTMVP_ref_pic_flag equal to 1 indicates that set_HSTMVP_idx is present in the slice header of a non-IDR picture in CVS. set_HSTMVP_ref_pic_flag equal to 0 indicates that set_HSTMVP_idx is not present in a slice header.
Referring to Table 7, slice_HMVP_idx designates a reference index. In one embodiment, a reference index may be selected with respect to a list L0.
In the embodiments of the disclosure, more detailed embodiments will be described with respect to the above-described embodiments. Specifically, a method of indirectly using an affine HMVP candidate by using position and dimensional information in order to calculate or derive CPMV of a current block is proposed. In the disclosure, the derived CPMV may be referred to as an ingerited affine HVMP candidate. The inherited Affine HMVP candidate according to an embodiment of the disclosure may be used in the above-described Affine merge list and/or Affine AMVP list generation process.
Referring to
As an embodiment, the CPMV of the inherited affine HMVP of the current block may be derived by using Equations 26 and 27 below.
V0x=VB0x+(VB2_x−VB0x)*(posCurCU_Y−posRefCU_Y)/RefCU_height+(VB1x−VB0x)*(posCurCU_X−posRefCU_X)/RefCU_width [Equation 26]
V0y=VB0y+(VB2_y−VB0y)*(posCurCU_Y−posRefCU_Y)/RefCU_height+(VB1y−VB0y)*(posCurCU_X−posRefCU_X)/RefCU_width [Equation 27]
In Equations 26 and 27, posCurCU_Y represents the vertical coordinate value of the upper left sample of the current block 3501, and posRefCU_Y represents the vertical coordinate value of the upper left sample of the reference block 3502. posCurCU_X represents the horizontal coordinate value of the upper left sample of the current block 3501, and posRefCU_X represents the horizontal coordinate value of the upper left sample of the reference block 3502. RefCU_height represents the height of the reference block 3502, and RefCU_width represents the width of the reference block 3502.
In one embodiment of the disclosure, when adding an affine HMVP candidate (direct or inherited HMVP), restrictions may be added to select an affine HMVP candidate that can be used to generate an affine merge list or an affine AMVP list.
As an example, an affine HMVP candidate may be added to the affine merge list or the affine AMVP list only when the affine HMVP candidate is adjacent to the current block.
As another example, an affine HMVP candidate may be added to the affine merge list or the affine AMVP list only when the affine HMVP candidate is located (or exists) within a specific distance from the current block. For example, the specific distance may be a predefined pixel distance. The encoder/decoder may determine whether the affine HMVP candidate is located within a predefined specific distance to determine whether the affine HMVP candidate is available.
As another example, the affine HMVP candidate may be added to the affine merge list or the affine AMVP list only when the affine HMVP candidate is located (or exist) at a specific location based on the current block. For example, when the affine HMVP candidate is present at the specific location, it may be a case where the affine HMVP candidate is a left or upper neighboring block of the current block.
For an affine HMVP LUT that has N elements, the above-described verification process for all elements or the first M elements may be performed until the merge or AMVP list is full or until a predefined number of HMVP candidates is reached.
In one embodiment of the disclosure, a method is proposed in which the affine HMVP candidate is used for replacing an inherited affine candidate which has already existed in the affine merge list and/or the affine AMVP list.
Referring to
In addition, in one embodiment of the disclosure, an affine HMVP lookup table (LUT) may be initialized at the beginning of a slice, CTU row, or CTU. Through this, it is possible to improve the performance of parallel processing.
Hereinafter, in the following embodiments, a method for reducing the number of worst pruning checks from HMVP is proposed.
In an embodiment of the disclosure, when an HMVP candidate is added to a merge list, the number of pruning checks may be determined based on the number of available candidates in the merge list and the number of HMVP candidates that can be added to the merge list. Hereinafter, in describing the embodiments of the disclosure, for convenience of description, variables are defined as follows.
In one embodiment of the disclosure, the HMVP candidate may be added to the merge list according to the following conditions.
When an HMVP candidate is added to the merge list (ie, becomes a merge candidate), each HMVP candidate may need a pruning check to remove duplication between merge candidates. According to the existing image compression technique, the worst pruning check number required to add HMVP to the merge list may be calculated as shown in Table 8 below.
Referring to Table 8, according to the existing image compression technique, the pruning check may be performed on six HMVP candidates in the HMVP table (or HMVP list, HMVP candidate list).
Specifically, 1) when there is one candidate in the merge list, HMVP candidates added to the merge list may be 4. Then, pruning checks for six HMVP candidates may be performed. In this case, the number of worst pruning checks may be 4. 2) When there are two candidates in the merge list, HMVP candidates added to the merge list may be 3. Then, pruning checks for six HMVP candidates may be performed. In this case, the number of worst pruning checks may be 7. 3) When there are three candidates in the merge list, HMVP candidates added to the merge list may be 2. Then, pruning checks for six HMVP candidates may be performed. In this case, the number of worst pruning checks may be 9. 4) When there are four candidates in the merge list, HMVP candidates added to the merge list may be 1. Then, pruning checks for six HMVP candidates may be performed. In this case, the number of worst pruning checks may be 10.
In an embodiment of the disclosure, a method for reducing the number of worst pruning checks described above is proposed. If there are more merge candidates in the merge list, since the coding effect of HMVP decreases as the merge candidates (ie, non-HMVP candidates) increase, the number of HMVP candidates to be pruned may need to be reduced. Accordingly, in the embodiment of the disclosure, the encoder/decoder may be set the number of HMVP candidates to be checked (NHMVPChecked) equal to the number of available HMVP candidates to be added (NmrgToBeAdded) in order to reduce the number of worst pruning checks. In this case, the number of worst pruning checks may be calculated as shown in Table 9 below.
Referring to Table 9, the number of worst pruning checks for HMVP can be reduced from 10 to 6 compared to conventional image compression techniques.
Referring to Table 9, in an embodiment, 1) when there is one candidate in the merge list, HMVP candidates added to the merge list may be 4. Then, pruning checks for four HMVP candidates may be performed. In this case, the number of worst pruning checks may be 4. 2) When there are two candidates in the merge list, HMVP candidates added to the merge list may be 3. Then, pruning checks for three HMVP candidates may be performed. In this case, the number of worst pruning checks may be 6. 3) When there are three candidates in the merge list, HMVP candidates added to the merge list may be 2. Then, pruning checks for two HMVP candidates may be performed. In this case, the number of worst pruning checks may be 6. 4) When there are four candidates in the merge list, HMVP candidates added to the merge list may be 1. Then, pruning checks for one HMVP candidate may be performed. In this case, the number of worst pruning checks may be 4.
In an embodiment of the disclosure, in order to reduce the number of worst pruning checks, the encoder/decoder may set the number of HMVP candidates (NHMVPChecked) to be pruning-checked to be same as the sum of the number of available HMVP candidates (NmrgToBeAdded) to be added and K. Here, K represents a predefined constant value. As an example, when K is 1, the worst pruning check number may be calculated as shown in Table 10 below.
Referring to Table 10, in an embodiment, 1) when there is one candidate in the merge list, HMVP candidates added to the merge list may be 4. Then, pruning checks for five HMVP candidates may be performed. In this case, the number of worst pruning checks may be 4. 2) When there are two candidates in the merge list, HMVP candidates added to the merge list may be 3. Then, pruning checks for four HMVP candidates may be performed. In this case, the number of worst pruning checks may be 7. 3) When there are three candidates in the merge list, HMVP candidates added to the merge list may be 2. Then, pruning checks for three HMVP candidates may be performed. In this case, the number of worst pruning checks may be 8. 4) When there are four candidates in the merge list, HMVP candidates added to the merge list may be 1. Then, pruning checks for two HMVP candidates may be performed. In this case, the number of worst pruning checks may be 7.
In an embodiment of the disclosure, in order to reduce the pruning check in a worst case, the number of HMVP candidates to be checked (NHMVPChecked) may be defined as Equation 28 below.
NHMVPChecked=min(NHMVP,C*NmrgToBeAdded) [Equation 28]
In Equation 28, C represents a predefined constant value. If C is 2, the worst pruning check number may be calculated as shown in Table 11 below.
Referring to Table 11, in an embodiment, 1) when there is one candidate in the merge list, HMVP candidates added to the merge list may be 4. Then, pruning checks for six HMVP candidates may be performed. In this case, the number of worst pruning checks may be 4. 2) When there are two candidates in the merge list, HMVP candidates added to the merge list may be 3. Then, pruning checks for six HMVP candidates may be performed. In this case, the number of worst pruning checks may be 7. 3) When there are three candidates in the merge list, HMVP candidates added to the merge list may be 2. Then, pruning checks for four HMVP candidates may be performed. In this case, the number of worst pruning checks may be 9. 4) When there are four candidates in the merge list, HMVP candidates added to the merge list may be 1. Then, pruning checks for two HMVP candidates may be performed. In this case, the number of worst pruning checks may be 7.
When HMVP is added to the merge list and/or AMVP list, pruning needs to be performed again to avoid duplication with the candidates existing in the merge list and/or AMVP list. If the HMVP LUT is already pruned by the limited FIFO operation as described in
In one embodiment of the disclosure, the size of the HMVP lookup table (LUT) considering a HMVP candidate insertion process for constructing a merge list and/or AMVP list may be defined. Specifically, HMVP candidates may be added up to a predefined merge list size. For example, when the size of the maximum merge list is defined as 6, the HMVP may not be the 6th candidate. If merge candidates are available (or present) in the merge list, the HMVP candidate may not be added. In this case, the sixth candidate may be selected from candidates other than HMVP (or according to another method). Therefore, considering the above-described HMVP candidate insertion process, in one embodiment of the disclosure, the following HMVP LUT size selection method is proposed.
As an embodiment, the HMVP LUT table size may be defined or set equal to (MaxNumMergeCand-K). Here, MaxNumMergeCand indicates the maximum merge candidate list (or the maximum number of candidates included in the merge candidate list or the maximum number of merge candidates), and at this time, MaxNumMergeCand may be defined as 6. K represents a predefined constant. For example, K may be 1, and the HMVP LUT size may be 5.
According to an embodiment of the disclosure, by limiting HMVP to MaxNumMergeCand-K (as described above, for example, K is 1), the worst number of pruning checks (or worst case) may be reduced when adding HMVP to the merge list and/or HMVP table. (Limited FIFO operation described previously in
In an embodiment of the disclosure, an update process, such as the embodiment described below, may be applied to a table having HMVP motion candidates in consideration of the above-described HMVP table size. The following embodiments are examples of the HMVP table update process, and embodiments of the disclosure are not limited thereto.
HMVP Table Update Process
First, the input of the HMVP table update process may be defined as follows.
The output of this update process may be a modified array of the HMVP candidate list. In this process, mvCand represents a variable indicating a motion vector candidate having the motion vectors, the reference indices, and the prediction list utilization flags.
This update process may be performed in the following steps.
1. A variable identicalCandExist is set to false, and a variable tempIdx is set to 0. Here, identicalCandExist is a variable indicating whether the same motion information exists in the HMVP candidate list, and tempIdx is a variable indicating the index of the HMVP candidate, in the HMVP candidate list, having the same motion information as a current motion vector.
2. If HMVPCandNum is greater than 0, for each index HMVPIdx with HMVPIdx=0 . . . HMVPCandNum-1, the following steps may be applied until the identicalCandExist variable is true. Here, HMVPCandNum represents the number of HMVP candidates in the HMVP candidate list, and HMVPIdx represents the index assigned to HMVP candidates in the HMVP candidate list.
3. And, the HMVP candidate list may be updated according to the following steps.
(1) If identicalCandExist is true or HMVPCandNum is MaxNumMergeCand-K, the following steps may be applied. Here, MaxNumMergeCand is a variable indicating the size of merge list (or merge candidate list) (or the maximum number of candidates of the merge list, the maximum number of merge candidates), and K is an arbitrary constant. In an embodiment, K may be predefined. Further, in an embodiment, K may be set to 1, and MaxNumMergeCand-K may be 5 as the MaxNumMergeCand value is defined as 6. Also, in an embodiment, MaxNumMergeCand-K may be set to 5. For example, if identicalCandExist is true or HMVPCandNum is 5, the following steps may be applied.
(2) Otherwise (i.e., if identicalCandExist is false, and HMVPCandNum is less than MaxNumMergeCand-K), the following steps may be applied. As described above, in an embodiment, MaxNumMergeCand-K may be set to 5. For example, otherwise (i.e., identicalCandExist is false, and HMVPCandNum is less than 5), the following steps may be applied.
As an example, this update process may be called when the current slice is a P or B slice. At this time, the variable HMVPCandNum is set to 0 and the variable HMVPCandList may be defined as an element array of MaxNumMergeCand-K. As described above, in an embodiment, MaxNumMergeCand-K may be set to 5. For example, at this time, the variable HMVPCandNum may be set to 0 and the variable HMVPCandList may be defined as an element array of 5.
Hereinafter, another example of the HMVP table update process will be described.
First, the input of the update process may be defined as follows.
The output of this update process may be a modified array of the HMVP candidate list. In this process, mvCand represents a variable indicating a motion vector candidate having the motion vectors, the reference indices, and the prediction list utilization flags.
This update process may be performed in the following steps.
1. For each index HMVPIdx with HMVPIdx=0 . . . HMVPCandNum-1, the following steps may be applied in order until the variable sameCand becomes true. Here, sameCand is a variable indicating whether the same motion information exists in the HMVP candidate list.
2. The variable tempIdx is set to HMVPCandNum.
3. If sameCand is true or HMVPCandNum is MaxNumMergeCand-K, for each index tempIdx with tempIdx=(sameCand? HMVPIdx: 1) . . . HMVPCandNum-1, HMVPCandList[tempIdx] is copied or set to HMVPCandList[tempIdx-1]. K is an arbitrary constant. In an embodiment, K may be predefined. Further, in an embodiment, K may be set to 1, and MaxNumMergeCand-K may be 5 as the MaxNumMergeCand value is defined as 6. Also, in an embodiment, MaxNumMergeCand-K may be set to 5.
4. mvCand is copied to HMVPCandList[tempIdx].
5. If HMVPCandNum is less than MaxNumMergeCand-K, HMVPCandNum increases by one. As described above, in an embodiment, MaxNumMergeCand-K may be set to 5.
The aforementioned embodiments of the disclosure may be divided and described, for convenience of description, but the disclosure is not limited thereto. That is, the embodiments above described may be independently performed or one or more several embodiments may be combined and performed.
Referring to
In the method of processing a video signal based on history-based motion vector prediction, the decoder configures a merge candidate list based on neighboring blocks of a current block (S3701).
The decoder adds a history-based merge candidate of the current block to the merge candidate list (S3702).
The acquires a merge index indicating a merge candidate used for inter prediction of the current block in the merge candidate list (S3703).
The decoder generates prediction samples of the current block based on the motion information of the merge candidate indicated by the merge index (S3704).
The decoder updates a history-based merge candidate list based on the motion information (S3705).
As described above, as an embodiment, the history-based merge candidate may be added to the merge candidate list if the history-based merge candidate has motion information not overlapping that of a pre-defined merge candidate, among merge candidates included in the merge candidate list.
As described above, as an embodiment, the history-based merge candidate list may be defined to have a size determined based on the maximum number of merge candidates of the merge candidate list.
As described above, as an embodiment, the history-based merge candidate list may be defined to have a size of a value obtained by subtracting 1 from the maximum number of merge candidates of the merge candidate list.
As described above, as an embodiment, the size of the history-based merge candidate list may be defined as 5.
As described above, as an embodiment, the history-based merge candidate may be added to the merge candidate list if the history-based merge candidate has motion information not overlapping that of a pre-defined specific number of merge candidates, among merge candidates included in the merge candidate list.
As described above, as an embodiment, the history-based merge candidate may be added to the merge candidate list if the history-based merge candidate has motion information not overlapping that of a specific spatial merge candidate included in the merge candidate list.
As described above, as an embodiment, the history-based merge candidate may be derived from a pre-defined number of candidates within a history-based merge candidate list.
As described above, as an embodiment, the decoder may add a zero motion vector to the merge list when the merge list does not satisfy the maximum number of merge lists.
The embodiments described in the disclosure may be implemented and performed on a processor, a microprocessor, a controller or a chip. For example, the function units illustrated in the drawings may be implemented and performed on a computer, a processor, a microprocessor, a controller or a chip.
Furthermore, the decoder and the encoder to which the disclosure is applied may be included in a multimedia broadcasting transmission and reception device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a camera for monitoring, a video dialogue device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, a video on-demand (VoD) service provision device, an over the top (OTT) video device, an Internet streaming service provision device, a three-dimensional (3D) video device, a video telephony device, and a medical video device, and may be used to process a video signal or a data signal. For example, the OTT video device may include a game console, a Blueray player, Internet access TV, a home theater system, a smartphone, a tablet PC, and a digital video recorder (DVR).
Furthermore, the processing method to which the disclosure is applied may be produced in the form of a program executed by a computer, and may be stored in a computer-readable recording medium. Multimedia data having a data structure according to the disclosure may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices in which computer-readable data is stored. The computer-readable recording medium may include Blueray disk (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, for example. Furthermore, the computer-readable recording medium includes media implemented in the form of carriers (e.g., transmission through the Internet). Furthermore, a bit stream generated using an encoding method may be stored in a computer-readable recording medium or may be transmitted over wired and wireless communication networks.
Furthermore, an embodiment of the disclosure may be implemented as a computer program product using program code. The program code may be performed by a computer according to an embodiment of the disclosure. The program code may be stored on a carrier readable by a computer.
The decoding apparatus and the encoding apparatus to which the disclosure is applied may be included in a digital device. The term “digital device” includes all digital devices capable of performing at least one of transmission, reception, processing, and output, for example, of data, content, and services. Here, processing of data, content, service, etc. by a digital device includes an operation of encoding and/or decoding data, content, service, and the like. The digital device is paired or connected (hereinafter referred to as ‘pairing’) with other digital devices, external servers, etc. through a wired/wireless network to transmit and receive data, and converts it if necessary.
The digital device includes, for example, a fixed device (or a standing device) such as a network TV, an HBBTV (Hybrid Broadcast Broadband TV), a smart TV, an IPTV (internet protocol television), a PC, etc. and a mobile device (or a handheld device) such as a PDA (Personal Digital Assistant), a smart phone, a tablet PC, a laptop. In the disclosure described later, for convenience,
Meanwhile, the term “wired/wireless network” described herein refers to a communication network that supports various communication standards or protocols for interconnection and/or data transmission and reception between digital devices or between digital devices and external servers. These wired/wireless networks may include both current and future supported communication networks and communication protocols for them, and may be formed by a communication standard or protocols for wired connection, such as USB (Universal Serial Bus), CVBS (Composite Video Banking Sync), Component, S-Video (Analog), DVI (Digital Visual Interface), HDMI (High Definition Multimedia Interface), RGB, D-SUB and the like and formed by a communication standards for wireless connection such as Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Digital Living Network Alliance (DLNA), Wireless LAN (WLAN) (Wi-Fi)), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Down-link Packet Access), LTE (Long Term Evolution), Wi-Fi Direct (Direct), etc.
Hereinafter, in the case of simply referring to a digital device in the disclosure, it may mean a fixed device or a mobile device or include both depending on context.
Meanwhile, the digital device is, for example, an intelligent device that supports a broadcast reception function, a computer function, and at least one external input, and may support e-mail, web browsing, banking, game, and application through the wired/wireless network described above. In addition, the digital device may include an interface for supporting at least one input or control means (hereinafter referred to as an input means) such as a hand-operated input device, a touch screen, and a space remote control. The digital device may use a standardized general-purpose operating system (OS). For example, the digital device may add, delete, modify, and update various applications on a general-purpose OS kernel, and may configure and provide a more user-friendly environment through them.
Meanwhile, the external input described in the disclosure includes external input devices, that is, all input means or digital devices that are wired/wirelessly connected to the above-described digital devices and transmit/receive related data therethrough. Here, the external input may include all devices such as a high-definition multimedia interface (HDMI), a game device such as a play station or an X-Box, a smart phone, a tablet PC, a printer, or a smart TV.
And, the term “server” described in the disclosure includes a client, that is, all digital devices or systems that supply data to the digital devices described above, and is called a processor. Examples of such the server include a portal server that provides a web page or web content, an advertising server that provides advertising data, a content server that provides content, and a social media service (SNS).server, a service server or manufacturing server provided by a manufacturer, etc.
Furthermore, “channel” described herein means a path, means, etc. for transmitting and receiving data, and may be exemplified by a broadcasting channel. Here, the broadcast channel is expressed in terms of a physical channel, a virtual channel, and a logical channel according to the activation of digital broadcasting. The broadcast channel may be called a broadcast network. As described above, the broadcast channel refers to a channel for providing or accessing broadcast content provided by a broadcasting station, and the broadcast content is mainly based on real-time broadcasting and is also called a live channel. However, recently, the medium for broadcasting has become more diversified, and non-real time broadcasting is also activated in addition to real-time broadcasting. It may also be understood as a term meaning the entire channel. So, the live channel may be understood as a term meaning not only real-time broadcasting but also the entire broadcasting channel including non-real-time broadcasting in some cases.
In the disclosure, “arbitrary channel” is further defined in relation to a channel other than the above-described broadcast channel. The arbitrary channel may provide a service guide such as an electronic program guide (EPG) along with a broadcast channel. A service guide, a GUI (Graphic User Interface), or an OSD screen (On-Screen Dis-play screen) may be configured/provided with only the arbitrary channel.
Meanwhile, unlike a broadcast channel having a predetermined channel number between transceivers, the arbitrary channel is a channel randomly allocated in a receiver, and a channel number that is not basically overlapped with channel numbers for expressing the broadcast channels is allocated to the arbitrary channel. For example, when a specific broadcast channel is tuned, the receiver receives a broadcast signal that transmits broadcast content and signaling information therefor through the tuned channel. Here, the receiver parses channel information from the signaling information, and configures a channel browser, an EPG, and the like based on the parsed channel information and provides it to a user. When the user makes a channel change request through the input means, the receiver responds accordingly.
As described above, since the broadcast channel is promised in advance between transmitting and receiving terminals, when the arbitrary channel is allocated to a broadcast channel in an overlapped manner, it may cause confusion to the user or there may be a possibility of confusion, so it is preferable not to allocate duplicates as described above. Meanwhile, even if the arbitrary channel number is not overlapped with the broadcast channel number as described above, there is still a confusion in the channel surfing process of the user, and it is required to allocate the arbitrary channel number in consideration of this. This is because the arbitrary channel according to the disclosure may also be implemented to be accessed in response to a user's request for channel switching through an input means in the same way as a conventional broadcast channel. Accordingly, the arbitrary channel number may be defined and displayed in a form in which characters are written in parallel, such as arbitrary channel-1, arbitrary channel-2, and the like, rather than in a numeric form like a broadcast channel, for the convenience of user access to the arbitrary channel and for convenience in distinguishing or discriminating from the broadcasting channel number. In this case, although the display of an arbitrary channel number may be realized in the form in which characters are written as in arbitrary channel-1, the receiver may recognize and implement the arbitrary channel in a numeric form as the number of the broadcast channel internally. Also, the arbitrary channel number may be provided in the form of a number, such as a broadcast channel, and it is also possible to define and display channel numbers in various ways that can be distinguished from broadcast channels, such as video channel-1, title-1, and video-1.
The digital device executes a web browser for a web service, and provides various types of web pages to the user. Here, the web page includes a web page including a video content, and in the disclosure the video is processed separately or independently from the web page. And, the separated video may be allocated to the arbitrary channel as described above, and provided through a service guide, or the like, and output according to a channel switching request in a process of viewing a service guide or a broadcast channel. Further, for services such as broadcast content, games, and applications in addition to web services, predetermined content, images, audio, items, etc. are separately processed from the broadcast content, games, and applications themselves, and the arbitrary channel number may be assigned for its reproduction, processing, and the like, and may be implemented as described above.
Service systems including digital devices include a content provider CP 3810, a service provider SP 3820, a network provider NP 3830, and a home network end user HNED (Customer) 3840. Here, the HNED 3840 is, for example, a client 3800, that is, a digital device. The content provider 3810 produces and provides various content. As shown in
The service provider 3820 provides the content provided by the content provider 3810 as a service package to the HNED 3840 by packaging the content. For example, the service provider 3820 of
The service provider 3820 provides services to the client 300 in a uni-cast or multi-cast manner. And, the service provider 3820 may transmit data to a plurality of pre-registered clients 3800 at a time, and for this an Internet Group Management Protocol IGMP protocol may be used.
The above-described content provider 3810 and service provider 3820 may be identical or single entities. For example, the content provided by the content provider 3810 may be service packaged and provided to the HNED 3840, so the functions of the service provider 3820 together or vice versa are performed.
The network provider 3830 provides a network for data exchange between the content provider 3810 or/and the service provider 3820 and the client 3800.
The client 3800 may establish a home network to transmit and receive data.
Meanwhile, the content provider 3810 or/and the service provider 3820 in the service system may use conditional access or content protection means to protect transmitted content. In this case, the client 300 may use processing means such as a CableCARD (POD: Point of De-ployment), DCAS (Downloadable CAS), etc. in response to the restriction reception or content protection.
In addition, the client 3800 may also use a bidirectional service through a network (or communication network). In this case, the client 3800 may rather perform the function of the content provider, and the existing service provider 3820 may receive it and transmit it back to another client.
The digital device 3900 includes a network interface 3901, a TCP/IP manager 3902, a service delivery manager 3903, an SI decoder 3904, a Demultiplexer (demux) 3905, an audio decoder 3906, a video decoder 3907, a display module (display A/V and OSD) 3908, a service control manager 3909, a service discovery manager 3910, SI & metadata database 3911, a metadata manager 3912, a service manager 3913, an UI manager 3914 and the like.
The network interface 3901 receives or transmits IP packets through the network. That is, the network interface 3901 receives services, content, and the like from the service provider 3820 through the network.
TCP/IP manager 3902 get involved in packet transmission between a source and a destination for IP packets received by the digital device 3900 and IP packets transmitted by the digital device 3900. And the TCP/IP manager 3902 classifies the received packets to correspond to an appropriate protocol and outputs the classified packets the service delivery manager 3905, the service discovery manager 3910, the service control manager 3909, the metadata manager 3912, etc. The service delivery manager 3903 is responsible for controlling received service data. For example, the service delivery manager 3903 may use RTP/RTCP when controlling real-time streaming data. When the real-time streaming data is transmitted using RTP, the service delivery manager 3903 parses the received data packet according to the RTP and transmits it to the demultiplexer 3905 or stores it in the SI & metadata database 3911 under the control of the service manager 3913. And, the service delivery manager 3903 uses the RTCP to feed back the network reception information to a server providing a service. The demultiplexer 3905 demultiplexes the received packets into audio, video, and system information SI data, and transmits them to the audio/video decoder 3906/3907 and the SI decoder 3904, respectively.
The SI decoder 3904 decodes service information such as program specific information PSI, program and system information protocol PSIP, and digital video broadcasting-service information DVB-SI.
And, the SI decoder 3904 stores the decoded service information in the SI & metadata database 3911, for example. The service information stored in this way may be read and used by a corresponding configuration, for example, by a user's request.
The audio/video decoder 3906/3907 decodes each audio data and video data demultiplexed by the demultiplexer 3905. Thus decoded audio data and video data are provided to the user through the display module 3908.
The application manager may include, for example, the UI manager 3914 and the service manager 3913. The application manager manages the overall state of the digital device 3900, provides a user interface, and manages other managers.
The UI manager 3914 provides a graphical user interface GUI for a user using an on-screen display OSD or the like, and receives key input from a user to perform device operation according to the input. For example, when the UI manager 3914 receives a key input regarding channel selection from a user, the UI manager 3914 transmits the key input signal to the service manager 3913.
The service manager 3913 controls managers associated with a service, such as the service delivery manager 3903, the service discovery manager 3910, the service control manager 3909, and the metadata manager 3912.
And, the service manager 3913 creates a channel map and selects a channel using the channel map according to a key input received from the user interface manager 3914. And, the service manager 3913 receives the channel service information from the SI decoder 3904 and sets the audio/video packet identifier PID of the selected channel to the demultiplexer 3905. The PID set in this way is used in the demultiplexing process described above. Therefore, the demultiplexer 3905 filters the audio data, video data, and SI data using the PID.
The service discovery manager 3910 provides information necessary to select a service provider providing a service. When a signal regarding channel selection is received from the service manager 3913, the service discovery manager 3910 finds a service using the information.
The service control manager 3909 is responsible for selecting and controlling services. For example, the service control manager 3909 uses IGMP or RTSP or the like when the user selects a live broadcasting service such as a conventional broadcasting method, and uses RTSP to select and control the service when selecting a service such as VOD (video on demand). The RTSP protocol may provide a trick mode for real-time streaming. And, the service control manager 3909 may initialize and manage a session through the IMS gateway 3950 using an IP multimedia subsystem IMS and a session initiation protocol SIP. The protocol is one embodiment, and other protocols may be used according to implementation examples.
The metadata manager 3912 manages metadata associated with a service and stores the metadata in the SI & metadata database 3911.
The SI & metadata database 3911 stores service information decoded by the SI decoder 3904, metadata managed by the metadata manager 3912, and information necessary to select a service provider provided by the service discovery manager 3910. And, the SI & metadata database 3911 may store set-up data and the like for the system.
The SI & metadata database 3911 may be implemented using a non-volatile RAM NVRAM, a flash memory, or the like.
Meanwhile, the IMS gateway 3950 is a gateway that collects functions necessary for accessing an IMS-based IPTV service.
Referring to
The wireless communication unit 4010 may include one or more modules that enable wireless communication between the mobile device 4000 and the wireless communication system or between the mobile device and the network where the mobile device is located. For example, the wireless communication unit 4010 may include a broadcast receiving module 4011, a mobile communication module 4012, a wireless Internet module 4013, a short-range communication module 4014, and a location information module 4015.
The broadcast receiving module 4011 receives a broadcast signal and/or broadcast-related information from an external broadcast management server through a broadcast channel. Here, the broadcast channel may include a satellite channel and a terrestrial channel. The broadcast management server may mean a server that generates and transmits broadcast signals and/or broadcast-related information or a server that receives previously generated broadcast signals and/or broadcast-related information and transmits them to a terminal. The broadcast signal may include a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, and may also include a combined signal in which the data broadcast signal is combined with the TV broadcast signal or the radio broadcast signal.
The broadcast related information may mean information related to a broadcast channel, broadcast program, or broadcast service provider. Broadcast-related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 4012.
Broadcast-related information may exist in various forms, for example, an electronic program guide EPG or an electronic service guide ESG.
The broadcast receiving module 4011 may receive a digital broadcast signal using a digital broadcast system such as, for example, ATSC, digital video broadcasting-terrestrial DVB-T, satellite DVB-S, media forward link only MediaFLO, handheld DVB-H, integrated services digital broadcast-terrestrial ISDB-T and the like. Of course, the broadcast receiving module 4011 may be configured to be suitable for other broadcasting systems as well as the digital broadcasting system described above.
The broadcast signal and/or broadcast-related information received through the broadcast receiving module 4011 may be stored in the memory 4060.
The mobile communication module 4012 transmits and receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include various types of data according to transmission and reception of a voice signal, a video call signal, or a text/multimedia message.
The wireless Internet module 4013 includes a module for wireless Internet access, and may be built in or external to the mobile device 4000. As wireless Internet technology, wireless LAN (WLAN) (Wi-Fi), wireless broadband Wibro, world interoperability for microwave access Wimax, and high speed downlink packet access HSDPA may be used.
The short-range communication module 4014 refers to a module for short-range communication. Short-range communication Bluetooth, RFID (radio frequency Identification), infrared communication (IrDA, infrared data association), UWB (ultra wideband), ZigBee, RS-232, RS-485, etc. may be used.
The location information module 4015 is a module for obtaining location information of the mobile device 4000, and may use a global positioning system GPS module as an example.
The A/V input unit 4020 is for audio or/and video signal input, and may include a camera 4021, a microphone 4022, and the like. The camera 4021 processes image frames such as still images or moving pictures obtained by an image sensor in a video call mode or a shooting mode. The processed image frame may be displayed on a display unit 4051.
The image frames processed by the camera 4021 may be stored in the memory 4060 or transmitted to the outside through the wireless communication unit 4010. Two or more cameras 4021 may be provided depending on the use environment.
The microphone 4022 receives an external sound signal by a microphone in a call mode or a recording mode, a voice recognition mode, etc., and processes it as electrical voice data. The processed voice data may be converted and output in a form that can be transmitted to the mobile communication base station through the mobile communication module 4012 in the call mode. The microphone 4022 may be implemented with various noise reduction algorithms for removing noise generated in the process of receiving an external sound signal.
The user input unit 4030 generates input data for the user to control the operation of the terminal. The user input unit 4030 may be configured by a key pad, a dome switch, a touch pad (pressure type/capacitive type), a jog wheel, a jog switch, or the like.
The sensing unit 4040 displays the current state of the mobile device 4000, such as the open/closed state of the mobile device 4000, the location of the mobile device 4000, the presence or absence of user contact, the orientation of the mobile device, and acceleration/deceleration of the mobile device. It senses and generates a sensing signal for controlling the operation of the mobile device 4000. For example, when the mobile device 4000 is moved or tilted, the position or tilt of the mobile device may be sensed. And, whether power is supplied to the power supply unit 4090 or whether external devices are coupled to the interface unit 4070 may be sensed. Meanwhile, the sensing unit 4040 may include a proximity sensor 4041 including near field communication NFC.
The output unit 4050 is for generating output related to vision, hearing, or tactile sense, and may include a display unit 4051, an audio output module 4052, an alarm unit 4053, and a haptic module 4054.
The display unit 4051 displays or outputs information processed by the mobile device 4000. For example, when the mobile device is in a call mode, a user interface UI or a graphic user interface GUI related to the call is displayed. When the mobile device 4000 is in a video call mode or a shooting mode, the photographed and/or received video, UI or GUI is displayed.
The display portion 4051 may include a liquid crystal display LCD, a thin film transistor-liquid crystal display TFT LCD, an organic light-emitting diode OLED, and a flexible display, and a 3D display.
Some of these displays may be of a transparent type or a light transmissive type so that the outside can be seen through them. This may be called a transparent display, and a typical example of the transparent display is a transparent OLED TOLED. The rear structure of the display unit 4051 may also be configured as a light transmissive structure. With this structure, the user can view objects located behind the terminal body through the area occupied by the display unit 4051 of the terminal body.
Two or more display units 4051 may be present depending on the implementation form of the mobile device 4000. For example, a plurality of display units may be spaced apart from one surface or integrally disposed in the mobile device 4000, or may be respectively disposed on different surfaces.
When the display unit 4051 and a sensor detecting a touch operation (hereinafter referred to as a ‘touch sensor’) form a mutual layer structure (hereinafter referred to as a ‘touch screen’), the display unit 4051 may be used as an input device in addition to an output device. The touch sensor may have a form of, for example, a touch film, a touch sheet, or a touch pad.
The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 4041 or capacitance generated in a specific portion of the display unit 4051 into an electrical input signal. The touch sensor may be configured to detect not only the touched position and area, but also the pressure at the time of touch.
If there is a touch input to the touch sensor, the corresponding signals are sent to the touch controller. The touch controller processes the signals and then transmits corresponding data to the controller 4080. Accordingly, the control unit 4080 may know which area of the display unit 4051 has been touched, and the like.
The proximity sensor 4041 may be disposed in an inner area of the mobile device surrounded by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object in the vicinity using mechanical force or infrared light without mechanical contact. The proximity sensors have a longer lifespan and higher utilization than contact sensors.
Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, it is configured to detect the proximity of the pointer due to a change in electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.
Hereinafter, for convenience of description, the act of causing the pointer to be recognized as being positioned on the touch screen without being touched by the pointer on the touch screen is referred to as a “proximity touch”, and the act of actually touching the pointer on the touch screen is referred to as “contact touch”. The location on the touch screen that is a proximity touch with a pointer means a location where the pointer corresponds vertically to the touch screen when the pointer is touched close.
The proximity sensor detects a proximity touch and a proximity touch pattern (eg, proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.). Information corresponding to the sensed proximity touch operation and the proximity touch pattern may be output on the touch screen.
The audio output module 4052 may output audio data received from the wireless communication unit 4010 in a call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode, or the like or audio data stored in the memory 4060. The audio output module 4052 may also output audio signals related to functions (for example, call signal reception sound, message reception sound, etc.) performed in the mobile device 4000. The audio output module 4052 may include a receiver, a speaker, and a buzzer.
The alarm module 4053 outputs a signal for notifying the occurrence of the event of the mobile device 4000. Examples of events generated in the mobile device include call signal reception, message reception, key signal input, and touch input. The alarm unit 4053 may output a signal for notifying the occurrence of an event by other forms, for example, vibration, in addition to a video signal or an audio signal.
The video signal or the audio signal may also be output through the display unit 4051 or the audio output module 4052, so that the display unit and the audio output modules 4051 and 4052 may be classified as part of the alarm module 4053.
The haptic module 4054 generates various tactile effects that the user can feel. Vibration is a typical example of the tactile effect generated by the haptic module 4054. The intensity and pattern of vibration generated by the haptic module 4054 are controllable. For example, different vibrations may be synthesized and output or sequentially output.
In addition to vibration, the haptic module 4054 may generate various tactile effects including the effects of stimulation such as a pin arrangement that vertically moves with respect to the contact surface of the skin, the ejection force or inhalation force of the air through the ejection or intake, grazing on a skin surface, contact with the electrode, electrostatic force and the like, and the effect by reproducing a feeling of cold and warm using an element capable of absorbing heat or generating heat.
The haptic module 4054 may not only deliver the tactile effect through direct contact, but also implement it so that the user can feel the tactile effect through muscle sensations such as fingers or arms. Two or more haptic modules 4054 may be provided according to a configuration aspect of the mobile device 4000.
The memory 4060 may store a program for the operation of the control unit 4080, and may temporarily store input/output data (eg, a phone book, message, still image, video, etc.). The memory 4060 may store data related to various patterns of vibration and sound output when a touch is input on the touch screen.
The memory 4060 may include a storage medium of at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), a random access memory RAM, a static random access memory SRAM, a read-only memory ROM, an electrically erasable programmable read-only memory EEPROM, a programmable read-only memory PROM, a magnetic memory, an optical disk. The mobile device 4000 may operate in connection with a web storage that performs a storage function of the memory 4060 on the Internet.
The interface unit 4070 serves as a passage with all external devices connected to the mobile device 4000. The interface unit 4070 receives data from an external device, receives power, and transmits data to each component inside the mobile device 4000, or allows data inside the mobile device 4000 to be transmitted to the external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting devices equipped with an identification module, an audio input/output port, a video I/O port, an earphone port, and the like may be included in the interface unit 4070.
The identification module is a chip that stores various information for authenticating the usage rights of the mobile device 4000, and may include a user identification module UIM, a subscriber identity module SIM, a universal subscriber identity module USIM, and the like. The device equipped with the identification module (hereinafter referred to as ‘identification device’) may be manufactured in a smart card format. Therefore, the identification device may be connected to the terminal 4000 through the port.
When the mobile terminal 4000 is connected to an external cradle, the interface unit 4070 may become a passage through which power from the cradle is supplied to the mobile terminal 4000, or a passage though which various command signals input from the cradle by the user. Various command signals or power input from the cradle may be operated as a signal for recognizing that the mobile terminal is correctly mounted on the cradle.
The control unit 4080 typically controls the overall operation of the mobile device. For example, it performs control and processing related to voice calls, data communication, video calls, and the like. The control unit 4080 may include a multimedia module 4081 for multimedia playback. The multimedia module 4081 may be implemented in the control unit 4080, or may be implemented separately from the control unit 4080. The control unit 4080, in particular the multimedia module 4081, may include the encoding device 100 and/or the decoding device 200 described above.
The control unit 4080 may perform a pattern recognition process capable of recognizing handwriting input or picture drawing input performed on a touch screen as characters and images, respectively.
The power supply unit 4090 receives external power and internal power under control of the control unit 4080 and supplies power required for the operation of each component.
The various embodiments described herein may be implemented in a computer or similar device-readable recording medium using, for example, software, hardware, or a combination thereof.
According to the hardware implementation, embodiments described herein i may be implemented using at least one of application specific integrated circuits ASICs, digital signal processors DSPs, digital signal processing devices DSPDs, programmable logic devices PLDs, field programmable gate arrays FPGAs, processors, controllers, micro-controllers, microprocessors, and electrical units for performing functions. In some cases, the embodiments described herein may be implemented by the control unit 4080 itself.
According to the software implementation, embodiments such as procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code may be implemented in a software application written in an appropriate programming language. Here, the software code is stored in the memory 4060 and may be executed by the control unit 4080.
Another example of the digital device 4100 may include a broadcast receiving unit 4105, an external device interface unit 4156, a storage unit 4140, a user input interface unit 4150, a control unit 4170, a display unit 4180, audio output unit 4185, a power supply unit 4190 and a photographing unit (not shown). Here, the broadcast receiving unit 4105 may include at least one tuner 4110, a demodulation unit 4120, and a network interface unit 4130. However, depending on the case, the broadcast receiving unit 4105 may include a tuner 4110 and a demodulator 4120, but may not include the network interface unit 4130, and vice versa. Also, although not shown, the broadcast receiving unit 4105 may be equipped with a multiplexer to multiplex the signal tuned by the tuner 4110 and then demodulated by the demodulation unit 4120 and the signal received through the network interface unit 4130. Also, although not shown, the broadcast receiving unit 4105 may be equipped with a demultiplexer to demultiplex the multiplexed signal, the demodulated signal or the signal that has passed through the network interface unit 4130.
The tuner 4110 receives an RF broadcast signal by tuning a channel selected by a user or all pre-stored channels among radio frequency RF broadcast signals received through an antenna. And, the tuner 4110 converts the received RF broadcast signal into an intermediate frequency IF signal or a baseband signal.
For example, if the received RF broadcast signal is a digital broadcast signal, it is converted into a digital IF signal DIF, and if it is an analog broadcast signal, it is converted into an analog baseband video or audio signal CVBS/SIF. That is, the tuner 4110 may process both digital broadcast signals or analog broadcast signals. The analog baseband video or audio signal CVBS/SIF output from the tuner 4110 may be directly input to the controller 4170.
The tuner 4110 may receive a single carrier RF broadcast signal according to an advanced television system committee ATSC scheme or a RF broadcast signal with multiple carriers according to a digital video broadcasting DVB scheme.
Meanwhile, the tuner 4110 may sequentially tune and receive RF broadcast signals of all broadcast channels stored through a channel storage function among RF broadcast signals received through an antenna, and convert them into an intermediate frequency signal or a baseband signal.
The demodulation unit 4120 receives and demodulates the digital IF signal DIF converted by the tuner 4110. For example, when the digital IF signal output from the tuner 4110 is an ATSC system, the demodulation unit 4120 performs 8-vestigal side band 8-VSB demodulation, for example. Also, the demodulation unit 4120 may perform channel decoding. To this end, the demodulation unit 4120 may include a trellis decoder, a de-interleaver, a Reed-Solomon decoder, and the like, and perform trellis decoding, deinterleaving, and Reed Soloman decoding.
For example, when the digital IF signal output from the tuner 4110 is a DVB method, the demodulation unit 4120 performs coded orthogonal frequency division modulation COFDMA demodulation, for example. Further, the demodulation unit 4120 may perform channel decoding. To this end, the demodulation unit 4120 may include a convolution decoder, a deinterleaver, and a lead-soloman decoder, and perform convolution decoding, deinterleaving, and read soloman decoding.
The demodulation unit 4120 may output a stream signal TS after demodulation and channel decoding. In this case, the stream signal may be a signal in which a video signal, an audio signal or data are multiplexed. For example, the stream signal may be an MPEG-2 transport stream TS in which an MPEG-2 standard video signal, a Dolby AC-3 standard audio signal, and the like are multiplexed. Specifically, the MPEG-2 TS may include a header of 4 bytes and a payload of 184 bytes.
Meanwhile, it is possible for the above-described demodulation unit 4120 to be provided separately according to the ATSC method and the DVB method. That is, the digital device may separately include an ATSC demodulator and a DVB demodulator.
The stream signal output from the demodulator 4120 may be input to the controller 4170. The control unit 4170 may control demultiplexing, video/audio signal processing, and the like, and control an image output through the display unit 4180 and an audio output through the audio output unit 4185.
The external device interface unit 4156 provides an environment in which various external devices are interfaced to the digital device 4100. To this end, the external device interface unit 4135 may include an A/V input/output unit (not shown) or a wireless communication unit (not shown).
The external device interface 4156 may be connected in a wired/wirelessly manner to external devices such as a digital versatile disk (DVD), a blu-ray, game devices, cameras, camcorders, computers (laptops, tablets), smartphones, Bluetooth devices, and cloud. The external device interface unit 4156 transmits a video, audio, or data (including image) signal input from the outside through the connected external device to the controller 4170 of the digital device. The control unit 4170 may control the processed image, audio, or data signal to be output to the connected external device. To this end, the external device interface unit 4156 may further include an A/V input/output unit (not shown) or a wireless communication unit (not shown).
The A/V input/output unit may include a USB terminal, a composite video banking sync CVBS terminal, a component terminal, an S-video terminal (analog), and a DVI (digital visual interface) terminal, an HDMI (high definition multimedia interface) terminal, an RGB terminal, a D-SUB terminal and the like, to input video and audio signals of an external device to the digital device 4100.
The wireless communication unit may perform short-range wireless communication with other electronic devices. The digital device 4100 may be networked with other electronic devices according to a communication protocol, for example, Bluetooth, radio frequency identification RFID, infrared data association IrDA, ultra wideband UWB, ZigBee, digital living network alliance DLNA, etc.
And, the external device interface unit 4156 may be connected to at least one of various set-top boxes and various terminals described above, and perform input/output operations with the set-top box.
Meanwhile, the external device interface unit 4156 may receive an application or a list of applications in an adjacent external device and transmit it to the control unit 4170 or the storage unit 4140.
The network interface unit 4130 provides an interface for connecting the digital device 4100 with a wired/wireless network including an Internet network. The network interface unit 4130 may include, for example, an Ethernet terminal or the like for connection with a wired network, and use communication standards such as, for example, a wireless LAN (WLAN) (Wi-FI), wireless broadband (Wibro), world interoperability for microwave access (Wimax), and high speed downlink packet access (HSDPA) for connection with a wireless network.
The network interface unit 4130 may transmit or receive data with other users or other digital devices through a connected network or another network linked to the connected network. In particular, some content data stored in the digital device 4100 may be transmitted to another user registered in advance in the digital device 4100 or to a selected user or selected digital device among other digital devices.
Meanwhile, the network interface unit 4130 may access a predetermined web page through a connected network or another network linked to the connected network. That is, it is possible to connect to a predetermined web page through a network and transmit or receive data with the corresponding server. In addition, content or data provided by a content provider or a network operator may be received. That is, it is possible to receive content such as a movie, advertisement, game, VOD, broadcast signal, and related information provided by a content provider or a network provider through a network. Further, it is possible to receive update information and update files of firmware provided by a network operator. The network interface unit 4130 may also send data to the Internet or a content provider or network operator.
Also, the network interface unit 4130 may select and receive a desired application from among applications that are open to the public through a network.
The storage unit 4140 may store a program for processing and controlling each signal in the controller 4170, or may store a signal-processed image, audio, or data signal.
Also, the storage unit 4140 may perform a function for temporarily storing an image, audio, or data signal input from the external device interface unit 4135 or the network interface unit 4130. The storage unit 4140 may store information related to a predetermined broadcast channel through a channel memory function.
The storage unit 4140 may store an application or application list input from the external device interface unit 4135 or the network interface unit 4130.
Also, the storage unit 4140 may store various platforms described later.
The storage unit 4140 may include at least one storage medium among, for example, flash memory type, hard disk type, multimedia card micro type, and a card type memory (for example, SD or XD memory, etc.), RAM and ROM (EEPROM, etc.). The digital device 4100 may play and provide content files (video files, still image files, music files, document files, application files, etc.) stored in the storage unit 4140 to the user.
The user input interface unit 4150 transmits a signal input by the user to the control unit 4170 or a signal from the control unit 4170 to the user.
For example, the user input interface unit 4150 may receive a control signal related to controls power on/off, channel selection, and screen setting from the remote control device 4200 according to various communication methods such as an RF communication method and an infrared IR communication method and process it. Or the user input interface unit 4150 may process a control signal from the control unit 4170 to be transmitted to the remote control device 4200.
And, the user input interface unit 4150 may transmit a control signal input from a local key (not shown) such as a power key, a channel key, a volume key, and a value-setting key to the controller 4170.
The user input interface unit 4150 may transmit a control signal input from a sensing unit (not shown) that senses a user's gesture to the control unit 4170, or transmit a signal from the control unit 4170 to a sensing unit (not shown). Here, the sensing unit (not shown) may include a touch sensor, a voice sensor, a position sensor, and a motion sensor.
The controller 4170 may de-multiplex the stream input through the tuner 4110, the demodulator 4120, or the external device interface 4156 or processes the demultiplexed signals to generate and out a signal for video or audio output. The control unit 4170 may include the above-described encoding apparatus and/or decoding apparatus.
The image signal processed by the controller 4170 may be input to the display unit 4180 and displayed as an image corresponding to the image signal. Also, the image signal processed by the control unit 4170 may be input to an external output device through the external device interface unit 4156.
The audio signal processed by the control unit 4170 may be audio output to the audio output unit 4185. Also, the audio signal processed by the controller 4170 may be input to the external output device through the external device interface 4156.
Although not illustrated in
The control unit 4170 may control the overall operation of the digital device 4100. For example, the control unit 4170 may control the tuner 4110 to tune an RF broadcast corresponding to a channel selected by a user or a pre-stored channel.
The control unit 4170 may control the digital device 4100 by a user command input through the user input interface unit 4150 or an internal program. In particular, it is possible to access a network and download a desired application or application list into the digital device 4100.
For example, the control unit 4170 controls the tuner 4110 in order for a signal of a channel selected according to a predetermined channel selection command received through the user input interface unit 4150 to be input, and process video, audio, or data signals of the selected channel. The control unit 4170 allows the channel information, etc. selected by the user to be output through the display unit 4180 or the audio output unit 4185 along with the processed image or audio signal.
As another example, according to an external device image playback command received through the user input interface unit 4150, the control unit 4170 may allow a video signal or an audio signal input through the external device interface unit 4135 from an external device (for example, a camera or camcorder) to be output through the display unit 4180 or the audio output unit 4185.
Meanwhile, the control unit 4170 may control the display unit 4180 to display an image. For example, the control unit 4170 may control the display unit 4180 to display a broadcast image input through the tuner 4110, an external input image input through the external device interface unit 4156, an image input through the network interface unit, or an image stored in the storage unit 4140. At this time, the image displayed on the display unit 4180 may be a still image or a video, and may be a 2D video or a 3D video.
Also, the control unit 4170 may control to play content. At this time, the content may be content stored in the digital device 4100, or received broadcast content, or external input content input from the outside. The content may be at least one of a broadcast image, an external input image, an audio file, a still image, a connected web screen, and a document file.
When entering an application view item, the control unit 4170 may control to display a list of applications or applications that is located within the digital device 4100 or that can be downloaded from an external network.
The control unit 4170 may control to install and operate an application downloaded from an external network along with various user interfaces. And, the control unit 4170 may control an image related to an application to be executed to be displayed on the display unit 4180 by a user's selection.
Although not shown in the drawing, it is also possible to be further equipped with a channel browsing processing unit for generating a thumbnail image corresponding to a channel signal or an external input signal.
The channel browsing processing unit may receives a stream signal TS output from the demodulation unit 4120 or a stream signal output from the external device interface unit 4156, extract an image from the input stream signal, and generate a thumbnail image.
The generated thumbnail image may be input to the control unit 4170 as it is or as encoded. Also, the generated thumbnail image may be encoded in a stream form and input to the control unit 4170. The control unit 4170 may display a thumbnail list including a plurality of thumbnail images on the display unit 4180 using the input thumbnail images. The thumbnail images in this thumbnail list may be updated sequentially or simultaneously. Accordingly, the user can easily grasp the contents of a plurality of broadcast channels.
The display unit 4180 converts image signals, data signals, OSD signals and the like processed by the controller 4170 or image signals and data signals received from the external device interface unit 4156 into R, G, and B signals, respectively and generates a drive signal.
The display unit 4180 may be a PDP, LCD, OLED, flexible display, 3D display, or the like.
The display unit 4180 may be configured as a touch screen and used as an input device in addition to an output device.
The audio output unit 4185 receives a signal processed by the controller 4170, for example, a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs it as audio. The audio output unit 4185 may be implemented as various types of speakers.
Meanwhile, in order to sense a user's gesture, as described above, a sensing unit (not shown) having at least one of a touch sensor, a voice sensor, a position sensor, and a motion sensor may be further provided in the digital device 4100. The signal detected by the sensing unit (not shown) may be transmitted to the control unit 4170 through the user input interface unit 4150.
A photographing unit (not shown) for photographing a user may be further provided. Image information photographed by the photographing unit (not shown) may be input to the control unit 4170.
The control unit 4170 may detect a user's gesture by individually or in combination with an image captured by the photographing unit (not shown) or a signal sensed by the sensing unit (not shown).
The power supply unit 4190 supplies power throughout the digital device 4100.
In particular, the power supply 4190 may supply power to the control unit 4170 that can be implemented in the form of a system on chip SOC, the display unit 4180 for image display, and the audio output unit 4185 for audio output.
To this end, the power supply unit 4190 may include a converter (not shown) that converts AC power into DC power. For example, when the display unit 4180 is implemented as a liquid crystal panel having a plurality of backlight lamps, a PWM-operable inverter (not shown) may be further provided for luminance change or dimming. driving.
The remote control device 4200 transmits a user input to the user input interface unit 4150. To this end, the remote control device 4200 may use Bluetooth, RF (radio frequency) communication, infrared IR communication, UWB (Ultra Wideband), ZigBee (ZigBee) method, etc.
In addition, the remote control device 4200 may receive an image, audio, or data signal output from the user input interface unit 4150, display it on the remote control device 4200, or output voice or vibration.
The digital device 4100 described above may be a digital broadcast receiver capable of processing a fixed or mobile ATSC type or DVB type digital broadcast signal.
Furthermore, the digital device according to the disclosure may omit some components or further include components not illustrated, as required. As described above, the digital device may not have a tuner and a demodulator, and may also receive and play content through a network interface unit or an external device interface unit.
An example of the control unit may include a demultiplexing unit 4210, an image processing unit 4220, an on-screen display OSD generating unit 4240, a mixer 4250, a frame rate converter FRC 4255, and a formatter 4260. In addition, although not illustrated, the control unit may further include a voice processing unit and a data processing unit.
The demultiplexing unit 4210 demultiplexes an input stream. For example, the demultiplexing unit 4210 may demultiplex input MPEG-2 TS into video, audio, and data signals. Here, the stream signal input to the demultiplexing unit 4210 may be a stream signal output from a tuner or demodulator or an external device interface.
The image processing unit 4220 performs image processing of the demultiplexed image signal. To this end, the image processing unit 4220 may include a video decoder 4225 and a scaler 4235.
The video decoder 4225 decodes the demultiplexed video signal, and the scaler 4235 scales the resolution of the decoded video signal to be output on the display unit.
The video decoder 4225 may support various standards. For example, the video decoder 4225 performs the function of the MPEG-2 decoder when the video signal is encoded in the MPEG-2 standard, and the function of the H.264 decoder when the video signal is encoded in the digital multimedia broadcasting DMB method or the H.264 standard.
The video signal decoded by the video processing unit 4220 is input to the mixer 4250.
The OSD generating unit 4240 generates OSD data according to a user input or by itself. For example, the OSD generating unit 4240 generates data for displaying various data on a screen of the display unit 4180 in a graphic or text form based on a control signal of the user input interface unit. The generated OSD data includes various data such as a user interface screen of a digital device, various menu screens, widgets, icons, and viewing rate information.
The OSD generating unit 4240 may generate data for displaying subtitles of broadcast images or broadcast information based on EPG.
The mixer 4250 mixes the OSD data generated by the OSD generating unit 4240 and the image signal processed by the video processing unit and provide it the formatter 4260. Because the decoded video signal and the OSD data are mixed, the OSD is displayed overlaid on a broadcast video or an external input video.
The frame rate converter FRC 4255 converts a frame rate of an input video. For example, the frame rate converter 4255 may convert the input 60 Hz image frame rate to have a frame rate of, for example, 120 Hz or 240 Hz, depending on the output frequency of the display unit. As described above, various methods may exist in the method for converting the frame rate. For example, when the frame rate converter 4255 converts the frame rate from 60 Hz to 120 Hz, the same first frame is inserted between the first frame and the second frame, or a third frame predicted from the first frame and the second frame is inserted between the first frame and the second frame. As another example, when the frame rate converter 4255 converts the frame rate from 60 Hz to 240 Hz, three identical frames or predicted frames may be inserted between existing frames. If a separate frame conversion is not performed, the frame rate conversion unit 4255 may be bypassed.
The formatter 4260 changes the output of the frame rate converter 4255 to match the output format of the display unit. For example, the formatter 4260 may output R, G, and B data signals, and these R, G, and B data signals may be output as low voltage differential signaling LVDS or mini-LVDS. Also, when the output of the frame rate converter 4255 is a 3D video signal, the formatter 4260 may support 3D service through the display unit by configuring the output as a 3D format according to the output format of the display unit.
An audio processing unit (not shown) in the control unit may perform audio processing of the demultiplexed audio signal. The audio processing unit (not shown) may support various audio formats. For example, even when an audio signal is encoded in formats such as MPEG-2, MPEG-4, AAC, HE-AAC, AC-3, BSAC, the audio processing unit may be provided with a decoder corresponding thereto.
Also, the audio processing unit (not shown) in the control unit may process a base, treble, volume control, and the like.
A data processing unit (not shown) in the control unit may perform data processing of a demultiplexed data signal. For example, the data processing unit may decode the demultiplexed data signal even when it is encoded. Here, the encoded data signal may be EPG information including broadcast information such as start time and end time of a broadcast program broadcast on each channel.
Meanwhile, the above-described digital device is an example according to the disclosure, and each component may be integrated, added, or omitted depending on the specification of the actual digital device. That is, if necessary, two or more components may be combined into one component, or one component may be subdivided into two or more components. Also, a function performed in each block is for describing an embodiment of the disclosure, and the specific operation or device does not limit the scope of the disclosure.
The digital device may be an image signal processing device that performs signal processing of an image stored in the device or an input image. As another example of the image signal processing apparatus, a set-top box STB, a DVD player, a Blu-ray player, a game device, a computer and the like may be further exemplified from which the display unit 4180 and the audio output unit 4185 shown in
The digital device according to an embodiment may simultaneously display a main image 4310 and an auxiliary image or sub image 4320 on the screen 4300. The main image 4310 may be referred to as a first image, and the auxiliary image 4320 may be referred to as a second image. The main image 4310 and the auxiliary image 4320 may include a video, a still image, an electronic program guide EPG, a graphical user interface GUI, an on-screen display OSD, and the like and are not limited thereto. The main image 4310 may mean an image that is relatively smaller in size than the screen 4300 of the electronic device while being simultaneously displayed on the screen 4300 of the electronic device along with the auxiliary image 4320, and may be referred to as a picture in picture PIP. In
The main image 4310 and the auxiliary image 4320 may be related to each other directly or indirectly. As an example, the main image 4310 may be a streaming video, and the auxiliary image 4320 may be a GUI that sequentially displays thumbnails of videos including information similar to the streaming video. As another example, the main image 4310 may be a broadcast image, and the auxiliary image 4320 may be an EPG. As another example, the main image 4310 may be a broadcast image, and the auxiliary image 4320 may be a GUI. Examples of the main image 4310 and the auxiliary image 4320 are not limited thereto.
In one embodiment, the main image 4310 is a broadcast image received through a broadcasting channel, and the auxiliary image 4320 may be information related to a broadcast image received through a broadcast channel. Information related to a broadcast image received through a broadcast channel may include, for example, EPG information including an integrated channel schedule and broadcast program detailed information, and broadcast program review information, but is not limited thereto.
In another embodiment, the main image 4310 is a broadcast image received through a broadcast channel, and the auxiliary image 4320 may be an image generated based on information pre-stored in a digital device. The image generated based on the information pre-stored in the digital device may include, for example, a basic user interface UI of the EPG, basic channel information, an image resolution manipulation UI, and a bedtime reservation UI, and does not limited thereto.
In another embodiment, the main image 4310 is a broadcast image received through a broadcast channel, and the auxiliary image 4320 may be information related to a broadcast image received through a network network. The information related to the broadcast image received through the network may be, for example, information obtained through a search engine based on the network. More specifically, for example, information related to a character currently being displayed on the main image 4310 may be obtained through a network-based search engine.
However, the example is not limited to this, and information related to a broadcast image received through a network may be obtained by using, for example, an artificial intelligence AI system. More specifically, for example, an estimated-location in map of a place currently being displayed on the main image 4310 may be obtained by using network-based deep-learning, and the digital device may receive information about the estimated location on the map of the place currently being displayed on the main image 4310 through the network.
The digital device according to an embodiment may receive at least one of image information of the main image 4310 and image information of the auxiliary image 4320 from the outside. The image information of the main image 4310 may include, for example, a broadcast signal received through a broadcasting channel, source code information of the main image 4310, and IP packet (internet protocol packet) information of the main image 4310 received through a network, but is not limited thereto. Similarly, the image information of the auxiliary image 4320 includes, for example, a broadcast signal received through a broadcast channel, source code information of the auxiliary image 4320, IP packet information of the auxiliary image 4320 received through a network, etc., but is not limited to. The digital device may decode and use the image information of the main image 4310 received from the outside or the image information of the auxiliary image 4320. However, in some cases, the digital device may store image information of the main image 4310 or image information of the auxiliary image 4320 internally.
The digital device may display the main image 4310 and the auxiliary image 4320 on the screen 4300 of the digital device based on the image information of the main image 4310 and information related to the auxiliary image 4320.
In one example, the decoding apparatus 200 of the digital device includes a main image decoding apparatus and an auxiliary image decoding apparatus, and the main image decoding apparatus and the auxiliary image decoding apparatus may respectively decode image information of the main image 4310 and image information of the auxiliary image 4320. A renderer includes a main video renderer (first renderer) and an auxiliary video renderer (second renderer). The main image renderer may display the main image 4310 on the first area of the screen 4300 of the digital device based on the information decoded by the main image decoding apparatus, and the auxiliary image renderer may cause the auxiliary image 4320 to be displayed on the second area of the screen 4300 of the digital device based on the information decoded by the auxiliary image decoding apparatus.
In another example, the decoding apparatus 200 of the digital device may decode image information of the main image 4310 and image information of the auxiliary image 4320. Based on the information decoded by the decoding apparatus 200, the renderer may process the main image 4310 and the auxiliary image 4320 together to be simultaneously displayed on the screen 4300 of the digital device.
That is, according to this document, it is possible to provide a method for processing an image service in a digital device. The image service processing method may comprise: receiving image information, decoding a (main) image based on the image information, rendering or displaying the decoded image in a first area on the display, and rendering or displaying an auxiliary image an auxiliary image in a second area on the display. In this case, the step of decoding the first image may follow the decoding procedure in the decoding apparatus 200 according to
For example, the auxiliary image may be an electronic program guide EPG, an on-screen display OSD, or a graphical user interface GUI. For example, the image information may be received through a broadcast network, and information regarding the auxiliary image may be received through the broadcast network. For example, the image information may be received through the communication network, and information regarding the auxiliary image may be received through the communication network. For example, the image information may be received through the broadcast network, and information regarding the auxiliary image may be received through the communication network. For example, the image information may be received through the broadcast network or the communication network, and information regarding the auxiliary image may be stored in a storage medium in the digital device.
In the aforementioned embodiments, the elements and characteristics of the disclosure have been combined in a specific form. Each of the elements or characteristics may be considered to be optional unless otherwise described explicitly. Each of the elements or characteristics may be implemented in a form to be not combined with other elements or characteristics. Furthermore, some of the elements and/or the characteristics may be combined to form an embodiment of the disclosure. The sequence of the operations described in the embodiments of the disclosure may be changed. Some of the elements or characteristics of an embodiment may be included in another embodiment or may be replaced with corresponding elements or characteristics of another embodiment. It is evident that an embodiment may be constructed by combining claims not having an explicit citation relation in the claims or may be included as a new claim by amendments after filing an application.
The embodiment according to the disclosure may be implemented by various means, for example, hardware, firmware, software or a combination of them. In the case of an implementation by hardware, the embodiment of the disclosure may be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.
In the case of an implementation by firmware or software, the embodiment of the disclosure may be implemented in the form of a module, procedure or function for performing the aforementioned functions or operations. Software code may be stored in the memory and driven by the processor. The memory may be located inside or outside the processor and may exchange data with the processor through a variety of known means.
It is evident to those skilled in the art that the disclosure may be materialized in other specific forms without departing from the essential characteristics of the disclosure. Accordingly, the detailed description should not be construed as being limitative, but should be construed as being illustrative from all aspects. The scope of the disclosure should be determined by reasonable analysis of the attached claims, and all changes within the equivalent range of the disclosure are included in the scope of the disclosure.
The aforementioned preferred embodiments of the disclosure have been disclosed for illustrative purposes, and those skilled in the art may improve, change, substitute, or add various other embodiments without departing from the technical spirit and scope of the disclosure disclosed in the attached claims.
This application is a continuation of U.S. application Ser. No. 18/127,857, filed Mar. 29, 2023, which is a continuation of U.S. application Ser. No. 17/941,628, filed on Sep. 9, 2022, which is a continuation of U.S. application Ser. No. 16/896,532, filed on Jun. 9, 2020 (now U.S. Pat. No. 11,477,478, issued on Oct. 18, 2022), which is a continuation of International Application No. PCT/KR2019/017607, filed on Dec. 12, 2019, which claims the benefit of U.S. Provisional Application No. 62/778,835, filed on Dec. 12, 2018, the contents of which are all hereby incorporated by reference herein in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10491902 | Xu | Nov 2019 | B1 |
| 20200059658 | Chien | Feb 2020 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20240171771 A1 | May 2024 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62778835 | Dec 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 18127857 | Mar 2023 | US |
| Child | 18387809 | US | |
| Parent | 17941628 | Sep 2022 | US |
| Child | 18127857 | US | |
| Parent | 16896532 | Jun 2020 | US |
| Child | 17941628 | US | |
| Parent | PCT/KR2019/017607 | Dec 2019 | WO |
| Child | 16896532 | US |
